Achieving Information Sharing in Federal Agencies via Rapid Data Services Enablement and SOA

Achieving Information Sharing in Federal Agencies via Rapid Data Services Enablement and SOA

Chuck Mosher & Tony Vachino

MetaMatrix

October 31, 2006

MetaMatrix and the SOA CoP Demo

2

Agenda

• Data Services Rationale & Best Practices

• MetaMatrix Products & Capabilities

• Achieving Information Sharing– Service Enabling Data Assets– Resolving Semantics

• Demo

• Summary/Q & A

3

Handling The Data Challenge

Resolving data semantic and structural mismatches

SOA-enabling legacy data systems (i.e., Net Centricity)

Mapping data sources to vocabularies like XDM, NIEM, C2IEDM, HL7, …

Handling multi-source requests (data aggregation, mediation, fusion, federation)

Minimizing development and maintenance cost of custom code by metadata-based MDA

MetaMatrix can Help

Getting the right information to the right person at the right time requires:

4

Program Challenges• Multiple sources

• Different interfaces/drivers• Different physical structures• Different semantics

• Single interface to data desired• Real-time access to data• Performance• Maintainability as data changes• Maintainability as apps change

Mission Challenges• Time-to-deploy• Agility - Responsiveness to change• Automation – Reduce cost of new development and operations• ROI of enterprise information

Agency Challenges• 100’s/1000’s of data sources• 100’s/1000’s of applications• Multiple access points/modes for apps• Understanding relationships/semantics• Data consistency• Data reuse – bridging data silos• Support for Web Services & SQL• Control & manageability, compliance• Security & auditing

Information Resources

Communities of Interest

Information Challenges

?

5

Information Virtualization

Unified Semantic Layer

Information Virtualization Layer

Data Federation Layer

Data Access/Connectivity Layer

Enterprise Data Sources

Unification of different concepts across systemsSingle-query access to heterogeneous systemsUniform, standardized access to any system

6

Facilities ImagesUnitsSensors

Persons ODS Parts Personnel Documents

Information Services

MetaMatrix Information Integration Platform

Information on demand

High performance

Minimal replication

Manageable

Secure

Data Interoperability Through Information Services

C2

Intel

Weapon Systems

Dynamically Created COIs

LogisticsSensors

Personnel

Finance

Etc.

7

What is a Data Service?

MasterData

OperationalData Store

AgencyApplication

Data Service

SQL SQL APICall

XML/SOAP

• Decouple data sources from application– Data implementation shielded

from application• Semantic/Format Mediation

– Standard vocabulary • Single access point

– Web Service/XML– SQL

• Federation– Single source or multi-source

• Scalability– Security, performance

Bridge theGap

SQL

8

Data Services: Designed for Agility

• Data Services Best Practices– Provide transparency across all sources

– Define known relationships today and accommodate future relationships

– Support independence of mission systems

– Support ownership of operational data sources at the source

– Provide accelerated mechanisms for integrating new sources

– Support existing security policy and add degrees of security

• The value of a managed metadata abstraction layer– "Future Proofing" (future standards, exchange models, platforms)

– Limited skill set requirements

– Fixed long term costs for integration middleware

• Building consensus– Assure data owners they will continue to have control, and …

– Vocabulary of existing production systems will not be impacted

– Offer an option where legacy data migration is not 'required' 1st

9

Data,ContentSources

Logical Data Model

Data Services Approaches

T

Org, Person, Image,

Location

MaterializedLogical Model

<X>

</X>

<X>

</X>

<X>

<X>

<X>

</X>

<X>

</X>

<X>

<X>

Data Services for Multiple Purposes:

• Simplified access to value-added (tagged) data in real-time• Value-added (tagged) data materialized & staged

• Phased-in migration from legacy to new• Managed archiving via classification, retention tags

• Enhanced search via consistent content tags

Model-Driven Integration LayerModel-Driven Integration Layer

Data,ContentSources

Logical Data ModelT

Organization, Customer, Imagery, Location

MaterializedLogical Model

<X>

</X>

<X>

</X>

<X>

<X>

<X>

</X>

<X>

</X>

<X>

<X>

AgileInformation

Services

<X>

</X>

<X>

</X>

<X><X>

<X>

</X>

<X>

</X>

<X><X>

<X>

</X>

<X>

</X>

<X><X>

Enriched Data/Content Store

10

Search Engine Index / Metadata Catalog

Master Data Person / Facility / Vehicle

Enterprise Data Services

Stage SOAApp’s

Federal Agencies

Data Access Services• SQL, Web Service/XML• Staged Data (optional)

OntologyMgmt /Reasoning

Enterprise Service Bus / Intranet / Extranet

Distributed Data Services

Land/Sea

State/Local• Security/Authentication• Operations Management • Error / Exception Management

• Orchestration• Encryption• High Availability

MediationXSLT, Multi-source

Information Exchange Topology

11

Agenda




• Demo

• Summary/Q & A

12

MetaMatrix I.P.

MetaMatrix has 2 distinct innovations that work in concert to yield significant business benefits:

Model-basedModel-based

ExtensibleExtensible

Sharable, reusableSharable, reusable

Standards-basedStandards-based

Information ModelingInformation Modeling

Federated QueryingFederated Querying

Cost-based optimizerCost-based optimizer

Read/write/transactionsRead/write/transactions

Uniform API, any sourceUniform API, any source

Battle-tested/hardenedBattle-tested/hardened

13

Designing data services

MetaMatrix Approach to Data Services

xml

databases

warehouses

spreadsheets

services

<sale/> <value/></ sale >

geo-spatial

rich media

…Enterprise Enterprise Information Information

Sources (EIS)Sources (EIS)

Information Information ConsumersConsumers

Reusable,Reusable,Integrated Data Integrated Data

ObjectsObjects

ExposedExposedDataData

ServicesServices

<WSDL><WSDL>(contract)



Custom Apps

Web Services,Business Processes

Packaged Apps

Reporting, Analytics

EAI, Data warehouses

OD

BC

JDB

CS

OA

P

Logistics

Intelligence

14

• Transformations from one or more sources

• Transformations defined with:– Joins/unions– Criteria– Functions

• Elements mapped to dictionary

• Business definitions captured

Data Service Abstraction Layers

15

MetaMatrix Integration Server

Information Consumers

Web Svc XML RDBMSPackaged Connectors

Siebel,SAP

OracleApps

CICSVSAM

MetaMatrix Catalog

MetaMatrix Designer- Design and deploy data services

MetaMatrix Products

JMSODBC JDBC SOAP

QueryProcessor

ProcessorProcessorOptimizerOptimizer

Integration ServerVirtualDataBases

VDBVDBVDBVDB

IntegratedSecurity

UsersUsers

RolesRoles

EntitlementsEntitlements

AccessModels

ViewsViews XMLDocsXMLDocs

<a>

</a>

<b>

</b>…

ServicesServices

in outproc

MetaMatrix Connector Framework

MetaMatrixServer

16

Secure Access – Accredited

MetaMatrixClient AppClient Appusernamepassword

Membership Provider

Membership Provider

usernamepassword

authenticates

Connector

Connector

Connector

Connector

Data Source

Optionally accessessource-specific information

source-specific

trustedpayload

MetaMatrixClient AppClient App

Membership Provider

Membership Provider

Authentication Service

Authentication Service

logoninfo

authenticates,generates payload

trustedpayload payload

trustedpayload

authenticates,optionally modifies payload

payload

Username/Password Logon • Connector connects with same ID for all queries• Optional: Integrated with existing authentication system

Trusted Payload Logon:• Connector uses different credentials per connection, per query • Optional: Integrated with existing authentication system

Data Source

17

Process X

Process Y

Processes[BPM/BPEL]

Ontologies[OWL/RDF]Taxonomies

ServiceA

ServiceB

Web Services [WSDL]

Classification Schemes

Taxonomy A

KeyWords B

Relational

XMLXML

XML

XMLTransformations

DatatypesXMLRel

Rel

Domain[UML/ER]

MetaMatrixDesigner

MetaMatrixCatalog

GenericTypedRelationships

Models & Files[versioned]

Models & Files[versioned]

Search Index

Search Index

Web Reporting

Web Reporting<X>

</X>

<X>

</X>

…

<X>

</X>

<X>

</X>

…

WSDL

Public Health

Justice

Environment

Geo-spatial

Recreation

Immunization

Warrant

Wildlife

…

Camping

Public Health

Justice

Environment

Geo-spatial

Recreation

Immunization

Warrant

Wildlife

…

Camping

Public Health

Justice

Environment

Geo-spatial

Recreation

Immunization

Warrant

Wildlife

…

Camping

Public Health

Justice

Environment

Geo-spatial

Recreation

Immunization

Warrant

Wildlife

…

Camping

Public Health

Justice

Environment

Geo-spatial

Recreation

Immunization

Warrant

Wildlife

…

Camping

Public Health

Justice

Environment

Geo-spatial

Recreation

Immunization

Warrant

Wildlife

…

Camping

Application/Configuration

Managing Data Service Metadata

18

MetaMatrixEnterprise

MetaMatrixDimension

MetaMatrixQuery

MetaMatrix Product Lines

MetaMatrix Enterprise • Web services & SQL• Modeling enterprise data• Scalable deployment server• Metadata management• Application/legacy connectors

MetaMatrix Dimension • Web service-enablement of data sources• Expose business views as XML• Lightweight modeling – rapid integration• Standard WAR-based deployment

MetaMatrix Query • Embeddable Java component • Federated query engine• Query optimization• Standard JDBC to all sources• Standard SQL to all sources

En

terp

rise

Pro

ject

, No

de

ISV

/ P

roje

ct

19

Agenda




• Demo

• Summary/Q & A

20

T

«Text File»

«Relational»

«Application»MetaMatrix:Mapping from Data to XML

Source: Data Sources containingInformation to integrate

Target: Fixed (potentially complex) XML SchemaNeed:Data complying to Schema

Mediation: XML From Non-XML Sources

«XML»

<person> <addresses> … </addresses> <accounts> <accountID=…> … </accountID> </accounts></person>

21

• Model XML Docs, Schemas

• Build XML Doc. models from XML Schemas

• Map XML Doc. models to other data models

• Enable data access via XML

Map Data Sources to XML & Deploy

MetaMatrix Designer – for XML-centric Data Services

22

Dimension – Choose your approach

• Rapid design & deployment of Web Services• Expose integrated data as XML-based business views• Deployment of Web Services as standard Web apps• Runtime execution optimized through use of MetaMatrix Query Engine

Dimension Models

Web Server

Data Sources

Business Views

<XML><XML><XML>

Web Service Operations

WSDLXSD

Source Models

DeployImport Map Model

WARastoto

Start Here?

Start Here?

23

Agenda




• Demo

• Summary/Q & A

24

T

Authoritative Sources:• Mapped to logical

Multiple Internal/External Information Sources

Application views of information:

• Relational, XML

T T

XML Document<a>

</a>

<b>

</b>…

T

TT

ODBC/JDBC JDBC SOAP

WebServices

WebServices

Search Applications

Search Applications

BusinessIntelligence

Applications


Applications

Logical Data Model:• Agency or COI-specific• Rationalize, harmonize,

mediate

C2, Logistics, Intelligence, …

COI Data Dictionary

bldg_id SITENUM Facility_ID

Location_ID

bldg_type Depot_Number

Location_Type

25

FBI CBP NYC NY NJ

SemanticData Services

Matched (Confidence of 90%)

Gender ID

Person Sex Code

Ontology

“Sex” semantically related to “Gender”

Semantic Matching - example

Data Sources

Semantic Data Services– key component of information sharing

and interoperability programs – automated semantic mapping to aid

domain experts in quickly reconciling disparate schemas and vocabularies

– more rapid deployment of a mediation solution

MatchIt – an extensible ontology-driven tool– variety of algorithms for determining

semantic equivalence– discovers similarities between

elements of heterogeneous data, automatically exposing potential semantic matches.

– matches elements of data sources to target schemas of Data Services, such as TWPDES, GJXDM, NIEM, C2IEDM, HL7

26

Automated Term Discovery (Interpret)

A comprehensive list of terms automatically discovered across all sources

All the available definitions found in the MatchIT knowledge-base

All the usage instances where each term was used in any of the sources

Results of the automated

tokenization

27

Contextualize (Interpret)

Automated term tokenization

Automated semantic linking using the default knowledge-base contained within MatchIT

ArticleAmount

Amount Article

Sum

Assets

Creation

Synonym

Type-of

28

Semantic Matching (Mediate)

• With relationships pre-established within the knowledge-base…

• Identify the Target and the Source(s) and run the match.

ArticleAmount

ProductShares

Automatically linked by a specific % distance

29

Facilitate Decision Making (Mediate)

Helps facilitate rapid decision making

Target element for matching

Automatically calculated semantic distance between terms

Source candidate for matching

30

J-8 Force Structure

J-7 Operational Plans

J-6 C4CS

TData Sources- Authoritative- Redundant

- Overlapping

Multiple Internal/External Information Sources

T T

ODBC/JDBC JDBC SOAP

WebServices

WebServices

Portal Applications

Portal Applications


Applications


Applications

Enterprise-wide or COI-driven Data Models

• Rationalization• Harmonization• Data Catalogs (DDMS)

Support Multiple Enterprise Semantic Models

J-5 Plans & Policy

J-4 Logistics (GCSS)

J-3 Operations

J-2 Intelligence

J-1 Manpower / Personnel

31

Agenda




• Demo

• Summary/Q & A

32

Agenda




• Demo

• Summary/Q & A

33

MetaMatrix – Quick Facts

Middle-ware, model-driven, data management

DoD proven (DISA, NSA, TRANSCOM, etc.)

Version 5 – Mature product which is still unique and ahead of the competition

NIAP certified and NSA-credentialed

Can handle the enterprise (or COI) perspective as well as the bottom-up perspective (data service enablement of legacy systems)

Can rapidly implement data integration strategies

34

Major US Federal Government Customers

NSA - Multiple Programs (NES Base-lined) In-Q-Tel/CIA TRANSCOM – Command Metadata

Management System Air Force - Command and Control Center DISA - Global Combat Support Systems

(GCSS) DISA – Anti Drug Network (ADNET) DLA – Integrated Data Environment (IDE) Mitre – Air Force ESC/DoD DDMS work UK – NSA Equivalent, CJIT

35

• On-demand information– Real time data integration– Information sharing between business units

• Enabling SOA in an evolving world– Consume and produce Web services– And still provide full support for ODBC, JDBC, and legacy

• Federation of disparate information– Rationalized to controlled vocabularies– Relational + XML + Web Services + Enterprise Apps + Legacy

• Faster time to market– Integrated information in days, weeks– Tight coupling of design & implementation phases– Leveraging the skill-set of the data architects for integration

• Costs across application lifecycle reduced– Model-driven abstraction layer eases development/maintenance– Better management of data assets across the enterprise

MetaMatrix Value PropositionMetaMatrix Value PropositionHighly cost-effective COTS tool for rapid enterprise

information integration and exchange