Interoperability API OverviewInteroperability API Overview

May 21, 2003

AgendaAgenda

• Welcome / Meeting Objectives – Mark Z.• EM-XML “Snapshot” – Matt Walton• Introduction – Dick Munnikhuysen• Tech Overview of DMIS – Joe Kessler• Interoperability in DMIS – Neil Bourgeois• Reportable Level API – Jerry Warden• API and EM-XML Standards. – Gary Ham• Roll out and Access – Dick Munnikhuysen• Q&A - All

ObjectivesObjectives

• Explain the DM approach to APIs

• Provide a technical overview

EM XML “Snapshot”EM XML “Snapshot”

Matt Walton

What’s that mean in the real world?What’s that mean in the real world?Consider:• St Louis Riverfront Festival, July 4• Terrorist rocket into chlorine tank car• Lethal plume across:

• Unprotected thousands• Multiple municipal jurisdictions• 2 states• 2 FEMA regions

With an interoperability service, organizations can:• Share information• Gain early awareness• Coordinate response • Save lives and minimize property damageDespite differing automated systems

What’s the solution?What’s the solution?• Leverage technology to gain efficiency• Develop a national emergency information interoperability service

Nation

Region

Local

Interoperability Service enabling horizontal & vertical information sharing

StateEOCEOC

ICP

What’s an Interoperability Service?What’s an Interoperability Service?An infrastructure with common service functions that

enable heterogeneous automated information systems to “talk to each other.”

Incident Command System A

Incident Management System B

Incident Management

System C

Interoperability Services

Transaction Management Transaction Queuing

Transaction Re-synchronizationTime SynchronizationAccess Reconciliation

Communications Infrastructure

Transaction Alerts

Acknowledgements

Posting Receipts

Import/Export

Two basic modes:

CMI-Services Interoperability

Hub

Emergency Manager

EMS

Police

Fire Dept.

Internet

Internal collaboration

Your COG

Collaboration View – Updated

from within COG

Shared View – of your

information

Current Configuration

External sharing via Post function

Internet

State Police

Transportation

State EM

DPH

Another COG

Shared View – Posted from

your COG

How do COGs work?How do COGs work?Future Configuration

CMI-Services Interoperability

Hub

EMS

Fire Chief

Internal collaboration

Local Fire COG

Shared View – of your

information

Station 1

Police Chief

Precinct 2

Precinct 1

Local Police COG

Internet

Local CMIS Interoperability

Hub

Internet

State Police

Transportation

State EM

DPH

State DEM COG

Shared View – Posted

from Local COGs

External sharing via Post function

In response to HSPD-5, converging roads to NIMS technologies. . .In response to HSPD-5, converging roads to NIMS technologies. . .

DM

Dhelp collaboration tools and expert

reference

DMIS interoperability infrastructure and

basic responder tools

Geospatial One Stop

MatureNIMS

NIMS quick start

E-Authentication

SAFECOM

EM-XML

Government / science / industry collaboration developing technologies for homeland security

Today!

Technical Overview

Goals & ConstraintsGoals & Constraints

• Goals– Provide foundation for information sharing– Basic capabilities for “have not” stakeholders– Long-term focus on broader information sharing

• Server Constraints– Flexibility to adapt to shifts in technology– Capacity to scale to the national level– Good measure of privacy

• Desktop Constraints– Good performance on “real world” stakeholder hardware– Demonstrative of overall integration strategy

• Strategy– Establish overarching paradigm & implement framework– Create basic tools to validate paradigm & provide core functionality– Extend paradigm and refine focus on vendor interoperability

Computing RequirementsComputing Requirements

• Core access is through SOAP/XML– Platform independent– Language independent

• Operating system– Windows, Unix, Linux, others …– No constraints are imposed, interaction is platform-independent

• Programming languages– Java, C++, Microsoft .NET, others …– We have sample clients written in various environments

• Internet access• Peripherals required by client applications

– e.g. camera/microphone for on-line conferencing

Architectural FrameworkArchitectural Framework

• Based on “messaging” technology– Resilient to disruption in communication – Location independence– Provides single point of entry into server (enhances security)

• Works in concert with hardware infrastructure– Load balancing and redundancy– Reliability & scalability

• Platform for interoperability– Messaging is an ideal foundation for a Service Oriented Architecture– Messages can be neatly wrapped using SOAP

• Features built on messaging foundation– Interoperability features– Collaboration features– Offline operation features– Notification features

Application & Service Integration

Application & Service Integration

• Services reside on logical server– Most business logic is encapsulated remotely– Services are invoked using messages, collectively forming an API– Functions can be presented via GUI, HTTP, SOAP, etc.

• Desktop is “container” for applications– Provides basic services such as authentication– Applications “snap in” by

• Implementing specific interfaces• Being declared inside initialization files

– Interaction with services is through messaging• Overall integration strategy

– “Separation of Church and State”, or API-based integration– Modular services offering various functions– Modular clients focused on presentation– Integration points to be opened at both ends (server & desktop)

DisasterHelpPortal Web-Browser

DMIS-Client(Autonomous-Node)

Java-Application

DMIS-Client(Remote-Access

Browser-Launchedw Java Web-Start)

DMIS InteroperabilityPlatform

Interoperable-Client(Web-Services Interface)

DisasterHelpPortal

ExternalWeb-Services

External Web-MapServices

DMIS -WebResources

RS

P

B

D1

D3

D2

D3

P

D1 & D2 RS

B P

E Disaster ManagementDistributed Services Model

(Operational)

DMIS -MappingServices

E

GIS Data via GOS(Geospatial One-Stop

E-Gov Initiative)

Weather Forecast Datavia Accuweather

OSINT (Open Source Intelligence)Capabilities Catalog

Street Level Maps viaGDT & Navtech Data

Launchedfrom Portal

Main Contentfrom DMIS

RS

E

(RS) Remote Interoperability Services(P) DisasterHelp Portal(D) DMIS Clients(B) Web-Browser

ERS

External WebResources

Other E-Gov PartnerAgencies

DHelp-PortalWeb-Browser

DMIS-Client(Autonomous-Node)

Java-Application

DMIS-DeployedInteroperability Platform

DMIS-Client(Remote-Access

Browser-Launchedw/ Java Web Start)

DMIS InteroperabilityPlatform

Interoperable-Client(Web-Services Interface)

DHelp Portal

RS

DS

P

B

D1

D3

D2

RSDS

DS

PRS

(RS) Remote Interoperability Services(DS) Deployed Interoperability Services(P) DisasterHelp Portal(D) DMIS Clients(B) Web-Browser

P

RS E

Disaster ManagementDistributed Services Model

(Under Development)

DS

AlternativePath

ExternalWeb-Services

External Web-MapServices

DMIS -WebResources

DMIS -MappingServices

GIS Data via GOS(Geospatial One-Stop

E-Gov Initiative)

Weather Forecast Data viaNWS-NOAA, Accuweather

OSINT (Open Source Intelligence)Capabilities Catalog

Street Level Maps viaGDT & Navtech Data

External WebResources

Other E-Gov PartnerAgencies

E

B P

D3

P

D1 & D2 RS

E

RS

E

DS

ExternalInteroperable

Systems

EM-XML Gov’t andIndustry Consortium

Levels of Interoperability “Maturity”Levels of Interoperability “Maturity”Reportable – exchange of reports / attachments as files associated with a set of (primarily) delivery related data

Presentable – delivery of diverse information structures across applications without semantic assessment or translation Processable – application of algorithms to process or validate the semantic meaning of exchanged information

Standards based (i.e., EM-XML) – standard data structures and business rules for processing“Common Processable Form” - a standard dynamic structure with capabilities for dynamically capturing structure, business rules for processing, and even self-processable objects.

Expose Other Services – expose services provided by Interoperating Systems and other DMIS services (e.g., Structured Weather Data – Accuweather and NWS/NOAA in the future, National Street-Level Map Data (GDT, NavTech), Responder Alerting, etc.)

Interoperability

An infrastructure with common service functions that enable heterogeneous automated information systems to “talk to each other.”

Incident Command System A

Incident Management System B

Incident Management

System C

Interoperability Services

Transaction Management Transaction Queuing

Transaction Re-synchronizationTime SynchronizationAccess Reconciliation

Communications Infrastructure

Transaction Alerts

Acknowledgements

Posting Receipts

Import/Export

What’s an Interoperability Service?What’s an Interoperability Service?

The ApproachThe Approach

• Standards-based web services

• SOAP/XML

Why Web Services?Why Web Services?

• "Interoperable" means suitable for and capable of being implemented in a neutral manner on multiple operating systems and in multiple programming languages.

• The World Wide Web is more and more used for application to application communication. The programmatic interfaces made available are referred to as Web services.

• Web services interoperate across platforms, operating systems, and programming languages.

TermsTerms

TIE

Tactical Information Exchange; the service that allows the interchanging of incident data.

Incident

The core of the TIE; an incident is any event requiring the notice of users of DMI-Services. This is often a disaster demanding the attention of various disaster management services, but could also be a training scenario.

Terms (cont)Terms (cont)

COG

A Collaborative Operations Group within DMI-Services. This is a group of users that share incident data; the DMI-Services Web Services allows users from one COG to post information to or receive information from other COGs. In order to use the services, users must be members of a DMI-Services COG. For details about creating your own COG and gaining access to DMI-Services Web Services, see http://www.dmi-services.org/registration_center.asp.

Terms (cont)Terms (cont)

Post

To “post” an incident is to make it available to other COGs.

Attachment

Logically, any file that is attached to an incident. This could be a map of the area, a statement from bystanders, or any other file germane to the incident in question.

APIAPI

• Small API at first for basic capability

Typical interaction (Retrieval)Typical interaction (Retrieval)

1. Ping() the service to make sure service is up, and user is authenticated.

2. A call to getAllIncidents() is made to retrieve a list of TieIncidents.

3. A call can then be issued on getIncident() to obtain detailed incident information in the form of a TieIncident object.

    Typical interaction (Retrieval – cont)    Typical interaction (Retrieval – cont)

4. A call to getAttachmentList() returns a list of AttachmentDescriptors for a given incident.

5. Attachments can then be downloaded via a call to getAttachments().

 

Typical interaction (Posting)Typical interaction (Posting)

1. Ping() the service to make sure service is up, and user is authenticated.

2. A call can then be issued on getCogs() to obtain a list of SimpleCog objects.

3. Finally, call postIncident() with the list of SimpleCogs to post to, the TieIncident, and an optional list of AttachmentDescriptor objects.

Key systemsKey systems

 Certain calls through the Tie interface require an ID that references a particular incident. Since many clients will be using their own key systems, and some will not, we will provide a means to use either. When posting an incident, if the vendorUniqueId is not set, we will provide one as the return value of the postIncident() function call. This value can then be used in subsequent calls to reference that incident. For example, this value can be used to re-post an incident by setting vendorUniqueId to it.

AttachmentsAttachments

There is a correlation between the files specified in the array of AttachmentDescriptor and the attachments appended to the soap message to be dispatched. They correspond in order. The only mandatory member in this object is the fileName, which is just the filename portion without path information. For a post with attachments inThisMessage should be set to ‘true’ and deleteMe should be set to false. To update and version an incident that has attachments which will be deleted in the current version, set deleteMe to true and inThisMessage to false.

Authentication OverviewAuthentication OverviewAuthentication is handled through basic HTTP. An “Authorization: Basic”

header must accompany every request. Typically, the stub implementation will provide a convenient way to set it.

Incident Management System DMIS

Authentication ExamplesAuthentication Examples

Java example((TieSoapBindingStub)tie).setUsername(“9999/yourLoginHere");((TieSoapBindingStub)tie).setPassword(“easyPassword");

VB example

"ws" is our stub class in this example:

Dim myCred = New System.Net.NetworkCredential(“9999/yourLoginHere", “easyPassword")

Dim myCache = New System.Net.CredentialCache()myCache.Add(New Uri(ws.Url), "Basic", myCred)

ws.Credentials = myCachews.PreAuthenticate = True

 

Web Service SessionsWeb Service Sessions

Client session (cookie) management is not absolutely required. However, performance gains may be realized if implemented.

Java example

((TieSoapBindingStub)tie).setMaintainSession(true);

VB example

Dim cookieJar As CookieContainer ' Check to see if the cookies have already set upIf (ws.CookieContainer Is Nothing) Then

' Assign the CookieContainer to the proxy class.ws.CookieContainer = New CookieContainer()

End If

SOAP MESSAGE EXAMPLESOAP MESSAGE EXAMPLE

POST /axis/services/Tie HTTP/1.0Content-Type: text/xml; charset=utf-8Accept: application/soap+xml, application/dime, multipart/related, text/*User-Agent: Axis/1.1betaHost: computerNameCache-Control: no-cachePragma: no-cacheSOAPAction: ""Content-Length: 394qAuthorization: Basic Mi91c2VyMzpwd2QwMDM=Cookie: JSESSIONID=1EA6288365804756081F2E8338BAFCB6

<?xml version="1.0" encoding="UTF-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"

xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <ns1:getAllIncidents soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"

xmlns:ns1="http://dmi-services.org"/> </soapenv:Body></soapenv:Envelope>

9999/yourLoginHere:easyPassword

TypesTypes

TieIncident{

vendorUniqueId : stringname : stringnumber : stringdate : dateTimeversion : longhasAttachments : booleandescription : stringremarks : stringconfidence : IncidentConfidencecategory : IncidentCategorycog : SimpleCOGownerCog : SimpleCOGnotificationLevel :

IncidentNotificationLevelphase : IncidentPhaseseverity : IncidentSeveritystatus : IncidentStatustype : IncidentTypesites[] : IncidentSite

}

AttachmentDescriptor{

deleteMe : booleanfileName : stringinThisMessage : booleanmodifiedDate : longpathName : string

}IncidentSite{

address : AddresscriticalInfrastructure : intdescription : stringhasContacts : booleanid : longname : stringplottableOnMap : booleanpopulated : booleanprimarySite : booleanpropertyUse : stringtimeZoneId : int

}

SimpleCOG

{id : long

name: string

}

Types (cont)Types (cont)

Address

{addressLine1 : string

addressLine2 : string

city : string

cogId : long

crossStreets : string

empty : boolean

id : long

latitudeCoordinate : double

locatable : boolean

longitudeCoordinate : double

stateCode : USStateCodeType

zip : string

}

IncidentStatus

{Unknown | Worsening | Improving | Under Control | Closed

}

Enumerated types:

IncidentConfidenceIncidentCategoryIncidentNotificationLevelIncidentPhaseIncidentSeverityIncidentType

Method descriptionsMethod descriptions

ping()This method takes no parameters and has no return value. It is used simply to verify that the service is up and running and user is authenticated.

getAllIncidents()This method takes no parameters and returns a listing of core incident information returned in the form of an array of TieIncident.

Method descriptions (cont) Method descriptions (cont) getIncident()

This method takes a incident ID and version number and returns detailed incident information returned in the form of a TieIncident.

getAttachmentList()

This method takes an incident ID and version number and returns an array of AttachmentDescriptors.

Method descriptions (cont)  Method descriptions (cont)  getAttachments()

This method takes an incident ID, a version number, an array of file names and a boolean to specify whether attachments should be retrieved in DIME format or in conformance with the SOAP with Attachments specification (MIME).

 getCogs()

Returns a list of valid SimpleCOGs. May be used to get the possible COGs for a postIncident() operation.

Method descriptions (cont)  Method descriptions (cont)  postIncident()

This method takes three arguments: an instance of TieIncident, an array of SimpleCOGs and an array of AttachmentDescriptors. The array of SimpleCOGs should not contain duplicates or the source COG in the mailing list. The return value is the ID (either provided or a DMIS- generated ID as a result of the post) for the incident and could be used in subsequent calls.

Relating the “Reportable” API to EM – XML Standards

• Uses a simple wrapper around an information payload such that it can use DMI-S COG structure and distribution permissions to share information using DMIS “shared information space control” functionality.

• Provides sharing to and from DMI-S COGS interacting with data sources external to DMI-S.

• Can also provide “through service” from external source to external destination using the DMI-S controlled information space.

• Can be standards compliant, but is not standards enforcing or standards exclusive

• Could be used to increase the level of standards adoption.

The Reportable APIThe Reportable API

• Can be any file of any type: “standard” or not• If not standard:

– The file must “known” on both ends– Become, in effect, point-to-point processing through a

distributed interface

• Current and Potential Standards: – Standard document types (e.g., Word, Excel)– Process capable standards (e.g., EM-XML)– Others (e.g., shape files, other OASIS and non-

OASIS XML structures, etc.)

Reportable API: The AttachmentReportable API: The Attachment

• Build attachment payloads that validate against an adopted standard.

• Add elements to the DMI-S interoperability API that point to the standard represented in the payload.

• Payloads could then be pre-validated in DMI-S against the standard.

• Destinations could register their interest and/or ability to handle particular payloads.

• The DMI-S “information space” provides security and a consistently organized common sharing environment under DHS auspices that preserves local management needs.

Sharing EM-XML Standard DataSharing EM-XML Standard Data

• This is what could be done at the pure “reportable” level of interoperability.

• Standards (such as EM-EML) add even more convenience at increased interoperability levels as operational interfaces (its not just the data) find their way into the schemas.

• (But that is a future discussion).

Future Levels of InteroperabilityFuture Levels of Interoperability