Work Group Meeting onHTAP-Relevant IT Techniques, Tools and Philosophies:

DataFed Experience and Perspectives

Rudolf B. HusarCAPITA, Washington University, St. Louis, MO

Ispra, JRC, March 14. 2004

• Data handling approach• Software tools• Participant involvement• Data dissemination• Integration problems• Summary

Policy Guidance for HTAP IT

Terry Keating, HTAP Task Force Co-Chair:

• Transparency of the HTAP Process– Acceptance of the Tech Report will depend on openness

• Inclusiveness and Ease of Participation– Facilitate participation by smaller contributors

Current Air Quality Information ‘Ecosystem’

AQ info is distributed over many ‘dimensions’: Geography, Content, Agency, Program…AQ info content includes: emissions, ambient & satellite data and model outputsInfo is provided and consumed by different agencies, (NASA, NOAA, EPA…)Providers have different access protocols, formats, and information usage conditions

Lack of Interoperability

Poor data & model utilization

Less informed decision making

GEOSS Info Flow ArchitectureA General Framework Accepted by Members

Model

Model

Data

Data

Federated Network for Air Quality Data and Processing ServicesProject Team:

Software Architecture: R. HusarSoftware Implementation: K. Höijärvi

Data and Applications: S. Falke, R. Husar

Data Handling Approach

• The challenge is to design a general supportive infrastructure• Simply connecting the relevant provides and users for each info product is messy

Federated Data System for Air Quality

• The info system infrastructure needs to facilitate the creation of info products

AQ Compliance

Nowcast/Forecast

Status & Trends

Find Data Gaps

ID New Problems

………

Info Needs

Reports

• Providers supply the ‘raw material’ (data and models) for ‘refined’ info products

EmissionSurface Satellite

Model

Single Datasets

Providers

Wrappers

Where?

What?

When?

Federate Data

Structuring

• Structuring the heterogeneous data into where-when-what ‘cubes’ simplifies the mess

Slice & Dice

Explore Data

Viewers

• The ‘cubed’ data can be accessed and explored by slicing-dicing tools

Programs

Integrate

Understand

• More elaborate data integration and fusion can be done by web service chaining

Non-intrusive Linking & Mediation Data UsersData Providers

Data Handling Approach:

DataFed

 Approach: Mediation Between Users and Data ProvidersDataFed assumes autonomous data management (a la Internet)Non-intrusive third-party data wrapping for unified web service (WS) accessEnd-user programming of applications through chaining of WS components 

ApplicationsBuilding browsers and analysis tools for distributed monitoring data    Serve as data and service resources for user programs (science, GIS tools)Support application projects, e.g. FASTNET, Exceptional Event Rule

Typical DataFed AQ Analysis Tools

Consoles: Data from diverse sources are displayed to create a rich context for exploration and analysis

CATT: Combined Aerosol Trajectory Tool for the browsing backtrajectories for specified chemical conditions

Viewer: General purpose spatio-temporal data browser and view editor applicable for all DataFed datasets

Quebec SmokeJuly, 7, 2002

SeaWiFS Satellite

Aerosol Chemical

Air Trajectory

Map Boarder

Web Service Composition

Software Tools:

Demo: Networked Data Access, Processing and Fusion

AeroCOM – VIEWS SO4

• http://webapps.datafed.net/dvoy_services/datafed.aspx?page=KYU_VIEWS• http://webapps.datafed.net/dvoy_services/datafed.aspx?page=KYU_V• http://webapps.datafed.net/dvoy_services/datafed.aspx?page=VIEWS_KYU

Networking

VisualizationGoal: 4D, User-selectable layers

Image below completed in 1998

Satellite Data Layers:

Land Reflectance (SeaWiFS)

Fire Pixels (ASTR Night)

High Cloud (GOES, Meteosat, GMS2)

Aerosol AOT (AVHRR)

4D Dynamic Visualization Demonstrate interactions, allow exploration

‘Google Earth’ for Earth Science is now possible

Partners

• NASA

• NOAA

• EPA

• USGS

• DOE

• NSF

• Industry…

Earth Science Information Partners

Air Quality Cluster

1. Serve as facilitator to Earth Science information community. 2. Promote efficient flow of ES data from collection to end-use. 3. Improve quality and usability of ES data and info systems 4. Expand the use of Earth science information

Participant Involvement

Data Dissemination & Use – Service Based

• Provide Catalog Services (Publish, Find data)• Allow Data Access (‘Bind’) – Use International Standards • Add tools for exploration and analysis

Near Real Time Data IntegrationDelayed Data Integration

Surface Air Quality AIRNOW O3, PM25 ASOS_STI Visibility, 300 sitesMETAR Visibility, 1200 sitesVIEWS_OL 40+ Aerosol Parameters

SatelliteMODIS_AOT AOT, Idea ProjectGASP Reflectance, AOTTOMS Absorption Indx, Refl.SEAW_US Reflectance, AOT

Model OutputNAAPS Dust, Smoke, Sulfate, AOTWRF Sulfate

Fire DataHMS_Fire Fire PixelsMODIS_Fire Fire Pixels

Surface MeteorologyRADAR NEXTRADSURF_MET Temp, Dewp, Humidity…SURF_WIND Wind vectorsATAD Trajectory, VIEWS locs.

WCS - Interoperable Data Access Service

netCDF – Machine independent encoding

ncML – XML data description

CF – Naming, structure convention

OGC Web Coverage Service - Interoperable Data Access

Query Data Syntax + Semantics

Coverage (parameter)

BBOX

Time Range

netCDF Example Profile

• SERVICE=wcs ‘service• REQUEST=GetCoverage,VER=1.0 ‘service method• COVERAGE=AIRNOW.pmfine ‘what• CRS=EPSG:4326 ‘projection• BBOX=-125,22,-61,51,0,0 ‘where• TIME=2005-06-6T15:00:00Z ‘when• FORMAT=netCDF ‘return format

GALEON Interoperability ExperimentGeo-interface for Atmosphere, Land, Earth, and Ocean netCDF

Unify Earth Science & GIS Data Flows

Strong European Participation

IT – S. Nativi, L. Bigagli

UK – Andrew Wolf

DE – Peter Bauman)

B. Domenico

B. Domenico

GALEON UNIDATA

U Florence/CNR-IMMA WCS Server

OGC WCS Demonstration: THREDDS_GFS 4Dim Dataset

Lat/Lon Box Elev Range Time RangeMap: BBOX=-180,-90,180,90, 1350,1350& TIME=2005-12-06/2005-12-06/PT3HTime: BBOX=-34,49.05,-34,49.05, 1350,1350& TIME=2005-12-05/2005-12-08/PT3HElev: BBOX=-34,49.05,-34,49.05, 0,18000 & TIME=2005-12-06/2005-12-06/PT3H

The form of the WCS query is the same for all slices through the data cube (views) The only difference in the views is the thickness of the slices in each dimension Return grid is in multiple formats (NetCDF, CSV, GML, PNG, … )

Map View Services WCS Query

Time View Services WCS Query

Elevation View Services WCS Query

Summary

• Current systems data & model analysis are heterogeneous

• Standardization is a key need for agile IT systems

• Non-intrusive mediators can achieve virtual standardization

• Technologies are currently available for dynamic NETWORKING

We eager to share our networked data, tools and methods

OGC WCS Demonstration: Grid, Image, Station Data Types

Coverage=THEEDDS.T& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDFCoverage=SURF.Bext& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDF-tableCoverage=SEAW.Refl& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=GeoTIFFCOVERAGE=sst& BBOX=-126,24,-65,52,0,0 &TIME=2001-01-01,2001-01-01&FORMAT=NetCDF

UNIDATA – THREDDS/GALEON WCS

DataFed GALEON WCS

U Florence, It GALEON WCS

DataFed GALEON WCS

Grid

Grid

Image

Station

Services WCS Query

Services WCS Query

Services

WCS Query

Services

WCS Query

Single Data Model for All AQ Data

Most Views are slices through a cube of data organized by lat, lon, altitude, and time (X,Y,Z,T)

Multidimensional Data Cube

OGC Web Coverage Service (WCS) Specification

• HTTP GET/POST based interfaces• Services have XML service descriptions (“Capabilities”, “Description”)• Filter parameters allow selection of subsets of source data• Output formats advertised by each service instance

OGC WCS getCoverage SchemaSuitable for wrapping with SOAP envelope, WSDL access, loose coupling

WCS is for "coverages" – information representing space-time-varying phenomena

WCS describes, requests and delivers coverages in spatio-temporal domain

WCS version 1.1 is limited to grids/"simple” coverages with homogeneous range sets

AQ Monitoring Network Data Storage and Delivery through OGC Protocols

Relational Data Model

Star Schema

WMS

WCS SOS

Sen

sorMLW

FS

Observations

Sta

tion

In

fo.

Para

m/S

en

sor/M

eth

od

Data View ServicesW

MS

Stations Par-Meth

Observations

SO

S

WCS Data Access

Technology Support for ‘Integrated Solutions’

Air Quality Information System

through

Data Access through Adapters

DataFed SOAP,HTTP Get

OGC WCS HTTP Get, Post

OGC WMS HTTP Get

Station-Point SQL Server,

Files…

Sequence Image, file

nDim Grid OpenDAP NetCDF,

…

Other Traject., Event, Pic

Sources Diverse formats

Many data models

Data Wrapper Data into geo-cubes

Queries to views

Virtual Data Cube Global geo-cube data model Makes queries data-neutral

Others? e.g.

OpenDAP

Output Protocol dependent

User specified

GeoTable CSV,XLS,GML

GeoGrid GML,NetCDF..

GeoImage GeoTIFF,

PNG..

Other MS Dataset..

Query Adapter Maps query to protocol

User selects protocols