CUAHSI Hydrologic Information System: web services and related technologies HIS PIs: David Maidment...

CUAHSI Hydrologic Information

System:web services and related

technologiesHIS PIs: David Maidment Ilya Zaslavsky David Tarboton Michael Piasecki Jon Goodalland Rick Hooper (CUAHSI)

WaterML & Web Services team: David Valentine Tom Whitenack Tim Whiteaker Matt Rodriguez

www.cuahsi.org/his/

Test bed HISServers

Central HIS servers

ArcGIS

Matlab

IDL

MapWindow

Excel

Programming (Fortran, C, VB)

Desktop clients

Customizable web interface (DASH)

HTML - XMLW

SD

L - SO

AP

Hydrologic Information System Service Oriented Architecture

Modeling (OpenMI)

Global search (Hydroseek)

WaterOneFlow Web Services, WaterML

HIS LiteServers

External data providers

Deployment to test beds

Other popular online clients

ODM DataLoader

Streaming Data Loading

Ontology tagging (Hydrotagger)

WSDL and ODM registration

Data publishing

ODMTools

Server config tools

CUAHSI Member Institutions

112 Universities as of June 2007

HIS Goals

• Hydrologic Data Access System – better access to a large volume of high quality hydrologic data

• Support for Observatories – synthesizing hydrologic data for a region

• Advancement of Hydrologic Science – data modeling and advanced analysis

• Hydrologic Education – better data in the classroom, basin-focused teaching

Project co-PI in Phase 2

Collaborator in Phase I

CUAHSI HIS Partner Institutions

10 WATERS Testbed Sites

Windows 2003 Server4 GB Ram500 GB DiskQuad Core CPU

SQLServer 2005IIS (Internet Information Server)

DASH ArcGIS Server

Vis

ual S

tud

io 2

00

5

WaterOneFlow Web ServicesArcGIS 9.2

GIS Data Mxd Services

OD

Data

Loader

OD

M t

ools

COTS Software

HIS Applications Data

NWIS DV

NWIS IID

Your ODM

HISTeam

WATERSTestbed

WATERS Network Information System

SupercomputerCenters:NCSA,TACC

Domain Sciences:

Unidata, NCARLTER, GEON

Government:USGS, EPA,

NCDC, USDA

Industry:ESRI, Kisters,

OpenMI

HISTeam

WATERS Testbed

WATERS Network Information System

CUAHSI HIS

International Partners

CSIRO Land and Water ResourcesWater Resources Observations Network

(WRON)

European CommissionWater database design and model integration

(HarmonIT and OpenMI)

Digital WatershedHow can hydrologists integrate observed and

modeled data from various sources into a single description of the environment?

Rainfall & SnowWater quantity

and quality

Remote sensing

Water Data

Modeling Meteorology

Soil water

Water Data Web Sites

NWISWeb site output# agency_cd Agency Code# site_no USGS station number# dv_dt date of daily mean streamflow# dv_va daily mean streamflow value, in cubic-feet per-second# dv_cd daily mean streamflow value qualification code## Sites in this file include:# USGS 02087500 NEUSE RIVER NEAR CLAYTON, NC#agency_cd site_no dv_dt dv_va dv_cdUSGS 02087500 2003-09-01 1190USGS 02087500 2003-09-02 649USGS 02087500 2003-09-03 525USGS 02087500 2003-09-04 486USGS 02087500 2003-09-05 733USGS 02087500 2003-09-06 585USGS 02087500 2003-09-07 485USGS 02087500 2003-09-08 463USGS 02087500 2003-09-09 673USGS 02087500 2003-09-10 517USGS 02087500 2003-09-11 454

Time series of streamflow at a gaging station

USGS has committedto supporting CUAHSI’sGetValues function

Observation Stations

Ameriflux Towers (NASA & DOE) NOAA Automated Surface Observing System

USGS National Water Information System NOAA Climate Reference Network

Map for the US

Water Quality Measurement Sites in EPA Storet

Substantial variation in data availability from states

Data from Bora Beran, Drexel University

Water Quality Measurement Sites from Texas Commission for Environmental Quality (TCEQ)

Geographic Integration of Storet and TCEQ Data in HIS

Point Observations Information Model

• A data source operates an observation network• A network is a set of observation sites• A site is a point location where one or more variables are measured• A variable is a property describing the flow or quality of water• An observation series is an array of observations at a given site, for a given variable, with start time and end time• A value is an observation of a variable at a particular time• A qualifier is a symbol that provides additional information about the value

Data Source

Network

Sites

ObservationSeries

Values

{Value, Time, Qualifier}

USGS

Streamflow gages

Neuse River near Clayton, NC

Discharge, stage, start, end (Daily or instantaneous)

206 cfs, 13 August 2006

Return network information, and variable information within the network

Return site information, including a series catalog of variables measured at a site with their periods of record

Return time series of values

Observations CatalogSpecifies what variables are measured at each site, over what time interval,

and how many observations of each variable are available

Observations Data Model Schema (version 4.0)

Data Source and Network Sites Variables Values Metadata

Depth of snow pack

Streamflow

Landuse, Vegetation

Windspeed, Precipitation

Controlled Vocabulary Tables

e.g. mg/kg, cfs

e.g. depth

e.g. Non-detect,Estimated,

A site is a point location where one or more variables are measured

A data source operates an observation network A network is a set of observation sites

Metadata provide information about the context of the observation.A variable is a property describing the flow or quality of water

A value is an observation of a variable at a particular time

From Ernest To, David Maidment, CRWR

WaterOneFlow• Set of query functions • Returns data in WaterML

NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, USGS SNOTEL, ODM (multiple sites)

WaterML key elements

• Response Types

– SiteInfo

– Variables

– TimeSeries

• Key Elements– site– sourceInfo– seriesCatalog– variable– timeSeries

• values

– queryInfo

GetValues

GetVariableInfo

GetSiteInfo

variables

variablesResponse

variable

1

many

timeSeriesqueryInfo

criteria

timeSeriesResponse

variable

sourceInfo

queryURL

values

site queryInfo

criteria

sitesResponse

seriesCatalog siteInfo

queryURL

variable

series

variableTimeInterval

1

many

Structure of responses

More Information about WaterML…

next 20 slides…/we may skip them/

Or check the specification online at http://www.opengeospatial.org/standards/dp

DatasetInfoTypeSiteInfoType

Elements Defining Spatial Location

for observation sites for continuous surfaces

(other site information)

SourceInfoType

GeogLocationType

(other dataset information)

LatLonPointType

childelements

GeogLocationType

LatLonBoxType

LatLonPointType

SiteInfoResponseType

• Namespaces

• queryInfo

• site

Network

Sites

Variables

queryInfo example

userparameters

query URL

• Parameters sent to service

• URLs called (if external resource)

siteInfo

• Name

• Site Code

• Location

geoLocation

• geogLocation – geographic coordinates– LatLon point– LatLon box

• localSiteXY – projected coordinates

series

• variable – what is measured

• valueCount – how many measurements

• variableTimeInterval – when is it measured

TimePeriodType

variable

• variableCode – global identifier

• variableName

• units

Sites

Variables

Values

TimePeriodType

variableTimeInterval• TimePeriodType – date range (including

“last n days”

• TimeInstantType – single measurement

Compare with…

SiteInfo responsequeryInfo

site

name

code

location

seriesCatalog

variables

Series how many

when

TimePeriodType

VariablesResponseType

• variable – same as in series element

• Code, name, units Sites

Variables

Values

TimeSeriesResponseType

• queryInfo

• timeSeries– sourceInfo – “where”– variable – “what”– values

Sites

Variables

Values

sourceInfo• SiteInfoType

– Same as siteInfo element– code, name, location

• DataSetInfoType– For data continuous in space– LatLonPointType– LatLonBoxType

values

• Each time series value recorded in value element

• Timestamp, plus metadata for the value, recorded in element’s attributes

ISO Time

valuequalifier

value metadata examples

• qualifiers• censorCode (lt, gt, nc)• qualityControlLevel (Raw, QC’d, etc.)• methodID• offset

– offsetValue– offsetUnitsAbbreviation– offsetDescription– offsetUnitsCode

TimeSeries responsequeryInfo

location

variable

values

OGC Harmonization Best Practices

• WaterML text includes steps for harmonizing with GML/O&M

• Align spatial feature descriptions (e.g. using gml:Point, gml:Envelope)

• Align service signatures (getCapabilities)• Align terminology with O&M

More on OGC Interactions• WaterML is published as OGC Discussion paper• We are working with OGC O&M (Observations &

Measurements) authors to reconcile WaterML with OGC specs:– WOML proposal (Water Observations ML)

• Think of it as WFS + O&M type service (with GetSiteInfo proxied as OGC’s GetFeatureInfo request)

– The plan is to provide WOML-compliant services alongside WaterML 1.1

– There is an international group that Dave Valentine is coordinating, focused on developing a standard water information exchange schema (started at WaterML workshop in Canberra in September’07)

• As we are going with OGC through this harmonization, we shall be able to assist WQX with similar mappings (more below on mapping between WaterML and WQX)

Hydrologic Information Server

Microsoft SQLServer Relational Database

Observations Data & Catalogs Geospatial Data

GetSites

GetSiteInfo

GetVariables

GetVariableInfo

GetValues

DASH – data access system for hydrologyWaterOneFlow services

ArcGIS Server

SQL Server

ODMs and catalogs. All instancesexposed as ODM (i.e. have standard ODM tables or views: Sites, Variables, SeriesCatalog, etc.)

NWIS-IID

NWIS-DV

ASOS

STORET

TCEQ

BearRiver

. . .

Spatial store

Geodatabase or collection of shapefilesor both

NWIS-IID points

NWIS-DV points

ASOS points

STORET points

TCEQ points

BearRiver points

. . .

My new ODM

My new points

More databases

More synced layers

DASH Web Application

Background layers

(can be in the same or separate spatial store)

WOF services

Web services from a common template

NWIS-IID WS

NWIS-DV WS

ASOS WS

STORET WS

TCEQ WS

BearRiver WS

. . .My new WS

More WS fromODM-WS template

USGS

NCDC

EPA

TCEQ

Web Configuration fileStores information about registered networks

MXDStores information about layers

WSDLs

, web

serv

ice U

RLs Connection

strings

Layer info,

symbology, etc.

ODMDataLoader

2

6

5

3

1

4

WORKGROUPHISSERVERORGANIZATION

STEPS FORREGISTERINGOBSERVATIO

NDATA

New network registration steps

Using the ODM DataLoader or other, load your data into a blank ODM instance (this will create all ODM tables that HIS relies on)

Copy Web Services template to a new folder, edit the template web.config file to point to the new ODM, test to make sure the new service works as expected

Create a point layer (a feature class in GDB, or a shapefile) from the new ODM’s Sites table using the GetSitesTool.

Add the point layer to the MXD document, specify symbology, scale-dependent rendering, etc.

Add information about the new ODM, the associated web service, and the associated point layer, to HIS configuration file (see the first slide for the exact content)

Restart the HIS serviceRegister and test the new service at the HIS Central:

http://water.sdsc.edu/centralhis/6

5

4

3

2

1

7

Central HIS Data

Services

Catalog

Beyond Syntactic Uniformity: Semantic MediatorWhat we are doing now …..

NWIS

NAWQA

NARR

generic

request

GetValues

GetValues

GetValues

GetValues

GetValues

GetValuesGetValues

GetValues

GetValues ODM

Michael PiaseckiDrexel University

Hydroseekhttp://www.hydroseek.net

Supports search by location and type of data across multiple observation networks including NWIS, Storet, and university data

Semantic Tagging of Harvested Variables

CUAHSI HIS as a mediator across multiple agency and PI data

– Keeps identifiers for sites, variables, etc. across observation networks

– Manages and publishes controlled vocabularies, and provides vocabulary/ontology management and update tools

– Provides common structural definitions for data interchange

– Provides a sample protocol implementation– Governance framework: a consortium of

universities, MOUs with federal agencies, collaboration with key commercial partners, led by renowned hydrologists, and NSF support for core development and test beds

US Map of USGS Observations

Antarctica

Puerto Rico

Hawaii

Alaska

Different types of nutrients by decade: Available Data Total

Some physical properties by decade: Available Data Total

Measurement Counts by Decade

Measurement counts by Decade in the Western States

STORET DatacubeThe EPA STORET datacube contains 273K sites and 2.7M series. The EPA STORET datacube contains 273K sites and 2.7M series.

Florida is the source Florida is the source of about 25% of the of about 25% of the total records. total records.

Number of years of record by start decadeNumber of years of record by start decade

About 60% of the About 60% of the water quality records water quality records are short term are short term measurements (one measurements (one year or less in year or less in duration). The starting duration). The starting decade of the longer decade of the longer series are to the right.series are to the right.

93% of the series are93% of the series arewater quality data. water quality data.

Demo