Community for Data Integration Conference Call September 8, 2010 Dave Briar The USGS National Water Information System Design, Uses, and Future

Community for Data IntegrationConference Call

September 8, 2010Dave Briar

The USGSThe USGS

National Water Information SystemNational Water Information System

Design, Uses, and FutureDesign, Uses, and Future

Community for Data IntegrationConference Call

September 8, 2010Dave Briar

The USGSThe USGS

National Water Information ServicesNational Water Information Services

Design, Uses, and FutureDesign, Uses, and Future

slide 3

• core-NWIS / NWISWeb systemscore-NWIS / NWISWeb systems– Different functions and architecture – same philosophyDifferent functions and architecture – same philosophy

• Public and internal usesPublic and internal uses– Nontraditional and automated accessNontraditional and automated access

• Universal discovery and accessUniversal discovery and access– The promise of interoperable web servicesThe promise of interoperable web services

Topics:Topics:

slide 4

• Uses a data model optimized for its prime Uses a data model optimized for its prime functions -- the acquisition, processing, and long-functions -- the acquisition, processing, and long-term archiving of data.term archiving of data.

• Data processing (computation of derivatives, Data processing (computation of derivatives, review, qualification, and approvalreview, qualification, and approval) is highly interactive favoring local access to server.

• core-NWIS is developing its own internal services core-NWIS is developing its own internal services so that desktop-processing tools can be so that desktop-processing tools can be contracted for development by outside vendors.contracted for development by outside vendors.

core-NWIS:core-NWIS:

slide 5

• Uses a data model that is optimized for its prime Uses a data model that is optimized for its prime function – to provide highly available and efficient function – to provide highly available and efficient public access to USGS water data.public access to USGS water data.

• NWISWeb receives selected data sent from the 45 NWISWeb receives selected data sent from the 45 core-NWIS systems and creates a nationally core-NWIS systems and creates a nationally seamless view of those data for public access.seamless view of those data for public access.

• NWISWeb national architecture and design NWISWeb national architecture and design philosophy reflects history and USGS culture of philosophy reflects history and USGS culture of local control and “ownership” of their data.local control and “ownership” of their data.

NWISWeb:NWISWeb:

slide 6

-- NWIS database installation

slide 7

http://waterdata.usgs.gov/

Secure RepositorySecure RepositorySecure RepositorySecure Repository Public “View”Public “View”Public “View”Public “View”45 NWIS hosts

http://waterservices.usgs.gov/

slide 8

NNationalational AAggregateggregate DDatasetataset

Total sitesReal-time sites

Daily valuesGround-water levels

QW samplesQW values

Peaks

1,541,987 12,320324,567,200 8,437,274 4,836,150 88,893,776 699,583


Secure RepositorySecure RepositorySecure RepositorySecure Repository Public “View”Public “View”Public “View”Public “View”45 NWIS hosts

http://waterservices.usgs.gov/

slide 9

-- NWIS database installation

-- NatWeb Node with Firewall

slide 10

NWISWeb Data DeliveryNWISWeb Data Delivery

Menlo Park CA

Sioux Falls SD

Reston VA

Five redundant copies of all timeseries data for all sites

nationwide

Two copies of all data discrete / historical

for all sites nationwide

NAD1

NAD2

slide 11

core-NWISDatabase

LRGS

GOES

DOMSAT

slide 12

NWISDatabase

LRGS

GOES

DOMSAT

slide 13

NWISWeb Data TransferNWISWeb Data Transfer

NAD

Menlo Park CA

Sioux Falls SD

Reston VA

HUB1

NWIS Database

slide 14

NADHUB1

TS dataTS data TS data

All data

Sioux Falls SD

Reston VA

Menlo Park CA


slide 15

NAD

HUB2

HUB1

Sioux Falls SD

Reston VA

Menlo Park CA


slide 16

HUB2

TS data

TS data

TS data

All data

NAD

Sioux Falls SD

Reston VA

Menlo Park CA


slide 17

HUB2

TS data

TS data

TS data

All data

NAD

Sioux Falls SD

Reston VA

Menlo Park CA


slide 18

NWIS Database is single point of failureNWIS Database is single point of failure

NADHUB1

NWISDatabase

slide 19

NWISDatabase

LRGS

GOES

DOMSAT

slide 20

NADHUB1NWIS-RT in

“Hot Standby”

NWIS – NWIS-RT BackupNWIS – NWIS-RT Backup

Replication

NWISDatabase

DOMSAT

slide 21

NADHUB1NWIS-RT Serving WSC Data

NWIS – NWIS-RT BackupNWIS – NWIS-RT BackupDOMSAT

slide 22

Louisiana Data -- Hurricane KatrinaLouisiana Data -- Hurricane Katrina

slide 23

Louisiana Data -- Hurricane KatrinaLouisiana Data -- Hurricane Katrina

slide 24

2004 Tropical Storm Tracks2004 Tropical Storm Tracks

slide 25


• Public and internal usesPublic and internal uses

– Nontraditional and automated accessNontraditional and automated access


Topics:Topics:

slide 26

Testimonial:

Principal of a School: "Our school floods quite frequently. I have calibrated our response and reactions to this great data source. I deeply appreciate the availability, as do the 300+ families who benefit from the benefit our school gains from the data."

Principal of a School: "Our school floods quite frequently. I have calibrated our response and reactions to this great data source. I deeply appreciate the availability, as do the 300+ families who benefit from the benefit our school gains from the data."

slide 27

Testimonial:

"I live on lakefront property at Canyon Lake, Texas. Your website was like a bright light in a dark cave during the flood of 2002. You gave us the essential and critical information needed to make major decisions which would effect my family and home. Thank you for providing a portal to the big picture, enabling us to make timely decisions with up to date information. Making accurate decisions is difficult when threatened with flooding, but it would have been impossible without your website. Keep up the good work."

"I live on lakefront property at Canyon Lake, Texas. Your website was like a bright light in a dark cave during the flood of 2002. You gave us the essential and critical information needed to make major decisions which would effect my family and home. Thank you for providing a portal to the big picture, enabling us to make timely decisions with up to date information. Making accurate decisions is difficult when threatened with flooding, but it would have been impossible without your website. Keep up the good work."

slide 28

NWISWeb Usage, in millions of successful requests per month

0

10

20

30

40

50

60

2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

slide 29

Public and Internal UsesPublic and Internal Uses

• NWISWeb allows 4 levels of accessNWISWeb allows 4 levels of access– Public, cooperator, USGS, and local Public, cooperator, USGS, and local

office onlyoffice only

slide 30




• NWISWeb uses http GETNWISWeb uses http GET– All pages are “bookmarkable”All pages are “bookmarkable”– All data are retrievable in ASCII tab-All data are retrievable in ASCII tab-

delimited (RDB) formatdelimited (RDB) format

slide 31

Where What Output formatWhere What Output format

slide 32




• NWISWeb uses http GETNWISWeb uses http GET– All pages are “bookmarkable”All pages are “bookmarkable”– All data are retrievable in ASCII tab-All data are retrievable in ASCII tab-

delimited (RDB) formatdelimited (RDB) format– ~50+% of total hits are from automated ~50+% of total hits are from automated

programs (not a user at a browser)programs (not a user at a browser)

slide 33

slide 34

slide 35

slide 36

slide 37

slide 38

slide 39

slide 40

slide 41

slide 42


• Public and internal usesPublic and internal uses

– Nontraditional and automated accessNontraditional and automated access


Topics:Topics:

slide 43

Web Service DefinitionWeb Service Definition

• A Web service is defined by the W3C as:A Web service is defined by the W3C as: “ “A software system designed to support interoperable A software system designed to support interoperable

machine-to-machine interaction over a network“. machine-to-machine interaction over a network“.

• Web services are frequently just Web APIs that can be Web services are frequently just Web APIs that can be accessed over a network, such as the Internet, and accessed over a network, such as the Internet, and executed on a remote system hosting the requested executed on a remote system hosting the requested services.services.

• By this definition, NWISWeb has been a web service from By this definition, NWISWeb has been a web service from the beginning, however, discovery by and interoperability the beginning, however, discovery by and interoperability with other services/portals has been limited.with other services/portals has been limited.

slide 44

NWISWeb Data Discovery ServicesNWISWeb Data Discovery Services

• EPA “Window to my environment”EPA “Window to my environment”– March 2001March 2001– Site location and type (no catalog)Site location and type (no catalog)

• EDC ARC IMS map feature serviceEDC ARC IMS map feature service– March 2001 March 2001 – Expanded Site Info (basic local catalog)Expanded Site Info (basic local catalog)

• CUAHSI Hydrologic Information System (HIS)CUAHSI Hydrologic Information System (HIS)– January 2007 January 2007 – Expanded site Information including data period-of-Expanded site Information including data period-of-

record (extensive catalog)record (extensive catalog)

slide 45

Question: How much “discovery” information Question: How much “discovery” information should be cached at a Portal catalog verses should be cached at a Portal catalog verses supplied dynamically via its own service?supplied dynamically via its own service?

• No portal catalog: No portal catalog: – EPA “Window to my environment”EPA “Window to my environment”– Site location and type Site location and type

• Basic portal catalog: Basic portal catalog: – EDC IMS map feature serviceEDC IMS map feature service

– Expanded Site InfoExpanded Site Info

• Extensive portal catalog: Extensive portal catalog: – CUAHSI HISCUAHSI HIS– Expanded site Information including data period-Expanded site Information including data period-

of-recordof-record

slide 46

EPA “Window to my Environment”EPA “Window to my Environment”

• EPA WME does not maintain any USGS site information EPA WME does not maintain any USGS site information (catalog) local to the WME portal but uses NWISWeb as a (catalog) local to the WME portal but uses NWISWeb as a service to populate that information dynamically when a user service to populate that information dynamically when a user requests it.requests it.

• EPA WME submits a Lat/Long and Site-type request to EPA WME submits a Lat/Long and Site-type request to NWISWeb and then plots the site locations on-the-fly with links NWISWeb and then plots the site locations on-the-fly with links back to waterdata.usgs.gov to get the data.back to waterdata.usgs.gov to get the data.

slide 47

EPA “Window to my Environment”EPA “Window to my Environment”

• EPA WME does not maintain any USGS site information EPA WME does not maintain any USGS site information (catalog) local to the WME portal but uses NWISWeb as a (catalog) local to the WME portal but uses NWISWeb as a service to populate that information dynamically when a user service to populate that information dynamically when a user requests it.requests it.

• EPA WME submits a Lat/Long and Site-type request to EPA WME submits a Lat/Long and Site-type request to NWISWeb and then plots the site locations on-the-fly with links NWISWeb and then plots the site locations on-the-fly with links back to waterdata.usgs.gov to get the data.back to waterdata.usgs.gov to get the data.

• Advantages: USGS site information is always up-to-date and Advantages: USGS site information is always up-to-date and no operational overhead is incurred by having to maintain a no operational overhead is incurred by having to maintain a local copy of a catalog.local copy of a catalog.

• Disadvantages: Can only “Ask questions” of NWISWeb that Disadvantages: Can only “Ask questions” of NWISWeb that NWISWeb supports/understands.NWISWeb supports/understands.

slide 48

slide 49

CUAHSI HIS PortalCUAHSI HIS Portal

• CUAHSI HIS portal maintains an extensive catalog local to the CUAHSI HIS portal maintains an extensive catalog local to the portal detailing the location of measurement sites from many portal detailing the location of measurement sites from many academic and Federal agencies and what data are available at academic and Federal agencies and what data are available at each site allowing “data discovery”.each site allowing “data discovery”.

• HIS portal then provides web services for “data delivery” from HIS portal then provides web services for “data delivery” from the source entity/agency.the source entity/agency.

slide 50


• CUAHSI HIS portal maintains an extensive catalog local to the CUAHSI HIS portal maintains an extensive catalog local to the portal detailing the location of measurement sites from many portal detailing the location of measurement sites from many academic and Federal agencies and what data are available at academic and Federal agencies and what data are available at each site allowing “data discovery”.each site allowing “data discovery”.

• HIS portal then provides web services for “data delivery” from HIS portal then provides web services for “data delivery” from the source entity/agency.the source entity/agency.

• Advantages: An extensive catalog local to the portal Advantages: An extensive catalog local to the portal potentially allows the portal to offer more sophisticated data potentially allows the portal to offer more sophisticated data discovery options if those options are not available directly discovery options if those options are not available directly from the source system.from the source system.

• Disadvantages: Once the catalog is cached at the portal it is Disadvantages: Once the catalog is cached at the portal it is instantly out-of-date and there is a continual maintenance instantly out-of-date and there is a continual maintenance overhead in refreshing the catalog information.overhead in refreshing the catalog information.

slide 51

CUAHSI initiativesCUAHSI initiatives

slide 52


NWISWeb dump files “currently” supplied to populate NWISWeb dump files “currently” supplied to populate the CUAHSI portal catalog:the CUAHSI portal catalog:

618248 Apr 14 18:29 USGS_DV_DD_POR_list.gz618248 Apr 14 18:29 USGS_DV_DD_POR_list.gz

9127817 Apr 14 18:29 USGS_POR_list_0000-00-00_removed.gz9127817 Apr 14 18:29 USGS_POR_list_0000-00-00_removed.gz

10337889 Apr 14 18:29 USGS_POR_list.gz10337889 Apr 14 18:29 USGS_POR_list.gz

66484565 Apr 14 18:30 USGS_QW_PARAM_POR_list.gz66484565 Apr 14 18:30 USGS_QW_PARAM_POR_list.gz

40765012 Apr 14 18:29 USGS_QW_PGROUP_POR_list.gz40765012 Apr 14 18:29 USGS_QW_PGROUP_POR_list.gz

71660218 Apr 14 18:29 USGS_SITEFILE_list.gz71660218 Apr 14 18:29 USGS_SITEFILE_list.gz

109992 Apr 14 18:29 USGS_UV_DD_list.gz109992 Apr 14 18:29 USGS_UV_DD_list.gz

slide 53

NWISWeb Site Information ServiceNWISWeb Site Information Service

• The proposed NWISWeb “Site Information” service for The proposed NWISWeb “Site Information” service for FY11 will return this same site and data period-of-record FY11 will return this same site and data period-of-record information in XML directly -- either to populate/ refresh a information in XML directly -- either to populate/ refresh a portal catalog or dynamically respond to a portal user’s portal catalog or dynamically respond to a portal user’s request.request.

• The NWISWeb Site Information service will also output The NWISWeb Site Information service will also output KML and so can be used directly by the public (and USGS) KML and so can be used directly by the public (and USGS) to support mapping applications.to support mapping applications.

• The “goal” is to make the Site Information service OGC The “goal” is to make the Site Information service OGC “Web Feature Service” as compliant as possible.“Web Feature Service” as compliant as possible.

slide 54

slide 55

Data Integration / InteroperabilityData Integration / Interoperability

• The interoperability of services depends both on the way The interoperability of services depends both on the way specific services are called as well as the format / schema of specific services are called as well as the format / schema of the information returned. the information returned.

• The Open Geospatial Consortium (OGC) has become the lead The Open Geospatial Consortium (OGC) has become the lead international organization for the development of geospatial international organization for the development of geospatial standards and data-sharing protocols.standards and data-sharing protocols.

• The recently released (Aug 2010) NWISWeb Instantaneous The recently released (Aug 2010) NWISWeb Instantaneous Values service returns data in the CUAHSI developed Values service returns data in the CUAHSI developed WaterML1.1 schema but does not fully comply with the CUAHSI WaterML1.1 schema but does not fully comply with the CUAHSI “Water One Flow” services naming conventions.“Water One Flow” services naming conventions.

slide 56

http://waterservices.usgs.gov/WOF/InstantaneousValues?location=06018500&variable=00060&period=P1D

slide 57


• The interoperability of services depends both on the way The interoperability of services depends both on the way specific services are called as well as the format / schema of specific services are called as well as the format / schema of the information returned. the information returned.

• The Open Geospatial Consortium (OGC) has become the lead The Open Geospatial Consortium (OGC) has become the lead international organization for the development of geospatial international organization for the development of geospatial standards and data-sharing protocols.standards and data-sharing protocols.

• The recently released (Aug 2010) NWISWeb Instantaneous The recently released (Aug 2010) NWISWeb Instantaneous Values service returns data in the CUAHSI developed Values service returns data in the CUAHSI developed WaterML1.1 schema but does not fully comply with the CUAHSI WaterML1.1 schema but does not fully comply with the CUAHSI “Water One Flow” services naming conventions.“Water One Flow” services naming conventions.

• The NWISWeb IV service is NOT presently compliant with the The NWISWeb IV service is NOT presently compliant with the OGC “Sensor Observation Service” protocol but that is a goal OGC “Sensor Observation Service” protocol but that is a goal over time.over time.

slide 58


• NWISWeb services will move towards supporting OGC NWISWeb services will move towards supporting OGC protocols (WaterML 2, WFS, SOS) but will likely need to also protocols (WaterML 2, WFS, SOS) but will likely need to also support other protocols as well.support other protocols as well.

• The Center for Data Integration and Analysis headed by Nate The Center for Data Integration and Analysis headed by Nate Booth is taking the lead on collaboration with OGC including Booth is taking the lead on collaboration with OGC including working on OGC “Interoperability experiments” with working on OGC “Interoperability experiments” with international partners and actively contributing to standards international partners and actively contributing to standards and protocol development.and protocol development.

• CIDA also developed the USGS/EPA Water Quality web CIDA also developed the USGS/EPA Water Quality web services portal and is heavily involved in portal development services portal and is heavily involved in portal development efforts with other agencies as well.efforts with other agencies as well.

slide 59

slide 60

Questions?Questions?


http://waterservices.usgs.gov/http://waterservices.usgs.gov/

[email protected]@usgs.gov

slide 61

slide 62

History of other NWIS “Web services”History of other NWIS “Web services”

• Interim USGS Web ServicesInterim USGS Web Services– August 2009August 2009

– All IV timeseries (public data only)All IV timeseries (public data only)

– Site information to service NWIS mappersSite information to service NWIS mappers

• Production USGS Instantaneous-values serviceProduction USGS Instantaneous-values service– August 2010August 2010

– All IV timeseries All IV timeseries

– WaterML 1.1, RDB, and JSON output formatsWaterML 1.1, RDB, and JSON output formats

• Production USGS Site-information serviceProduction USGS Site-information service– November(?) 2010November(?) 2010

– Service NWIS mappers and public KML/XML Service NWIS mappers and public KML/XML

– Service/refresh/update portal catalogsService/refresh/update portal catalogs

– OGC Web Feature ServiceOGC Web Feature Service

slide 63

Documents

Community for Data Integration Conference Call September 8, 2010 Dave Briar The USGS National Water Information System Design, Uses, and Future