1www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

Registration/Visualization of Volume Data in netCDF format

Ashraf Memon

2www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

Acknowledgement

• Chuck Meertens (UNAVCO)• Unidata• NSF Grant No. 0225673 (GEON)

3www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

Overview

• Introduction– Usecase

• Background– GEONgrid portal– netCDF data format– Opendap Server– Thredds Catalog– Thredds Data Server

• Architecture– Overview– Registration– Visualization

• Action Items and Timeline• Conclusion• Questions

4www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

Introduction

• UseCase– “Searching, Accessing, Integrating, and

Visualizing Distributed Data” – (Courtesy: Chuck Meertens)

– Specific Example:Searching, Accessing, Integrating and Visualizing distributed volume data stored in netCDF format and/or served on openDAP server. Performing these tasks through GEONgrid Portal.

5www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

Background

• GEONgrid PortalGEON Portal incorporates - Search, Contribute, Access, Integrate, Process, and Visualization of resources (data, tools, applications, and ontologies)

-Data Formats and Service types supported

-Shape

-RDBMS

-GeoTIFF

-EXCEL

-OGC WMS Services

-ASCII

-Web Services

-Arbitrary Data Objects (CUAHSI .zip file bundles)

6www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

Working in the GEON Portal:GEON Search with constraints

Sample search for:Resource Type - “Relational Database”Keyword - “GPS”

Search for ResourcesCurrent Types:ASCIICUAHSI DataGMT RasterGeoTiffRelational DatabaseShapefileToolWMS ServiceWeb ServiceFuture to handle 4-D Data:netCDFOPeNDAPSearch Constraints:Metadata Relation, Type, Subject, KeywordSpatial CoverageTemporal CoverageOntology/Concept Relation

(Courtesy: Chuck Meertens)

7www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

netCDF

• NetCDF (network Common Data Form) is an interface for array-oriented data access for representing scientific data and a library that provides an implementation of the interface.

• Together, the interface, library, and format support the creation, access, and sharing of scientific data.

8www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

netCDF

• A dataset has dimensions, variables, and attributes.

• Attributes can be global or apply to individual variables.

• There is a very limited set of low level data types.

9www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

OPeNDAP

• The Open Source Project for a Network Data Access Protocol (OPeNDAP)

• The DAP has been designed to be as general as possible without being constrained to a particular discipline or world view.

• The DAP is a discipline neutral data access protocol; it can be used in astronomy, medicine, earth science,…

10www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

OPeNDAP Server• Servers receive requests and provide responses via the

DAP.• Servers convert the data from the form in which they are

stored to the OPeNDAP data model.• Servers provide for subsetting of the data.

Data Data Data Data Data Data Data

Matlab

HDF4 JDBC

FreeFromFITS

CDF CEDAR

Data

netCDF

netCDF HDF4 Matlab

Data

DSP

DSP

Data

JGOFS

Tables SQL FITS CDFFlat

Binary CEDAR

Data

CODAR

Data

ESML

GeneralCODAR

11www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

OPeNDAP ClientsOPeNDAP Clients

netCDF C netCDF Java

IDVFerret GrADS VisAD ncBrowse Matlab ExcelIDL Access

MatlabClient

IDLClient

12www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

THREDDS Catalog

• THREDDS (Thematic Realtime Environmental Distributed Data Services)

• THREDDS is – a prototype system for cataloguing, browsing,

and accessing remote, retrospective, and other data types through a set of middleware technologies.

– is infrastructure needed for publishing and accessing scientific data in a similarly convenient fashion.

13www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

HTTP Tomcat Server

THREDDS Data Server

Datasets

Catalog.xml

hostname.edu

THREDDS ServerApplication

NetCDF-Javalibrary

•OPeNDAP

•HTTPServer

•WCSWCS

Application

IDV

OPeNDAP

WMS Server

14www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

THREDDS Data Server

• Integrates data access with THREDDS catalogs and services

• Tomcat/Servlet, 100% Java, single war file• Data input is netCDF Java 2.2 library• Data output:

– OPeNDAP – HTTP Server– OGC Web Coverage Server (gridded)

15www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

srb

ThreddsCatalog

http

ftpgridftpOGC*

*OGC: WMS/WCS/WFS

OPeNDAPOPeNDAPDataServers

-netCDF-Freeform

UNAVCO/GEONPoP Data Server1D/2D/3D/4D

-Tomography-GPS data/vectors-Earthquakes-Focal Mechanisms-Strain rate-Topography-Image maps-Geodynamics-Faults-Paleogeography

Plus IDV visualization“Bundles” (.xidv files)

Thredds-Catalog elements

Digital LibraryMetadata elements(via OAI or DLESEWebservice)

WMS Catalog *OGC=WMS/WFS/WCS

UNAVCO/GEON PoP Server DetailsData, Models, Catalogs, Metadata

Data Access

Dataset Catalogs and Metadata Access

16www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

UNAVCO/GEON Data Node

OPeNDAP ServerExample: Seismic Tomography in NetCDF format

WMS ServerExample: GPS Seamless Archive Postgres/PostGIS Database

THREDDS/OPeNDAPCatalog

WMS Catalog

HTML File

Other distributed data servers

Integrative Data Viewer

UNAVCO/ GEON Data Server and Visualization Access

Local File

GEON goal, by May, 2006: registration of netCDF files and OPeNDAP servers into the GEON Portal;

embedded GEONSearch within the IDV

17www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

GEON Portal

• Data/Model Registration •Data Search /Discovery

UNAVCO GEON PoP• Data/Metadata Server Node• Data Conversion Webservice

Users view

GEON Query &Data Preview

Data

Data access

Metadata Access

Thredds-Catalog-Digital LibraryMetadata

ProcessedDataUser Contributed

“Concepts”

RawData

Data Contribut

or

Volume Data(ascii grid or netCDF)

Data User

18www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

Archival

Data Registration ArchitectureOPeNDAP

SRB

HPSS GEONCatalog

Data Registration

Service

THREDDS Data Server

.nc

adn

fgdc

.nc.xidv

OPeNDAP URL THREDDS

OPeNDAP URL

UserMachine

Register OPeNDAP URL (URL, ADN, FGDC)

Register netCDF (URL, ADN, FGDC)

Register xidv (.xidv, ADN, FGDC)

WCS

19www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

Search, Access and Visualization

GEONCatalog

Data IntegrationAnd Visualization

Service

THREDDS Data Server

.nc

adnfgdc

.jnlp(.xidv)

OPeNDAP URL THREDDS

OPeNDAP URL

UserMachine

Webstart (.jnlp, ixidv)

Download, OPeNDAP, WCS, JPG

WCS

Tools Repository

Tools

OPeNDAP URL

wms

20www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

Action Items

• Installation and Understanding TDS (1 week)• GEON hosts TDS (WCS/OPeNDAP/THREDDS) Server of hosted

integrated data (1/2 week)• Data registration system for .nc file (3 Weeks)• Data Registration for OPeNDAP URL (3 Weeks)• netCDF download from GEON-hosted data (possible http link to

data) (Already Implemented)• GEON points to non hosted THREDDS server (2 weeks)• GEON hosts THREDDS catalog of integrated data (only selected

data, proxy server-could be non hosted service as well as GEON hosted) (2 weeks)

• GEON provides webstart to IDV with IDV catalog (points to hosted and non hosted data) (1 week)

• GEON provides webstart to IDV with IDV catalog a default view appropriate for type of data (1 week)

• GEON also provides a jpg preview from IDV scripting API (2 days)• GEON to host WCS through TDS (1 week)

21www.geongrid.orgCYBERINFRASTRUCTURE FOR THE GEOSCIENCES

Questions

• We are opening the phone lines for questions, please call in at 1.888.629.9921 and put in the passcode 5383118#