Effective discovery of geospatial data: a geospatial catalogue perspective

10/31/2007

Effective discovery of geospatial data: a geospatial catalogue perspective

Dr. Yuqi [email protected]

Research Assistant ProfessorCenter for Spatial Information Science and Systems

George Mason University

Washington/Northern Virginia Chapter of IEEE/GRSS Technical Meeting

Wednesday, October 31, 2007

Page 2

CSISS Center for Spatial Information Science and Systems

Contents

• Geospatial data discovery problems• Geospatial data discovery systems

– System architectures– Referenced Metadata standards– Referenced Catalogue Service standard

• Geospatial Catalogue Federation– Case study: GMU CSISS CFS product– Main Challenges– Proposed federation strategies– Product system– Discussion

• GMU CSISS CSW/CFS Applications • Summary

Page 3


Background

• Large volume of geospatial data– has been accumulated over the last several decades

through mapping, survey and observation

• Petabyte level– NASA EOSDIS project is expected to archive one

petabyte per year of raw data that are distributed among data centers.

– On November 20, 2003, the NASA Land Processes Distributed Active Archive Center (LP DAAC) data archive holdings crossed the one petabyte threshold in volume*.

– 1 petabyte = 1*10**15 bytes= 8*10**6 Second * 1 Gb/s (~ 92.59 days)= 8*10**7 Seconds * 100 Mb/s (~925.9 days)

*http://edcdaac.usgs.gov/petabyte.asp

Page 4


Significant Problem and Question

• Problem– Large volume of geospatial data has to be

maintained in few data centers, while these data are highly needed in all research carried out by research staff, professors and students in every college, university and government agencies.

geospatial data

• Question– How can users be helped in evaluating the

fitness of a particular data set, among hundreds of collections and millions of granules, for user for their specific decision or assessment?

End user

Page 5


Geospatial Data Discovery Mechanism

• Overview

– Step 1:• Organizing textual information about

the identification, the extent, the quality, the spatial and temporal schema, spatial reference, and distribution of every piece of data set

• Metadata (data about data)

geospatial data

geospatial metadata

discoveryinterface

End user

– Step 2:• Providing catalogue discovery

interface for this metadata information to end users

– Step 3:• Enabling direct data download or

customization through online software modules, or “services”

Page 6


Geospatial Metadata

• The metadata required for effective data management varies with the type of data and context of use.

– Standards + Profiles

geospatial data

geospatial metadata

discoveryinterface

End user

• Standards:– ISO

• ISO 15836:2003– Dublin Core metadata element set– Stage code: 60.60 (2003-11-26)

• ISO 19115:2003– Geographic information -- Metadata– Stage code: 60.60 (2003-05-08)

• ISO 19115:2003/Cor 1:2006– Stage code: 60.60 (2006-07-05)

• ISO 19115-2– Geographic information -- Metadata -- Part 2:

Extensions for imagery and gridded data– Under development– Stage code: 40.00 (2007-10-25)

• ISO 19119:2005– Geographic information -- Services– Stage code: 60.60 (2005-02-10)

• ISO 19139:2007– Geographic information -- Metadata -- XML

schema implementation– Stage code: 60.60 (2007-04-17)

Page 7


Geospatial Metadata (Cont.)

• The metadata required for effective data management varies with the type of data and context of use.

– Standards + Profiles

geospatial data

geospatial metadata

discoveryinterface

End user

• Standards:– US

• FGDC-STD-001-1998– Content Standard for Digital Geospatial

Metadata• FGDC-STD-012-2002

– Content Standard for Digital Geospatial Metadata: Extensions for Remote Sensing Metadata

– NASA• ECS Science Metadata

Page 8


Geospatial Metadata Discovery Interface

geospatial data

geospatial metadata

discoveryinterface

End user• The discovery interface varies with the type/structure of the underlying metadata and context of use.

• From the user’s point of view:– Simple web page navigation with no search

functionality• E.g. THREDDS

Page 9


Geospatial Metadata Discovery Interface (Cont.)

• The discovery interface varies with the type/structure of the underlying metadata and context of use.

geospatial data

geospatial metadata

discoveryinterface

End user



– Web page navigation with limited search functionalities

• E.g. NASA GCMD

Page 10




geospatial data

geospatial metadata

discoveryinterface

End user




• E.g. NASA GCMD– Web-based GUI with enhanced search functionalities,

no public API interface• E.g. EOS Data Gateway (EDG)

Page 11


Geospatial Data Discovery Process through EOS Data Gateway (EDG) System

Page 12




geospatial data

geospatial metadata

discoveryinterface

End user






LP DAAC GES DISC

Page 13




geospatial data

geospatial metadata

discoveryinterface

End user






– Web-based GUI with enhanced search functionalities, with proprietary API interface

• E.g. NASA ECHO– IIMSAQL Query Language

ECHO Service Core

GES DISCLP DAAC

GMU CSISS ECHO OGC Wrapper

Page 14


geospatial data

geospatial metadata

discoveryinterface

End user

15 Terabytes Images

GMU CSISS OGC Catalogue Service Core

ebRIM Wrapper ISO WrapperOGC Core

Data DownloadData Download GeoBrain Online GeoBrain Online Analysis System (GeOnAS)Analysis System (GeOnAS)








– Web-based GUI with enhanced search functionalities, with proprietary API interface

• E.g. NASA ECHO– IIMSAQL Query Language

– Web-based GUI with enhanced search functionalities, with open API interface

• E.g. GMU CSISS/LAITS CSW

Page 15


GMU CSISS/LAITS CSW

- Designed and Developed from Aug. 2003- Support OGC CSW 2.0.1 and 2.0.2

Page 16


Geospatial Catalogue Service Standard

• OGC Catalogue Service is the only available standard

– specifies the interfaces between clients and catalogue services through the presentation of abstract and implementation-specific models.

– Catalogue Service and its clients

• OGC’s perspective:– Catalogue Service supports the ability to publish

and search collections of descriptive information (metadata) for data, services, and related information objects.

– Metadata in catalogues represent resource characteristics that can be queried and presented for evaluation and further processing by both humans and software.

– Catalogue services are required to support the discovery of and binding to registered information resources within an information community.

geospatial data

geospatial metadata

discoveryinterface

End user

Catalogue Service

Catalogue Service Client

Page 17


Geospatial Catalogue Service Standard (Cont.)

http://www.opengeospatial.org/standards/cat

Page 18


Current Status of Geospatial Catalogue System

geospatial data

geospatial metadata

discoveryinterface

End user

GMU CSISS OGC Catalogue Core

ebRIM ISOOGC Core

ECHO Service Core


Page 19


New Problems and Questions

geospatial data

geospatial metadata

discoveryinterface

End user


ebRIM ISOOGC Core

ECHO Service Core


Page 20



geospatial data

geospatial metadata

discoveryinterface

End user


ebRIM ISOOGC Core

ECHO Service Core


Page 21



geospatial data

geospatial metadata

discoveryinterface

End user


ebRIM ISOOGC Core

ECHO Service Core


Page 22



geospatial data

geospatial metadata

discoveryinterface

End user


ebRIM ISOOGC Core

ECHO Service Core


Page 23



• Different agencies have developed their own geospatial catalogues to facilitate discovery, access, and sharing of large volumes of geospatial data, either observed satellite images or simulation data.

• These geospatial catalogues are becoming accessible online through their query interfaces.

• Scientists who conduct multi-disciplinary research may need to search multiple catalogues in order to find the data they need. Such work is very time-consuming and tedious, especially when the catalogues may use different metadata models and catalog interface protocols.

• It is very desirable if those catalogues can be integrated into a catalogue federation, which will present a well-known metadata model and interface protocol to users and hide the complexity and diversity of the affiliated catalogues behind the interface.

• With the federation, users only need to work with the federated catalogue to find the data they need instead of working with catalogues individually.

• Catalogue federation service - integrating multiple legacy catalogues to facilitate distributed and integrated data discovery.

Page 24


Federation Context

geospatial data

geospatial metadata

discoveryinterface

End user


ebRIM ISOOGC Core

ECHO Service Core


Catalogue FederationCatalogue Federation

Page 25


Federation Case Study – GMU CSISS CFS SystemGMU CSISS CFS System

CommunityCatalogues

End user

GMU CSISS Catalogue Federation ServiceGMU CSISS Catalogue Federation ServiceGMU CSISS Catalogue Federation ServiceGMU CSISS Catalogue Federation Service

NASA ECHONASA ECHO GMU CSISS OGC CSWGMU CSISS OGC CSW DOE Earth System GridSimulation Data Catalogue

DOE Earth System GridSimulation Data Catalogue

DiscoveryInterface

GMU GUIGMU GUIGMU GUIGMU GUIThird Party SystemThird Party SystemThird Party SystemThird Party System

Page 26


Federation Case Study – GMU CSISS CFS System (Cont.)GMU CSISS CFS System (Cont.)

• We analyzed the following aspects of each catalogue– Metadata Conceptual Model– Query Language– Communication Protocol

Page 27



• Challenges in Federating NASA ECHO, GMU CSW, and ESG Catalogues:

– 1. Protocol AdaptationGMU CSW and the ESG catalogue support HTTP protocol (GET/POST) binding, while NASA ECHO uses SOAP to maintain the connection with the clients. The federation server should use the correct protocol when communicating with each Catalogue service. The protocol the clients may use to talk to the federation server itself is another concern. After all protocols have been defined and identified, the federation server should support protocol adaptation internally.

Page 28




– 2. Query DispatchingThe federation server is responsible for dispatching a query to the affiliated catalogue services. A dispatching model should be defined to deal with the following issues:

• Transparency: Whether the federation user is aware of these affiliated catalogue services and whether users can define which catalogue services are of interest in their queries.

• Sequence: Whether the federation server dispatches the users’ queries to these affiliated catalogue services in a predefined sequence, whether this sequence can be changed in runtime, and whether the federation users can define this sequence in their queries.

Page 29




– 3. Query Translation:The translation of queries is another major issue. The federation has to deal with the following problems:

Metadata Query Objects: The metadata objects queried using one set of query criteria may not have counterparts in another schema. For example, the federation service cannot fulfill queries for objects defined in GMU CSW and NASA ECHO for those simulation-specific metadata objects referenced only in the ESG catalogue schema. Another issue is that the same registry object has different names, in different schemes, e.g., Granule in NASA ECHO versus DataGranule in GMU CSW.

Query Format: Both GMU CSW and the ESG Catalogue accept queries in OGC Filter format, while ECHO only accepts IIMSAQL format. The federation server needs to transform an individual query into the different proprietary formats. The spatial query criterion and temporal query criterion are expressed differently in the NASA ECHO granule query payload and the GMU CSW granule query payload.

Query Language Functionality: Some complex query predicates in one query language cannot be identically expressed in another one. For example, the OGC Filter specification supports nested Boolean queries. Such queries can be supported at best with difficulty on ECHO IIMSAQL, and some cannot be supported at all.

Page 30




– 4. Results Integration:Catalogue query results from multiple Catalogue Services may need to be integrated before being sent back to users. As these different sets of metadata results may not use the same schema, the rules the federation server uses to re-organize metadata information while keeping the original content should be well designed. Furthermore, whether the clients can define the format of the query result of interest and, if so, how, also needs to be addressed.

Page 31



• We propose the following federation strategies:– 1. Protocol Adaptation

• As this federation is supposed to provide a single access point to multiple, autonomous information sources, it may follow the mediator-wrapper architecture, where the federation works as a mediator, and wrappers may be deployed for communicating with specific catalogue services if protocol adaptation is needed.

– 2. Query Dispatching• 1) Opaque: In this scenario, the federation service fully controls the distributed

query process, with the clients having no awareness of the affiliated Catalogue Services.

• 2) Translucent: The federation service may expose the affiliated Catalogue Services to the users, but the users can define neither which Catalogue Services their query can be forwarded to nor the sequence of queries.

• 3) Transparent: The federation service may expose the affiliated Catalogue Service to the users, and the user chooses which catalogue service shall be used for each inquiry and the order of the inquiries.

Page 32



• Proposed federation strategies:

– 3. Query Translation• Query Translation in the federation has two aspects: semantic and syntactic.

• A federation usually maintains a global schema that is exposed to end-users. Metadata attribute terms in user queries always follow this global schema. Before being dispatched to an underlying affiliated catalogue service, they should be transformed appropriately. This transformation logically involves four layers: metadata term, query criterion, query criteria, and query payload, as shown in the following picture.

Page 33



• Proposed Federation Strategy– 4. Query Result Integration

• A federation service needs to integrate query results from multiple underlying Catalogue Services before sending them back to the clients. It may choose to implement one of three kinds of integration mechanisms.

– Opaque: In this case, the federation service defines, maintains and advertises a unique information model. Each query result from affiliated Catalogue Services should, if necessary, be transformed to this information model. The original metadata information can be kept in the final transformed query results.

– Translucent: The federation service does not maintain a complete, unique information model but defines a common subset of metadata objects that are supported by all the affiliated Catalogue Services, such as name, and spatial and temporal range. The federation service transforms only this part of the metadata information, while the remaining embedded original metadata information remains unchanged in the final response.

– Transparent: The federation service has no role in metadata integration. All the query results from affiliated Catalogue Services are simply grouped together, keeping the original metadata formats. In this scenario, the users are supposed to analyze each result fetched from federation service, since the results may not all conform to the same schema even though grouped together in one response.

Page 34



• Federation System Architecture

• The GMU CFS consists of two types of components: Mediator and Wrapper. • The Mediator is a key component of the GMU CFS. It accepts user’s queries through an OpenGIS CSW query

interface.• For better modularity and sustainability, the GMU CFS has externalized the query translation and the result

transformation module for ECHO to make a wrapper, i.e., OGC CSW for ECHO. This wrapper accepts client queries, in OGC Filter format, that are forwarded by the Mediator. They are transformed to ECHO IIMSAQL format by the Query Transformation module.

Page 35



• Federation System Context

Page 36



• Discussion – This GMU CFS system can integrate NASA ECHO, GMU CSW and the DOE

ESG Simulation Data Catalogue. One advantage of its design is that CFS follows the OpenGIS Catalogue Service standard as the communication protocol with the underlying affiliated catalogue services. As long as new catalogue services follow this standard, they can easily be integrated into the federation system.

– However, integrating new legacy catalogue services cannot be plug and play. Abstracting the specific information models, the catalogue registration mechanism, and query orchestration would be new issues to consider when scaling this federation beyond these three catalogue services.

Page 37



• Discussion (Cont.) – In the strategy described here, the specific information models need to be

carefully evaluated before incorporation into the global schema and subsequent exposure to client users, when including new catalogue services. Efforts such as the ISO Technical Committee (TC) 211 Geospatial Metadata Standard 19115, FGDC Content Standard for Digital Geospatial Metadata (FGDC, 1998) and Dublin Core (DCMI, 2006) are attempting to standardize the geospatial metadata information model, but, in many cases, their use is voluntary.

– A metadata crosswalk could be of help when mapping two distinct models, but is not very suitable for one-to-many mapping.

– An ontology-based approach would provide a new way to create a global schema.

Page 38



• Discussion (Cont.) – New catalogue services must be registered manually. In the design presented

here, the federation service discovers the underlying catalogue services at design time, rather than at run time. This strategy greatly simplifies the mechanism for federation, and lowers the complexity of implementation.

– In fact, without an automatic way to integrate the information model, it does not make much sense to register the catalogue service automatically.

Page 39


Publications

• Journal papers– Towards a Geospatial Catalogue Federation Service

• Y. Bai, L. Di, A. Chen, Y. Liu, Y. Wei.• Photogrammetric Engineering and Remote Sensing (PE&RS). 2007, Vol.73, No.6, pp.699-709.

– A Taxonomy of Geospatial Services for Global Service Discovery and Interoperability • Y. Bai, L. Di.• Computers & Geosciences. (Under review).

• Book chapters– Catalogue Information Models

• L. Di, Y. Bai.• Encyclopedia of Geographical Information Science.

– Geospatial Image Metadata Catalogue Services • Y. Bai, L. Di, A. Chen, Y. Liu, Y. Wei.• Encyclopedia_of_Geoinformatics.

• Conferences– Serving Satellite Remote Sensing Data to User Community through the OGC Interoperability Protocols

• L.Di, W.Yang, Y.Bai.• AGU 2005 Fall Meeting.

– GEOSS Registry System: Enabling the Registering and Discovering of Geospatial Web Services Worldwide • Y. Bai, L. Di, N. Doug, Y. Wei.• AGU 2008 Fall Meeting.

Page 40


GMU CSISS CSW/CFS Applications

• NASA NEHEA GeoBrain project– Mobilizing NASA EOS data and information through Web service and knowledge

management technologies for higher-education teaching and research

– PI: Liping Di at GMU

– Co-I: 9 Educational partners

– Task Lead: Peisheng Zhao at GMU

Page 41


GMU CSISS CSW/CFS Applications (Cont.)

• NASA AIST Grid/OGC project– Integration of OGC and Grid Technologies

for Earth Science Modeling and Applications


– Co-I:• Piyush Mehrotra

– NASA Ames Research Center• Dean Williams

– DOE LLNL

– Task Lead: Aijun Chen at GMU

Page 42



• GEOSS Component and Service Registry– Maintenance and enhancement of the GEOSS Registry for earth observation


– Task Lead: Yuqi Bai at GMU

Page 43



• GEOSS GeoNetwork Clearinghouse Candidate System– Catalogue Service Interface for Web Portals

• Task Lead: Yuqi Bai at GMU

Page 44


Summary

• Metadata and Catalogue system work behind the scenes to support geospatial data discovery

• Online discovery of textual metadata information has greatly facilitated direct discovery of large volumes of geospatial data

• Multiple metadata standards/profiles exist at the international level and national level

• OGC specifications (baseline/profiles) are main players regarding the definition of the the Catalogue’s behavior

• Heterogeneous Catalogue systems prevent further integration and interoperability

• It is desirable to have a Catalogue Federation to– provide a single interface to users, while– hiding the complexity and diversity of the affiliated catalogues behind the

interface• GMU CSISS Catalogue Federation product has been presented

– Research findings– System design and implementation– Lessons learned

• GMU CSISS CSW and CFS systems have been successfully used in several national and international projects (funded by NASA, FGDC, and GEO)

Page 45


Acknowledgements

• This study was supported by grants from NASA through the REASoN program (NNG04GE61A; Professor Liping Di, Principal Investigator).

• I appreciate my teammates for their kind support over the last three years.• Thank you for your attention this afternoon.

Happy Halloween !

Documents

Effective discovery of geospatial data: a geospatial catalogue perspective