Earth System CoG and the Earth System Grid Federation:

A Partnership for Improved Data Management and Project Coordination

BESSIG March 18, 2014Boulder, CO

Sylvia Murphy (NOAA/CIRES) (sylvia.murphy@noaa.gov), Luca Cinquini (JPL/NOAA), Cecelia DeLuca (NOAA/CIRES),

Allyn Treshansky (NOAA/CIRES)

Presentation Outline

• Overview of ESGF• ESGF Usage• Overview of CoG• CoG Capabilities• Future Work

Overview of ESGF

• The Earth System Grid Federation (ESGF) is a multi-agency, international collaboration of people and institutions working together to build an open source software infrastructure for the management and analysis of Earth Science data on a global scale.

• ESGF is a system of distributed and federated Nodes that interact dynamically through a Peer-To-Peer (P2P) paradigm.

• A client (browser or program) can start from any Node in the federation and discover, download and analyze data from multiple locations as if they were stored in a single central archive.

ESGF Usage

• CMIP5 (Phase 5 of the Climate Modeling Intercomparison Project)… possibly the largest coordinated scientific modeling effort of all time.» 40+ models, 25+ modeling centers, 17

countries» global, distributed archive comprising 2.5 PB

of data• obs4MIPs: NASA and DoE observations packaged as

CMIP5 output for easy comparison• ana4MIPs: reanalysis data• CORDEX: regional climate models, 2PB data• TAMIP: atmospheric models intercomparison• GeoMIP: geo-engineering models intercomparison• DCMIP: dynamic core models intercomparison

Overview of CoG

• CoG is a collaboration environment and hub to connect projects in the Earth sciences.

• It hosts software development projects, model intercomparison projects (MIPS), and university short-courses or workshops.

• It includes a configurable search to data on ANY ESGF data node.

• It provides projects with a wiki and customizable navigation to wiki content.

• It contains an ontology for the description and management of projects and provides a consolidated look at this content across a project’s network.

• It contains a file server for documents and images.

• It provides services for Earth system model metadata collection and display.

Some of the 74 projects hosted on CoG include:

• Ana4MIPs• Obs4MIPs• National Climate Predictions and

Projections Platform (NCPP)• Climate Informatics (University of

Michigan)• Earth System Documentation (ES-

DOC)• NOAA’s High Impact Weather

Prediction Project (HIWPP)• Earth System Prediction Capability

(ESPC)• Dynamical Core Model

Intercomparison Project (DCMIP)

Customizable Data Services…Interfacing with ESGF

• Search widget can be turned on/off.• Search can be narrowed to any ESGF node and to any

project (e.g. CMIP).• Search facets can be created, deleted, and grouped.• Help text can be added to the top of the search page.• Search results can be saved to a Data Cart associated

with a user. Items in the Data Cart persist. • Search results can be:

– Forwarded to the Live Access Server (LAS) for simple visualization.

– Downloaded directly via a WGET script.– Associated with model metadata if it exists.

ESGF Search Customization

Data Cart

• Items in the Data Cart can be sent individually or collectively to LAS or WGET.• The Data Cart is associated with a user and not a project.

Show Metadata

Wiki and Collaboration Tools

The CoG layout is color-coded: • The right-hand side (dark

yellow) is where services (data, news, project connectivity) are located.

• The Upper Navigation bar (dark teal) contains links to project-level metadata.

• On the left (light teal) is an auto-generated navigation system created when projects develop freeform content.

• The central portion of the site is a wiki that allows projects to create their own content.

Screenshot of the CoG project workspace for the 2012 Dynamical Core Model Intercomparison (DCMIP) Workshop.

Project Networks and the Project Browser

• Projects in CoG are arranged in a hierarchy of Parents, Peers, and Children.• The Project Browser displays the network and allows for inter-project navigation. • Projects can be tagged with keywords and projects can be searched for using

keywords.

Project-level Metadata Roll-up

• Management of information is a major problem in projects that involve many sub-projects, partners, multiple leads, and many resources.

• CoG acts as an index into project information that is necessary for coordination and collaboration and enables people responsible for overall coordination to quickly get consolidated views of information.

This example shows the Partners feature that allows projects to list their project partners and include a logo for each. Below the list for ED-DOC is a consolidated view of the partners for ES-DOC’s peer projects.

CoG Schema

The CoG schema contains classes to describe software development projects, short-courses or meetings, and overall project coordination. Projects select which metadata to display via a simple web form.

Project-level metadata is linked in standardized locations via the upper navigation bar.

UML Diagramhttps://earthsystemcog.org/site_media/projects/cog/cog_ontology.png

Resources

• Resources are pointers to data, files, and URLs.

• Resources folders can be created, moved, and deleted.

• Projects can turn on a set of standardized Resources folders (e.g. Presentations, Minutes).

• Saved data searches can be saved as a Resource.

• Each Resource can have a private wiki-based notes page to facilitate discussions.

News

• News is a way to send announcements across a project network.• News is visible in the news widget on any targeted project.• News will be added to social media (Google+, Facebook, Twitter, RSS) in a

future release.

Model Metadata Services

• The CoG Team is partnering with the international Earth System Documentation (ES-DOC) project to develop and use an Earth System Model metadata entry and view capability.

• The ES-DOC Viewer is a lightweight JavaScript plugin that will display any Common Information Model (CIM) record.

• The ES-Questionnaire collects standardized CIM metadata through a high-customizable web form. The output is saved to a community CIM repository.

Future Work

• CoG – ESGF Integration (through summer 2014):– CoG is going to replace the ESGF web front end.– CoG will be federated so that projects hosted on one CoG-ESGF instance will

be visible on others.– OpenID access added.– Look and feel will be more customizable to meet institutional branding

requirements.

• Possible other features:– Be able to export the CoG ontology. – Be able to list non-hosted projects the Project Browser. – Be able to version the wiki. – Enable RSS and social networking for news. – Enable non-wiki links in the left navigation bar.

Questions?

Questions and contacts:cog_support@list.woc.noaa.gov

CoG: https://earthsystemcog.org/ESRL ESGF data node: http://hydra.fsl.noaa.gov/esgf-web-fe/PCMDI ESGF data node: http://pcmdi9.llnl.gov/esgf-web-fe/

JPL ESGF data node: http://esg-datanode.jpl.nasa.gov/esgf-web-fe/