The Ocean, the Atmosphere, and the Grid

A UK perspective

David WebbSouthampton Oceanography Centre

NaturalEnvironmentResearchCouncil San Francisco 4/5th August 2001

Satellite Observations

Ocean Models

ECMWF Winds

Ship observations

Alace floats

Autonomous vehicles

Moorings

Bugs

The Oceanographers View ...

The user wants ...

Seamless movement from

data search

to data extraction

to visualisation

to comparison with other data

to analysis

to …

and back again.

Some of the problems ...•Very large individual data files - up to 8

Gb

•Large numbers of files - i.e. one every six hours

•Very diverse dataWe need: - Quick look data

- Sub-samples in space and time

- Track local and cache copies

- Map conversionsWe want to: - Inter-compare 2-D and 3-D fields

- Overlay station data on 2-D and 3-D fields- Simplify access -Use browsers -Move logic to the user.

Outline Data Grid

HPC

Browser

C/Fortran

Cache

GIS

Users

Cache

Data Source

Farm

Data Centres

Middleware

+ Agents

SOC

Metadata

BODC

BADC

3D System

WWW

Data Centres Middleware+ Agents

User interface

Organisation

•Simple data transforms

Change gridSub-sampleInterpolateFormats

•Complex transformsDensity/vorticity/etcExtreme eventsHeat fluxes

•Transform metadata

•Logical operationsHunt for related

dataData quality tests

•Handle caching/delays

…and W3C compliant

• MetadataDataCached dataModel GridsAlgorithms

•Formats + conventions

HDFnetCDF/ferret

•RetrievalSpeedDelays

•Pre-processingSub-samplingData compression

•Extreme events (Do you want 200 GB?)

•Browser (Netscape) and anonymous users

•Fortran/C program

•Matlab /GIS

•Handle caching/delays

•Data Sources

The Atmospheric Data Centre View ...

BADC- Migrating from “downloads” to E-Science

BADC Data & Catalogue

At USER Institution

At BADC

which will eventually be “griddable”.

Database Client

Computation

Graphics

3

we are about to provide tools at the data centre,

Database Client

Database

2

In the beginning, users downloaded files,

Graphics

Computation

1

New Concept: The Catalogue Interface Resource Broker (more than just the SRB)

Accessing Atmospheric Data at the BADC

via Grid Technologies

•The CIRB would provide access to data both within the BADC and in other locations. It would be “cache aware”.

•No one data centre would be the unique CIRB, rather each would be a peer in a network of CIRBs fronting each data source.

•The CRIB will provide access to the data lying in the other data stores. Security and resource issues will be dealt with by “e-science” software agents.

•Users would be able to access data via a GUI on their own machine, on our systems, or even via user written APIs.

Possible US Links:

NCAR - Atmospheric Data

NOAA - Ocean and Atmospheric Data

Los Alamos - Ocean Model Data

LLNL - Climate Change Model Data Sets

Summary

San Francisco 4/5th August 2001

The data grid needs:

√ Dataset sub-volumes

• Dataset sub-sampling

• Data cache and cache metadata

• Conversion and compression

User software to seamlessly integrate

• Data discovery from many data centres

• Plotting/comparison of data

• As GIS, Browser and subroutines

The compute grid needs:

• link to compute portals• access and security• libraries• scheduling for

- ensemble experiments - loose coupled models - fully coupled models

• automatic cataloguing• globally accessible databases• code maintenance