Bryan Lawrence on behalf of the NDG, BADC and BODC. Ray Cramer, Marta Gutierrez, Kerstin Kleese, Siva Kondapalli, Sue Latham, Roy Lowry, Kevin ONeill, Ag Stephens, Andrew Woolf NERC DataGrid Status: ESP June 2004 British Atmospheric Data Centre British Atmospheric Data CentreOutline NDG Aims and Metadata Taxonomy (Review ) Demonstration of NDG in action (no grid services yet, but shape of things to come should be clear) Stub-B New Tool: DataExtractor Status Issues with metadata Chemistry data at BADC Numerical Simulation Discovery & back to Status British Atmospheric Data Centre British Atmospheric Data Centre British Oceanographic Data Centre Simulations Assimilation Complexity + Volume + Remote Access = Grid Challenge British Atmospheric Data CentreNDG Metadata Taxonomy British Atmospheric Data Centre British Atmospheric Data CentreNDG Metadata Architecture Service based model: clear separation between discovery and use discovery service standards compliant and interoperable British Atmospheric Data Centre(D) - Discovery OAI Open Archives Initiative Digital Library Protocol for harvesting metadata. NDG Supports Multiple Discovery Services build your own Multiple Protocol Support will be built into the NDG Vanilla Discovery Service British Atmospheric Data CentreNDG Structure British Atmospheric Data Centre British Atmospheric Data Centre British Atmospheric Data CentreCan order responses by title or data centre (or default random) Choose to return either data or B-Metadata Look at DIFs in either HTML or XML Flexible Information Return British Atmospheric Data Centre British Atmospheric Data Centre British Atmospheric Data Centre British Atmospheric Data Centre British Atmospheric Data Centre British Atmospheric Data Centre British Atmospheric Data Centre British Atmospheric Data Centre British Atmospheric Data CentreRole of B metadata: domain ontology B metadata is a store of metadata intended to: Allow the production of the various industry standard discovery formats (DIF, DC, FGDC/GEO, 19115) Provide a more complete metadata store than that demanded by the usual discovery formats, leveraging the metadata holdings of the data centres Allow a smooth link across to the data browse and use elements of the NDG Expected to expand in importance as we can add more semantic detail to the schema British Atmospheric Data CentreB metadata a simplified view British Atmospheric Data CentreHow is the B metadata implemented? Core linking concept is the deployment Deployment Activity on behalf of an Activity of a Data Production Toolat an Observation Station that produces a Data Entity Data Production Tool Observation Station Data Entity Each of the main metadata objects has security data attached to it. This means that this can be applied to queries on the metadata Links the metadata records into a structure that can be turned into a navigable XML using Xquery or XSLT with any of the record types as the root element. British Atmospheric Data CentreStub B what is it? B metadata works well in databases, but what about: presentation standalone generation of D storing metadata locally as files Given a raw B record for a Data Entity contains just: the basic data entity details a series of references to related records no details such as: activity name, instrument name, station stub B is the base entity expanded through its own related deployments and internal references British Atmospheric Data CentreRole of Stub B Makes application developers lives easier, especially in the presentation of search results Allow off-line storage of metadata by users Basis of D production via XSLT Hook into main B repositories Potential discovery format (while there are lots around already this could allow more discipline dependent discovery) British Atmospheric Data CentreDiscovery Metadata Usage British Atmospheric Data Centre British Atmospheric Data Centre British Atmospheric Data Centre British Atmospheric Data Centre British Atmospheric Data Centre British Atmospheric Data CentreBackground activity being parallelised with GODIVA/CCLRC e-science collaboration (spectral -> gridpoint + CDMS + visualisation tools) Download either plot or the data that went into the plot. British Atmospheric Data CentreWhere are we? Major effort on defining feature types for observation types so we can build an OGC/ISO compatible data extractor for observations and numerical data. Main thrust for Andrew Woolf and 0.5 New FTE Ag Stephens contributing when time available Security Infrastructure Development Collaboration with CCLRC e-science, ECOGrid and 0.5 FTE Ongoing work on metadata definition and population: Oceanographic data Siva Kondapalli Chemistry data Main thrust for Sue Latham Numerical Modelling data DIF numerical definition (moving to ISO), BADC and UK Community Katherine Boutons work at NCAS/CGAM Remote Sensing Data Collaboration with NEODC and PML Ongoing work on databases and interfaces, DIF to ISO and B Kevin ONeill and Marta Gutierrez British Atmospheric Data CentreAuthorisation Role-based access : badc.nerc.ac.uk ukmo-obs researcher ukmo-obs nerc Key concept: Only hosts that trust each other share data, even within a larger virtual organisation: e.g. at BADC: ndg.bodc.nerc.ac.uk nerc ashoe nerc bodc Signed conditions of use form exists for this dataset British Atmospheric Data CentreNDG Security Certificate based, pass encrypted credentials between user and gatekeeper. British Atmospheric Data CentreExtending the CF convention for chemistry grep -i sulphate vars2.csv "Allen, Andrew and Grenfell, Lee ", Sulphate / coarse (ug/m3) "Allen, Andrew and Grenfell, Lee ", Sulphate / fine (ug/m3) "Bradbury, Carl ", SULPHATE LOADING (ug/m3) "James, Jonathan And Allen, Andrew ", Sulphate / coarse (ug/m3) "James, Jonathan And Allen, Andrew ", Sulphate / fine (ug/m3) "James, Jonathan And Allen, Andrew ", Sulphate / fine+coarse (ug/m3) "McArdle, Nicola and Thompson, Adrian ", sulphate (nmol m-3) "McArdle, Nicola and Thompson, Adrian ", sulphate (M) "McArdle, Nicola and Thompson, Adrian ", sulphate 1.2 m diameter (nmol m-3) "McArdle, Nicola and Thompson, Adrian ", sulphate bulk (nmol m-3) "McFadyen, Gordon ", Sulphate "Robertson, Leonie and Davison, Brian ", Coarse sulphate concentration (ug m-3) "Robertson, Leonie and Davison, Brian ", Fine sulphate concentration (ug m-3) Currently 35,000 Ames format files, mostly Atmospheric Chemistry Real problems with vocabulary, and units Spinning up a new project need community help! grep -i butane vars2.csv i-Butane (ppt) iso-Butane (ppt) n-Butane (ppt) iso-Butane pptv ISO-BUTANE (pptv) i-,n-butane ACSOE just one of many datasets with this problem British Atmospheric Data CentreDRAFT DIF Component (1) Key New Groups: Numerical Model ID Information Numerical Model Components (from) Atmosphere, Ocean-Dynamic, Ocean-Thermodynamic, Cryosphere, Land-Surface with possible appends: Chemistry, 4D-VAR, 3D-VAR, QG details for each Numerical Simulation ID Information Initial Condition Information details Forcing Information details British Atmospheric Data CentreDRAFT DIF Component (2) *=required +=repeatable Group: Numerical_Model Model_Name: Model_Version: * Model_Calendar: [model calendar valid] - eg CF calendar or ISO *+ Group: Model_Component * Model_Component_type: [Model Component Valid] Model_Component_Resolution: Group: Model_Component_VerticalDomain VerticalDomain_Top: VerticalDomain_Bottom: End_Group Model_Component_Timestep: Group: Model_Component_Summary [Multiple text lines allowed] End_Group URL: End_Group Group: Numerical_Model Model_Name:HadCM3 Model_Calendar: 360 day Group: Model_Component Model_Component_type: Atmosphere Model_Component_Resolution: 2.5 degrees latitude, 3.75 degrees longitude Group: Model_Component_VerticalDomain VerticalDomain_Top: 1000 hPa VerticalDomain_Bottom: 4 hPa End_Group Model_Component_Timestep: 0.5 hours Group: Model_Component_Summary Cullen et al Atmospheric Model adapted for climate use. End_Group Example Group: Model_Component Model_Component_type: Ocean Model_Component_Resolution: 1 degrees latitude, 1 degrees longitude Group: Model_Component_VerticalDomain VerticalDomain_Top: 0m VerticalDomain_Bottom: 6000m End_Group Model_Component_Timestep: 0.5 hours Group: Model_Component_Summary Bryan and Cox Ocean model End_Group URL:(e.g.) End_Group British Atmospheric Data CentreDraft DIF Components (3) *=required +=repeatable Group: Numerical_Simulation Numerical_Simulation_Name: * Numerical_Simulation_ID: [recorded using PURI] * Group: run_period * start_date: [yyyy-mm-dd-hh] * end_date: [yyyy-mm-dd-hh] * real_date: [yes,no] End_Group * Group Initial_Condition: Ensemble: [Numeric Value] Summary: End_Group *+ Group Forcing: Ensemble: [Numeric Value] Ensemble Parent : [uri or 0] Summary: End_Group Group: Numerical_Simulation Numerical_Simulation_Name: All Forcings Numerical_Simulation_ID: format_tbd_but_using_purl Group: run_period start_date: end_date: real_date: yes End_Group Group Initial_Condition: Ensemble: 4 Ensemble Parent: 0 Summary: initial conditions taken from the HadCM3 control integration End_Group Group Forcing: Summary: Volcanic forcing from Sato et al End_Group Group Forcing: Summary: Solar Forcing from Lean et al End_Group Group Forcing: Summary: CO2 from... End_Group Group Forcing: Summary: Anthropogenic SO2 from... End_Group Example Group: Numerical_Simulation Numerical_Simulation_Name: Ensemble Member of blah Group: run_period start_date: end_date: real_date: yes End_Group Group Initial_Condition: Ensemble: 1 Ensemble Parent: format_tbd_but_using_purl Summary: initial conditions taken from the HadCM3 control integration End_Group Group Forcing: Summary: Volcanic forcing from Sato et al End_Group Group Forcing: Summary: Solar Forcing from Lean et al End_Group Group Forcing: Summary: CO2 from... End_Group Group Forcing: Summary: Anthropogenic SO2 from... End_Group Example Ensemble Member British Atmospheric Data CentreWhere are we? Major effort on defining feature types for observation types so we can build an OGC/ISO compatible data extractor for observations and numerical data. Main thrust for Andrew Woolf and 0.5 New FTE Ag Stephens contributing when time available Security Infrastructure Development Collaboration with CCLRC e-science, ECOGrid and 0.5 FTE Ongoing work on metadata definition and population: Oceanographic data Siva Kondapalli Chemistry data Main thrust for Sue Latham Numerical Modelling data DIF numerical definition (moving to ISO), BADC and UK Community Katherine Boutons work at NCAS/CGAM Remote Sensing Data Collaboration with NEODC and PML Ongoing work on databases and interfaces, DIF to ISO and B Kevin ONeill and Marta Gutierrez British Atmospheric Data Centre(B) Metadata Model British Atmospheric Data Centre(B) Metadata Model Overview GIS/ISO Feature Types British Atmospheric Data CentreDataset Variables Multidimensional array... of other arrays... or from aggregated storage Rich spatiotemporal referencing (standards- compliant: ISO19108, ISO19111) (A) NDG Semantic Data Model