24
JERICO WP5 Data management and distribution Alessandro Crise Istituto Nazionale di Oceanografia e di Geofisica Sperimentale JERICO Project Kick-Off meeting Maison de la Recherche Paris, 24-25 May 2011

JERICO WP5 Data management and distribution

  • Upload
    nodin

  • View
    49

  • Download
    0

Embed Size (px)

DESCRIPTION

JERICO WP5 Data management and distribution. Alessandro Crise Istituto Nazionale di Oceanografia e di Geofisica Sperimentale. JERICO Project Kick-Off meeting Maison de la Recherche Paris, 24-25 May 2011. JERICO WP5 Data management and distribution: main goal. - PowerPoint PPT Presentation

Citation preview

Page 1: JERICO WP5 Data management and distribution

JERICO WP5

Data management and distribution

Alessandro Crise Istituto Nazionale di Oceanografia e di Geofisica Sperimentale

JERICO Project Kick-Off meetingMaison de la RechercheParis, 24-25 May 2011

Page 2: JERICO WP5 Data management and distribution

JERICO WP5Data management and distribution: main goal

The main goal is to design the JERICO Data Flow and Management infrastructure based on the implementation of an end-to-end ‘open and free’ conduit from data to users and upon the guidelines proposed in the GOOS Report 148 (2005).

The implementation will be consistent and complementary with the major European initiatives for the establishment and coordination of infrastructures for the management and distribution of data and products in Europe which can use JERICO data (SeaDataNet, EuroGOOS Regional Centres, MyOcean).

YET ANOTHER DATA MANAGEMENT SYSTEM?: NO!

Page 3: JERICO WP5 Data management and distribution

WP5 Structure

JERICO will need to ensure that the flow of real-time and delayed mode will be reliable, accessible and easy to distribute. The WP5 structure:

• Creating value for measured data: from raw data issued from the sensors to validated data stored and marked. • Facilitating data and meta-data flow to existing data repository infrastructures (SeaDataNet); • Helping in delivering real-time JERICO data to ROOSs and My Ocean

Page 4: JERICO WP5 Data management and distribution

Task 5.1: Quality assurance: create value for measured data

Task responsible: OGS

Partners: HCMR

Duration M1-M42

• This task is aimed at adding value to observations by associating them with characteristic uncertainties.

• By definition, the “expected” accuracy associated with an observation is an attribute of the measurement chain that produces it.

Page 5: JERICO WP5 Data management and distribution

Task 5.1: Quality assurance objectives

The objectives are:

Collecting pre-existing knowledge regarding the key identified sensors and critically revise state-of-art literature on in lab and in situ performances of the most popular sensors (in conjunction with WP4)

defining procedures for ascribing a total uncertainty to measurements that will account for the actual circumstances and conditions under which the measurements were taken.

Testing and verifying uncertainty computational procedures

Page 6: JERICO WP5 Data management and distribution

Task 5.1: Quality assurance methodologies

. MethodologyA novel model of a-priori uncertainty will be proposed as a sum of fixed and time-dependent source of error and uncertainty (precision and accuracy, drifts, experimental uncertainties, the representativity error, etc.) The variables temperature, salinity and fluorescence/chlorophyll will be considered at first. Controlled experiments will be performed in order to document and validate the different uncertainty components composing the final uncertainty estimates. All the relevant assessments carried out in WP4 will be included as basic information for the combined uncertainty estimation. WP4.1 will provide an updated summary of these achievements to WP5.1, no later than the dates set down in the WP4 D4.2 delivery schedule (M14)

Page 7: JERICO WP5 Data management and distribution

Task 5.2: Harmonization of Delayed Mode data management procedures with SEADATANET

Task responsible: IFREMER,

Partners: HCMR, MUMM, OGS

Duration: M1-M42

• This task will define, establish and oversee the data management infrastructure for dealing with delayed-mode data in JERICO.

• The goal is to provide JERICO users with relevant informations for seamless data integration with the EU SEADATANET initiative.

Page 8: JERICO WP5 Data management and distribution

Task 5.2 Methodology

- Survey the existing delayed-mode data handling practices of JERICO partners.

- Formulate a viable proposal for a common JERICO delayed-mode data management platform that can serve to reinforce the SEADATANET effort;

- the guidelines established by DG-Mare/EMODNET and WISE-Marine will be

also taken in account when doing this.- Create common vocabularies for JERICO delayed-mode data

formats and meta-databases, building upon the work that has already been done in SEADATANET.

- Implement delayed-mode data and metadata formats that are compatible with those of SEADATANET (in compliance with the EU INSPIRE Directive);

- the Sensor Web Enablement family of the OGC consortium family of standards (SensorML, Sensor Registry, O&M) will be considered in the implementation.

- Ensure easy sharing and secure archival of delayed-mode data within JERICO by employing common data transport formats and storage criteria making use of the experience gained by SEADATANET.

Page 9: JERICO WP5 Data management and distribution

Task 5.2 Metadata implementation

Discovery metadataTo let all know what have been deployed during the project (where, what, how, who…)

To help to retrieve data related to observatories

To be compliant with European directives (Inspire)

Enabling technology

Use of SeaDataNet EDIOS directory to describe observatories

Typing / Generation of metadata using SeaDataNet procedures and tools (Mikado) complemented with automatic tools developed in Emodnet-Physics to better use the RT Netcdf files provided in RT by the ROOSs

Page 10: JERICO WP5 Data management and distribution

Task 5.2 Data implementation

Delayed mode dataDelayed mode quality checkUniform file format (based on OceanSites NetCDF, compatibility with CF convention)

+ distribution in ASCII flat file format (Ocean Data View)

Full metadata (Common Data Index) generated by Mikado for long term reuse

Data access via web portal (all users)

Page 11: JERICO WP5 Data management and distribution

Task 5.3 Harmonization of Real Time data anagement procedures with MyOcean and EuroGOOS & the institution of

data access services for JERICO TOP activities

Task responsible: IFREMER,

Partners: CNR, NIVA, IMR, HCMR,PUERTO, SMHI

Duration: M1-M42

• This task will define, establish and oversee the data management infrastructure for dealing with real-time data in JERICO.

• The goal is to provide for seamless integration with the EuroGOOS and EU Marine Core Service’s MyOcean component and in order to render JERICO real-time data accessible for use in generating downstream services and products.

Page 12: JERICO WP5 Data management and distribution

Task 5.3 Methodology

• Survey the existing real-time data handling practices of JERICO partners.

• Formulate a viable proposal for a common JERICO real-time data management platform that can serve to reinforce EuroGOOS and MyOcean

• Create common vocabularies for JERICO real-time data formats and metadatabases, based on any existing

• recommendations by MyOcean/EuroGOOS for doing this.

• Implement real-time data and metadata formats based on any existing recommendations by EuroGOOS and MyOcean for doing this.

• Reconcile, wherever possible, the data quality assurance procedures/protocols for real-time data amongst the JERICO partners (link to WP4, WP5.1).

• Ensure easy exchange of real-time data within JERICO by employing common data transport formats and storage criteria, based on any existing recommendations by EuroGOOS/MyOcean for doing this.

Page 13: JERICO WP5 Data management and distribution

WP5.3 Implementation

Real-Time dataReal-Time Quality check based on EuroGOOS/MyOcean procedure that would need to be extended to new parameters whenever possible

Uniform file format (based on OceanSites NetCDF, compatibility with CF convention)

ftp file server (with an index file) per region

Oceanotron (OpenDap protocol) for direct access by remote software

Page 14: JERICO WP5 Data management and distribution

WP5 in JERICO Data flow

WP5 act as mediator in JERICO Data Flow ModelA priori quality assurance (WP4)

Data management procedures (WP5)• Time-dependent data uncertainty estimation• Real-time data management• Delayed mode data management

Value added products and data dissemination (WP6)

EuroGOOS ROOSs engagement (WP2)

Demonstation of data provision (WP7)

SeaDataNet

MyOcean

Emodnet Physical Parameters

EuroGOOS

Page 15: JERICO WP5 Data management and distribution

MyOcean Insitu TAC implementation of EuroGOOS regional approach

Page 16: JERICO WP5 Data management and distribution

MyOcean In-situ TAC role

Limited number of parameters: Physical: T&S, current, sea levelBiogeochemical: Chlorophyll/Fluorescence, Oxygen, Nutrients

Integrate in-situ data in product accessible through global and regional portals

Common formatCommon NRT QCCommon Quality flagsCommon distribution toolsSingle access point

Ensure a minimum level of quality on the data deliveredIn Near real time ( 24h to a week)In delayed mode

Assessment of the products at basin scale

Page 17: JERICO WP5 Data management and distribution

The European Commission, represented by DG MARE, concluded a service contracts aiming to:

Create a prototype components of the European Marine Observation and Data Network (EMODnet).

Provide access to archived and real-time data on physical variables in Europe's seas and oceans

determine how well the data meet the users needs.

EMODNET PHYSICAL PARAMETERS

Page 18: JERICO WP5 Data management and distribution

THE EMODNET PP PILLARS

EMODnet PP

EMODnet PP aims to bridge the communities behind MyOcean, EuroGOOS and SeaDataNet in order to strengthen their position inside the Commission agenda on IS and expand in future the participation to all partners

MyO

cean

Eu

roG

OO

S

Sea

Dat

aNet

JERICO WP5

Page 19: JERICO WP5 Data management and distribution

NEW FEATURES

Add new stations and models

Add profiles

Make use of the results from ECOOP

Add data from FerryBox and RV

Common quality control

Link to EDIOS

SEPRISE data in SDN ODV format

Put data on GTS

E-HYPE

EuroGOOS SEPRISE network

Page 20: JERICO WP5 Data management and distribution

Cooperation between MyOcean in-situ TAC and SeaDataNetMoU signed Beginning 2010

Observing systems

Marine governmental agencies and research institutes (including NODCs)

Other observations(e.g. scientific

cruises…)International organisations

(IOC/IODE, ICES, ICSU World Data System…)

delayed-mode

real-time

real-

time

Metadata(e.g. EDIOS)

Metadata(e.g. EDIOS)

VocabulariesVocabularies

Metadata (EDIOS, CDI, CAMIOON,

SensorML)

Metadata (EDIOS, CDI, CAMIOON,

SensorML)

Yearly regional products(climatologies, datasets)

Yearly regional products(climatologies, datasets)

Re-analysis users

Re-analysis users

UsersUsers

Operational users

Operational users

MyOcean in-situ TAC

SeaDataNet

Direct usersDirect users

Direct usersDirect users

Data integration(QC, « best copy », archival, re-analysis

datasets

Data integration(QC, « best copy », archival, re-analysis

datasets

Technical WGTechnical WG

Page 21: JERICO WP5 Data management and distribution

JERICO role in the European Data Management landscape

International organisations (IOC/IODE, ICES, ICSU World Data System…)

delayed-mode

real-

time

Metadata

(e.g. EDIOS)

Metadata

(e.g. EDIOS)

VocabulariesVocabularies

Metadata (EDIOS, CDI, CAMIOON,

SensorML)

Metadata (EDIOS, CDI, CAMIOON,

SensorML)

Yearly regional products(climatologies, datasets)

Yearly regional products(climatologies, datasets)

Re-analysis users

Re-analysis users

UsersUsers

Operational users

Operational users

MyOcean in-situ TAC

SeaDataNet

Direct usersDirect users

Data integration(QC, « best copy », archival, re-analysis

datasets

Data integration(QC, « best copy », archival, re-analysis

datasets

Technical WGTechnical WG

ObservingNetwork

Information & evidence pipeline

JERICO Coastal

Observatory Network (WP7)

Web Portal

real-time

Assessmentdatabase

‘EM

EC

O’ like

DataXML, CSV

‘EMECO-like’ Web –

enabled Datatools

(WP6)

Page 22: JERICO WP5 Data management and distribution

WP5 Milestones/Deliverables

Page 23: JERICO WP5 Data management and distribution

ACTIONS

To propose a formal agreement with SDN, EuroGOOS/MyO for JERICO data provision, to ensure more visibility to JERICO project

To find an agreement with DG MARE/EMODNET for the availability to JERICO of the developments achieved/to be achieved in EMODNET Physical Parameter lot.

Page 24: JERICO WP5 Data management and distribution

Thanks!