ESA LTDP Programme & LTDP WG Activities Overview

CCSDS DAI WG Meeting

31 March 2014

Mirko Albani, ESA/ESRIN

LTDP Main Drivers at ESA

Data and Knowledge Preservation

•Preserve any mission acquired data & associated knowledge as humankind asset

safeguarding MSs investment & effort in space R&D & operational data acquisition.

•With the assumption/awareness that one can never anticipate the exploitation value

of the data asset in the future.

•Preservation to be addressed in all EO mission phases; incremental approach for

past/current missions, systematic planning/approach for future ones.

Data Valorization

•Make sure full content of the ESA archives is fit for contemporary purpose and

maintained accessible and usable in the long term.

•Valorize the data assets for current/future exploitation through building long time

series of coherent data (harmonized formats and metadata) and data collections for

specific applications (e.g. Fundamental Climate Data Records); pursue multi-

disciplinary data use and “cross-fertilization”.

Optimize investments in future missions GS Dev&Ops, anticipating

LTDP requirements and operational scenarios.

ESA EO Long Time Data Series Holdings (excerpt)

Missions in red are covered in the LTDP Programme

ESA LTDP Programme Activities (EO Part) 1. LTDP Coordination and International Cooperation

• Continue leading international cooperation on LTDP in EO and

implement basic activities to sustain the European LTDP

cooperative framework for national and international missions.

• Consolidate the ESA EO Data Preservation Scenario and apply to all

ESA missions and data holdings.

2. EO Historical Missions Datasets Consolidation, Access,

Preservation & Curation

• Consolidate and archive ESA EO Historical Missions data &

associated information; ensure LTDP missions data discovery &

access as is today and perform full reprocessing & alignment to

current missions when feasible

3. LTDP Specific Developments and Evolutions

• Implement LTDP specific evolution and consolidation activities

necessary to meet programme objectives

European LTDP

Framework

European LTDP

Framework

EC Projects

ESA membership Standards

WGISS

DSIG Task IN-02

LTDP

Activities

International Context (since 2008)

EO International Coordination and Cooperation

Activities (1)

• ESA is coordinating the LTDP cooperation activities in the Earth

Observation domain with European partners through the LTDP WG

formed within the Ground Segment Coordination Body (GSCB).

• Implemented the basic rules of the European LTDP Framework in Earth

Observation: “LTDP Common Guidelines and Preserved Data Set

Content”

Reflecting the consensus of the European EO Data providers.

Reviewed at GEO, CEOS and with NASA.

Being reviewed with QA4EO.

• “EO Preservation Workflow” and harmonized high level procedures for E2E data

consolidation and data media transcription under definition.

LTDP CORE DOCUMENTS

LTDP Guidelines Issue 2.0 available at: http://earth.esa.int/gscb/ltdp/EuropeanLTDPCommonGuidelines_Issue2.0.pdf

Continuously evolving: next update in Q4 2014. CGMS WG (Korea, ESA, EUM, WMO) is considering the LTDP Guidelines as starting point for the generation of data preservation guidelines.

PDSC Issue 4.0 available at: http://earth.esa.int/gscb/ltdp/LTDP_PDSC_4.0.pdf

Continuously evolving: next update in Q4 2014

Preservation Workflow and E2E Consolidation Procedure being drafted: Based on the experience gained through historical missions data set curation activities;

Reviewed and commented by the ESA sensors experts team and by the LTDP WG.

• Completed three joint studies:

LTDP Initiatives and Standards Survey

LTDP User Requirements Study (FIRST)

Future Archive Technology Study (LAST)

0

1

2

3

4

5

6

7

8

9

10

Availability

Reliability

Maintenance

Migration

Interface

Performance

Security

Operations

Econ Aspects

Total ScoreHP StorageWorks P9000

Symmetrix DMX-4

Huawei Symantec OceanspaceTM S8100

Hitachi Universal Storage Platform® V

Fujitsu Storage Systems ETERNUS DX8400

IBM System Storage DS8800

Oracle Sun ZFS Storage 7740

Netapp FAS6200

EO International Coordination and Cooperation

Activities (2)

• Defined the initial data set to be preserved, including the related glossary.

• Several technical activities ongoing or planned:

LTDP/QA4EO Study

LTDP Architecture Definition Project

Interagency project for AVHRR coherent data series generation

(harmonized L0 data consolidation, processing, discovery & access).

SAFE Consolidation

Persistent Identifiers

• Guaranteed the information flow through workshops, web sites,

participation to conferences and LTDP related events.

European LTDP Framework

coordination:

LTDP Working Group (2)

EO International Coordination and Cooperation

Activities (3)

LTDP Web Sites

• Information flow guaranteed through workshops,

web sites, participation to conferences

• Institutional web site: http://earth.esa.int/gscb/ltdp/

Contains basic LTDP documents (e.g. Guidelines)

LTDP events and related material (e.g. Workshops)

• Technical Web Site:

https://wiki.services.eoportal.org/

Contains technical documentation and results of

LTDP Cooperation activities

Forum for technical discussions

Web Sites and communication

Back-up Slides

Based on a set of documents generated in coordination with European EO actors (LTDP WG)

• LTDP Definitions

• Preservation Workflow

• Preserved Data Set Content (PDSC)

• LTDP Common Guidelines

• E2E Consolidation Procedure

and dedicated ESA Documents:

ESA EO Data Preservation Scenario

1. ESA EO Datasets Appraisal Procedure 2. EO Datasets Preservation Requirements to be applied at ESA during all missions phases

ESA EO Data Preservation Scenario implementation

• Assessment of archived data and information holdings versus “PDSC” on-going for all ESA and ESA managed TPMs starting from historical ones and ERS-1/2.

• Data migration to latest available archive technology.

• Consolidation of Level-0 datasets (and associated information), and generation of two copies to be stored in two different locations; ingestion in tape libraries.

• Ensure basic discoverability via ESA catalogues, products generation and dissemination through the currently available processors and dissemination systems; reprocessing when feasible with particular focus on Climate Change Initiative requirements and needs.

• Progressive historical archives rationalization.

• Lessons learned used to update the Preservation Scenario.