Euclid Science Capabilities · (DPRD=GDPR) SOC Science Implementation Plan (SOC SIP) EC Science Implementation Plan (EC SIP) EC SGS WP Breakdown and Description (WPBD) EC SGS Documentation

Euclid Mission Meeting F.Pasian Bologna , 7-8 September 2011 1

EuclidConsortium

Ground Segment

Fabio Pasian, Marc Sauvage, Christophe Dabin

and EC-SGS PO Support Team, System Team, OU Leads, SDC Leads

The presented document is Proprietary information of the Euclid Consortium. This document shall be used and disclosed by the receiving Party and its related entities (e.g.

contractors and subcontractors) only for the purposes of fulfilling the receiving Party's responsibilities under the Euclid Project and that identified and marked technical data shall

not be disclosed or retransferred to any other entity without prior written permission of the document preparer.


EuclidConsortiumSummary

• Requirements (and documentation) flowdown

• Science Operations Concept

• EC SGS development

– data-centric approach

– drivers

– data processing functions, OUs, SDCs

– architecture

– interfaces

– managerial aspects (WBS and WPs, reporting, staffing, responsibilities)

• SGS as a whole: EC SGS and SOC collaboration

• Next steps







– drivers


– architecture

– interfaces



• Next steps


EuclidConsortiumRequirements flowdown

reference to presentations by

Jérȏme Amiaux and Philippe Gondoin


EuclidConsortiumDocumentation flowdown

Science

Management

Plan

(SMP)

Science

Operations

Concept

(SOCD)

Science

Implementation

Requirements

(SIRD)

Science

Requirements

Document

(SciRD)

Payload

Elements

Requirements

(PERD)

Mission

Requirements

Document

(MRD)

Data

Processing

Requirements

(DPRD=GDPR)

SOC Science

Implementation

Plan

(SOC SIP)

EC Science

Implementation

Plan

(EC SIP)

EC SGS WP

Breakdown and

Description

(WPBD)

EC SGS

Documentation

Tree

EC SWGs

EC SGS







– drivers


– architecture

– interfaces



• Next steps


EuclidConsortiumThe Euclid Ground Segment

Ground Station:

at least one (2)

DTCP = 4 hrs

MOC: monitors s/c

health+safety, and

instrument safety,

controls s/c attitude,

handles TM/TC for s/c

and instruments

SOC: executes the

planned surveys, i/f to

MOC, daily reports,

Level 1, Level Q,

manages EMA and

operates ELA

Data processing

SDCs: perform

science processing

and create science-

ready data products

(Levels 2 and 3) Science support SDCs:

provide simulated data

(Level S) or reprocessed

external data (Level E)

IOTs: maintain

instruments, monitor

their health, trend

analysis, and

production of weekly

instrument reports

Instrument-oriented

SDCs: host the IOTs

and are in charge of

instrument calibration

activities (Level 2)


EuclidConsortiumMission Planning Concept

MOC: provides

planning information,

including a predicted

orbit, planned events at

s/c and GS level, and

tools/data to

correctly plan the

spacecraft pointing

SOC: implements

survey strategy:

generates long term

plan, derives a series

of daily/weekly/monthly

observation sequences

PS: provides

survey strategy

IOTs: maintain a

routine calibration plan

which is delivered to

the SOC for execution;

submit unplanned

calibration requests to

SOC as observation

requests


EuclidConsortiumProcessing Levels and data products

Data Processing

Levels / Data

Levels

Data type after the

processing (output)

S/W

Development

Responsibility

Data

Production

Responsibility

Level E Euclidised External data SDC SDC

Level S Simulations SDC SDC

Level 1 Unpacked and edited

telemetry

SDC SOC

Level Q Quick-release data SDC SOC

Level 2 Instrument signature

removed, calibrated data

SDC SDC

Level 3 Science-ready data SDC SDC

SGS processing Levels, indicating data products and responsibilities

The software of Level 1 and Level Q are developed by the SDCs in charge of

Level 2 and delivered to SOC for integration within its premises.

Level Q is a subset of Level 2, integrated and run at SOC.


EuclidConsortiumData Releases

From To Data Products Notes

SOC EC Level 1 Daily

SOC EC Level Q Daily

SOC Public Level Q ELA releases: First release of 1 year of data at

14 months after start of routine operations,

subsequently every 12 months

EC

(SDCs)

EC

(EMA)

Level 2,

Level 3

periodically, internal

releases

EMC SOC Level 2,

Level 3

During the proprietary period, populating the

ELA

SOC Public Level 2,

Level 3

ELA releases: First release 26 months after

the start of the routine operations,

subsequently every 12 months.

Internal and external data releases


EuclidConsortiumData Products (defined in SMP)

• Level 1 data:

– Raw VIS and NISP images

– Processed housekeeping telemetry + associated ancillary information such as pointing history files

• Level 2 data:

– Calibrated +co-added images from VIS and NISP – validated for cosmology analysis

– PSF model and optical distortion maps

– Co-added spectra

• Level 3 data:

– Catalogues (including redshift, ellipticity, shear, etc)

– Dark matter mass distribution

– Shear and galaxy correlation functions and covariance errors

– Additional science catalogues

– Ground based information which was used in the derivation of the data products

• Transients:

– Transient events data products: include derived transient category (e.g. supernova, solar system object, etc) and brightness, target position and possible finding chart.

• Level Q data:

– Products defined so that they are suitable for most purposes in Astronomy, except for the main cosmological goals of the mission.







– drivers


– architecture

– interfaces



• Next steps


EuclidConsortiumData-centric approach: the EMA

• Key features of Euclid:

– amount of data that the mission will generate,

– heavy processing needed from raw data to science products,

– accuracy and quality control required at every step

• Data at the centre of the SGS

• The design of the SGS based on a data-centric approach: all SGS operations logically revolve around the Euclid Mission archive (EMA)

– a logical, rather than physical entity giving access to all mission-related analyses

– a central storage and inventory of the data products and their metadata including quality control

– orchestration of data exchange and metadata update involving SOC+SDCs through the EMA is performed by a monitoring and control function


EuclidConsortiumEMA → ELA

• Euclid Legacy Archive (ELA): public archive containing the data products to be distributed to the scientific community.

• The ELA is logical subset of the EMA

• The criteria for availability in the ELA are defined by the EST and the EMC, and are implemented in the EMA.

• After approval of the data products for public release, the ELA shall be delivered to ESA

• ELA the unique distribution channel of Euclid data products to the scientific community.


EuclidConsortiumDrivers for the SGS

• Instrumental drivers

– in order to achieve an optimal data processing by the SGS, close collaboration between the SGS and the IDTs (→ IOTs)

– core of the IOTs made up with scientists experienced both in the instruments and in the systems used to process the data

• Data drivers

– core Euclid science cannot be achieved without ground-based survey projects (DES + Pan-Starrs, LSST as a possibility)

• Simulation drivers

– SGS integration with Science and Instrument simulations (E2E)

• Interfaces and data model

– internal data interfaces handled through single + consistent DM

• Optimisation

– a number of processing steps or functions need to be performed within different instances → need to avoid duplication

– in particular, transversal (global) data quality tools (e.g. DQM) do not overlap with other elementary quality checking steps


EuclidConsortiumData processing functions


EuclidConsortiumSWGs, OUs and SDCs

• Science Working Groups

– turning science objectives into requirements placed on the pipeline products and performances

– verifying that the requirements are met (define V&V procedures)

• Organisation Units

– providing the algorithmic definition of the processing to be implemented by the SDCs and validate the implementation

• Science Data Centres

– implementing the data processing pipelines as specified by OUs

– procuring local h/w and s/w resources

– different activities:SDC-DEV (development – i.e. transforming algorithms into robust code)

SDC-PROD (integration on local infrastructure, production runs of pipeline)

• (individual Euclid scientists may belong to more than one of the above groups)







– drivers


– architecture

– interfaces



• Next steps


EuclidConsortiumLogical architecture


EuclidConsortiumData flow


EuclidConsortiumSGS design concepts

• Minimisation of data transfers

• A concept of distributed data products storage (bulk data products are stored at least twice among the SDCs and metadata are indexed inside the EMA) avoiding the unnecessary movement of huge amounts of data between the SDCs and the EMA/SDC

• A single EMA metadata repository which inventories and indexes all metadata (and corresponding data locations)

• A concept of software layers inside the SGS: metadata access layer (query/retrieve), data product access layer (open, read/write, get info,..), data processing layer

• A design allowing the flexibility to implement new software pipeline releases without redesigning the EMC SGS architecture

• In the distribution of work on the SDCs, we are already considering the data transfer aspects. For the functions/OU that will use or generate very large amounts of data, we have a minimum number (2 for redundancy) of SDCs in order to minimize the constraints on transfer.


EuclidConsortiumSGS development principles

• Simplicity of system design

• Component-based software engineering

– Modular approach: each module can be developed independently and wrapped in the language possibly adopted as standard (C/C++ and PYTHON as script language envisaged)

• Virtualisation

– executing pipeline software on virtual machines and separating pipeline software from the underlying hardware resources

• A common data model for each module, application and pipeline

• Persistence of the data model objects

– each frame in the data processing chain is described by the common data model and saved in the EMA along with all the parameters used for the data processing

• In the SIP we have described a plan to migrate from a logical architecture to a physical architecture







– drivers


– architecture

– interfaces



• Next steps


EuclidConsortiumInterfaces

• Internal to EC SGS

– data interface among SDCs based on common data model

– OU-SDC coordination (EC SGS Organisation Group)

– interface between IOTs

• Internal to EC

– SWG-OUs-SDCs

– SWG re. science simulations

– interface with Instrument Development Teamssmooth transition between phases: implementation → operations

– interface related to Calibration

– interface with Communications

• External to EC

– coordination with SOC

– OSEWG (Operations and System Engineering WG with MOC+SOC)







– drivers


– architecture

– interfaces

– managerial aspects (WBS and WPs, reporting, staffing, schedule)


• Next steps


EuclidConsortiumWork Breakdown

• WBS is based on Product Tree (i.e. the “breakdown of the project into successive levels of hardware and software products or elements, articulated to perform the functions identified”)

• Top-level WPs assigned to countries and institutes

• OUs WPs mapped onto data processing functions

• SDC WPs have similar structure (SDC-DEV, SDC-INT, SDC-PROD)


EuclidConsortiumCountries interests and responsibilities

• France: SGS Scientist, SysTeam Lead (deputy SGSM), OU-VIS (coord), OU-SIR (dpty), OU-SIM (dpty), OU-MER (dpty), OU-SPE (coord), OU-LE3 (coord), SDC-FR

• Italy: SGS Manager, PO Team, OU-NIR (coord), OU-SIR (coord), OU-MER (coord), OU-SPE (dpty), OU-LE3 (dpty), SysTeam contributions, SDC-IT

• Germany: OU-EXT (coord), OU-MER(dpty), OU-SHE (dpty), SDC-DE

• UK: OU-VIS (dpty), OU-LE3 (dpty), OU-SHE (coord), SysTeam contributions, SDC-UK

• Spain: OU-SIM (coord), OU-PHZ (dpty), SDC-SP

• Switzerland: OU-SHE (dpty), OU-PHZ (coord), SysTeam contributions, SDC-CH

• Netherlands: OU-NIR (dpty), OU-EXT (coord), SDC-NL

• Finland: OU contributions, SysTeam contributions, SDC-FI

• Norway: OU contributions

• Austria: OU contributions

• Romania: OU contributions

• Denmark: OU contributions

• Canada: OU contributions, SysTeam contributions (TBD)


EuclidConsortium

Each OU produces

algorithms which

are integrated and

executed in an SDC

(Science Data

Center)

OUs are transnational

An organization based on the

decomposition in Organization Units

(OU) corresponding to a subset of

overall EUCLID Data Processing

OU-VIS OU-NIROU-SIR OU-EXTOU-MEROU-PHZ OU-SPEOU-LE3OU-SHE

SOCESAC

MOCESOC

scientific

community

EMADDS

ELA

Ground

Station

Euclid

External data

(PanStarrs, DES, ...)

SDCSDC SDCSDC SDC

OU-SIM

VIS Imag Nir ImagNir Spectro Ext DataEuclidisationSpectro MeasLevel 3Morpho & ShearPhot Red Sh.

Simulation

SDC SDC

OU coordinator

OU Deputy Coordinator

EMA is built by EC and managed by SOC

ELA is an EMA subset, is a formal EC delivery

to ESA and is distributed by SOC

SGS: overall view

SDC


EuclidConsortiumManpower by function


EuclidConsortiumManpower by country


EuclidConsortiumEC SGS management (detail)

EC SGS

SDC leadSDC lead...

OU-1OU-1OU-1OU-1SDC WP leadsSDC WP leads

SDC leadSDC lead

OU-1OU-1OU-1OU-1SDC WP leadsSDC WP leads

EC SGS EC SGS

ManagerEMC SGS EMC SGS

Scientist

EC LeadEC Lead

(ECB Chair)

EC SGS

Project

Office

reporting

ESA

technical interaction reporting

EC SGS EC SGS

Scientist

OU leadOU lead...

OU-1OU-1OU-1OU-1OU WP leadsOU WP leads

OU leadOU lead

OU-1OU-1OU-1OU-1OU WP leadsOU WP leads

reporting

reporting

PA/QAPA/QA

ControllerController

ConfigurationConfiguration

EC SGS EC SGS

System Lead

EC SGS

Organisation

Group(OU and SDC

Leads)

EC SGS

Organisation

Group(OU and SDC

Leads)

System TeamSystem Team


EuclidConsortiumSchedule

(time axis not to scale)







– drivers


– architecture

– interfaces



• Next steps


EuclidConsortiumInterfaces with SOC

• Interfaces, collaboration & coordination ESA ↔ EC (SOC ↔ EC-SGS)

• Activities aimed at achieving a single SGS

• Common Tools and Facilities provided by SOC:

– Requirements management

– Document management (Livelink)

– Document wiki

– Planning and schedule tracking

– Software issue tracking and source code configuration control

– Data model definition (TBD)

– Instrument Parameter Database

• Participation in System Team (shared responsibilities)

• Joint planning: product tree, documentation tree, WBS, risk management, schedule, configuration control, PA/QA, SGS V&V

• OSEWG (Operations and System Engineering WG)







– drivers


– architecture

– interfaces



• Next steps


EuclidConsortiumDuring the Definition Phase ...

• Prepared EC SIP (+WPBD) → coherent and consistent plan for EC SGS

• Contributed to the creation of SOCD and SIRD

• Commented on SOC SIP

• Successfully closed actions related to recommendations


EuclidConsortiumSteps to be undertaken

• Flowdown of global science requirements

– Qualitative → quantitative requirements in DPRD

• Prepare detailed flowdown of DPRs onto data processing functions

– “Flowup” of DPRs to lower-level processing steps

– Increased detail on interactions and inter-relations among OUs

– Detailed DPRD-to-SIP matrix (assignment to finer-grained WPs)

• External spectroscopic data (surveys, follow-up)

• Analysis of needs for data processing tools common across SDCs

• Revise task assignments, negotiate with new partner countries

• Consolidate distribution of costs (input to MLA)

• ...


EuclidConsortium

Thank you

for your attention


EuclidConsortiumEC requirements flowdown


EuclidConsortiumQuality Control

Name Data

processing

level

Description Product

location

Quality Control 1 (QC1) Level 1

Level Q

Data integrity (file level) and Quick

Look Analysis

SOC

Quality Control 2 (QC2) Level 2

Level E

Instrumental fingerprints removal

(instrumental calibration)

SDCs

Quality Control 3 (QC3) Level 3 Data product quality checkup SDCs

Quality Control 4 (QC4) Level 3 Final scientific product quality

checkup

EST

Quality control domains

Quick quality controls are done by the SOC, more elaborate quality controls involving full pipeline

reductions and calibrations (e.g. effective tracking stability over different dithers of a single pointing field)

are done by the IOTs. All these quality controls are critical for the success of the mission.

The results of the quality control are stored in the EMA so that each participant can be supplied with

quality information. All Quality Control information is shared over all participants


EuclidConsortiumImplementation phases


EuclidConsortiumTop-level WPs

WP 4-1 Management

WP 4-1-01 EC SGS Management INAF-OATs (Italy) F. Pasian

WP 4-1-02 Project Control INAF-OATs (Italy) TBA

WP 4-1-03 PA/QA INAF-OATs (Italy) C. Vuerli

WP 4-1-04 Configuration management INAF-OATs (Italy) O. Mansutti

WP 4-1-05 Validation and Verification (V&V) INAF-OATs (Italy) C. Vuerli

WP 4-1-06 System Tests INAF-OATs (Italy) M. Frailis

WP 4-2 SGS System Engineering

WP 4-2-01 System Team Coordination CNES (France) C. Dabin

WP 4-2-02 EC SGS performance IAP (France) JM. Delouis

WP 4-2-03 Abstraction Layer ETH (Switzerland) M. Melchior

WP 4-2-04 Monitoring & Control CNES (France) C. Dabin

WP 4-2-05 Data Modelling IAP (France) JM. Delouis

WP 4-2-06 Testing tools and procedures INAF-OATs (Italy) C. Vuerli

WP 4-2-07 EMA metadata RUG (Netherlands) R. Williams

WP 4-2-08 EMA data transfer RUG (Netherlands) R. Williams

WP 4-2-09 Data & Processing Orchestration CNES (France) M. Poncet

WP 4-2-10 Common Tools RoE (UK) K. Noddle

WP 4-2-11 Data Quality Control Tools Univ. Helsinki (Finland)

INAF-O.A.Capodimonte

E. Keihänen

M. Brescia

WP 4-3 OUs

WP 4-3-01 OU-VIS: Vis imaging IAP (France)

MSSL (UK)

C. Grenet

N. Shane

WP 4-3-02 OU-NIR: NIR imaging INAF-OAR (Italy)

Uni Leiden (Netherlands)

A. Grazian

R. Bouwens

WP 4-3-03 OU-SIR: NIR spectroscopy INAF-IASFmi (Italy)

LAM (France)

M. Scodeggio

C. Surace

WP 4-3-04 OU-EXT: Non-Euclid data RuG (Netherlands)

USM (Germany)

G. Verdoes-Kleijn

J. Mohr

WP 4-3-05 OU-SIM: Simulations IEEC (Spain)

CPPM (France)

S. Serrano

A. Ealet

WP 4-3-06 OU-MER: Merging INAF-OAR (Italy)

IAS (France)

MPE (Germany)

A. Fontana

M. Douspis

J. Koppenhoefer

WP 4-3-07 OU-SPE: Spectral-z Measurements LAM (France)

INAF-OABo (Italy)

O. Le Fèvre

M. Mignoli

WP 4-3-08 OU-SHE: Shear IfA Edinburgh (UK)

EPFL (Switzerland)

Uni Bonn (Germany)

A. Taylor

F. Courbin

T. Schrabback

WP 4-3-09 OU-PHZ: Photo-z ISDC (Switzerland)

IEEC (Spain)

S. Paltani

F. Castander

WP 4-3-10 OU-LE3: Level 3 Data CEA (France)

Uni Trieste (Italy)

UCL (UK)

J-L. Starck

S. Borgani

F. Abdalla

WP 4-4 SDCs

WP 4-4-01 SDC UK Edinburgh (UK) K. Noddle

WP 4-4-02 SDC Switzerland ISDC (Switzerland) P. Dubath

WP 4-4-03 SDC France CNES (France) M. Poncet

WP 4-4-04 SDC Italy INAF-OATs (Italy) A. Zacchei

WP 4-4-05 SDC Netherlands RuG (Netherlands) O.R.Williams

WP 4-4-06 SDC Spain TBD (Spain) C. Neissner

WP 4-4-07 SDC Germany MPE (Germany) J. Koppenhoefer

WP 4-4-08 SDC Finland Univ. Helsinki (Finland) E. Keihänen

Documents

Euclid Science Capabilities · (DPRD=GDPR) SOC Science Implementation Plan (SOC SIP) EC Science Implementation Plan (EC SIP) EC SGS WP Breakdown and Description (WPBD) EC SGS Documentation