Combined use of AMDA, CAA, QSAS and CL based on time-table exchange.

C. Jacquey, V. Génot, E. Budnik, M. Bouchemit, M. Gangloff, CDPP/CESRB. Cecconi, CDPP/LESIA

C. Perry, CAA team

T. Allen, A. Rochel, QSAS team

E. Penou, J.A. Sauvaud, CESR

Cluster Workshop, Tenerife, March 12, 2008

This talk is related to:-Poster (Jacquey et al.) this evening-Talks on CAA (C. Perry), QSAS (T. Allen) and CL (E. Penou)

Some aspects of the studies of space physics

We need to analyse measurements obtained by several observatories

We need to analyse multi-instrument data

We need to compare the observations to simulation results

We need to extract and treat a lot of data coming from multiple resources This takes time and energy

Dynamical processes The time is the very first parameter

The studies consists of analysis/characterisation of events treated in case studies or in statistical studies

Creating Time-Tables (or event list) is a common step of the work

Time-Tables (Event Lists)Time-Table produced by the users

Time-Tables coming from statistical studies (e.g. magnetopause crossing lists) sometimes made available to the community (e.g. the Frey’s substorm list)

Time-Table coming from search tool based on orbitographic or conjonction criteria (e.g. the Query tool of the SPDF/SSCWeb)

Time-Table coming from search tool based on mathematical criteria applied to the content of the data (e.g. AMDA of CDPP)

Time-Table coming from visual search tool (e.g. AMDA of CDPP)

Exchanging Time-Tables between data centres, services and tools? saving time and energy. Increase of the scientific return.

AMDAAutomated Mutiple Dataset Analysis

Web based service

Transparent access to data

AMDA local database (CLUSTER, ACE, THEMIS, GEOTAIL, WIND, …, ISEE)

External database (CDAWeb, MAPSKP, THEMIS/CESR, …)

Produces and exploits time-tables

Visualisation editor

Parameter editor

Download data

Conditional search

External data

Visual search

Time-Table manager

AMDA DOES NOT WORK WITH InternetExplorer and Safari

AMDA is public (http://cdpp-amda.cesr.fr)

Others

DARTS

SSLTHEMIS database

Others…

SPDF

SSCWeb Time-Tables

+ many servicesGAIA

Virtual ObservatoriesVSPO, VMO, VHO, VIRBO, …

CL

QSAS

Resources: data centres, services, toolsCAA

Validated high resolutionCLUSTER data

Plots and tools

Mirror THEMIS database

CDPP database

AMDA Time-Tables

CDPP

CDAWeb

Science Case:study and characterisation of the local

properties of the cross-tail current Based on CLUSTER data when it

encircles the neutral sheet.(i) Current density, profile, gradient(ii) Thickness, location, motion(iii) Distribution function (i+ and e-),

flows, …(iv) Turbulence/wave level and

spectra(v) …

Influence factor:(i) Solar Wind pressure(ii) IMF(iii) Geomagnetic activity:

- State of the ionosphere.- Oxygen abundance- Ring current intensity

Use Case: the main steps

Access and extractionof ACE/WIND/GEOTAIL

Interplanetary data

Access and extractionof ground

(AE, AL, SYM, DST) data

Access and extractionof modelisation or

simulation data(Tsyganenko, CCMC, …)

Selectionof the periods

of NS encirclingcondition

Usertools

Basic treatments

Form

attin

g

Data

mer

ging

Time shifting

ResultsCataloguesMapsModels

Selectionof the periods

of NS encirclingcondition

AMDA

Access and extractionof ground

(AE, AL, SYM, DST) data

Access and extractionof ACE/WIND/GEOTAIL

Interplanetary data

CDAWeb, CDPP, AMDA, DARTS, … CAA

Access and extractionof CLUSTER data

Access and extractionof CLUSTER data

Specifictools

Specifictools

CL

Generictools

Generictools

CL, QSAS or AMDA

Creation of time-tables on AMDA/CDPP

TT1, time-table listing the periods when the conditions are fulfilled

TT2, extended time-table

Conditional search

Visualisation

Visual search

Managing Time-Tables

Text orXML VOTable

Parameter editor

Data dowload

Data merging

AMDA-Automated search

-Visual search Time-Tables

CAAValidated high resolution

CLUSTER data

CL

QSAS

Standardisedsub-databases

Other?

CDAWeb

DARTS

CDPP/CESRTHEMIS database

SSLTHEMIS database

Other?

Time-Tables

First step: producing sub-databases corresponding to the time-tables

Time-Tables

Data, parameters Data, parameters

User tools

How to do that?

Standardisation of the time-tables

Standardisation of the organisation of the sub-databases to be produced

Interface implementation on CAA for reading the time-table and producing sub-databases

Interface implementation on the tools QSAS and CL

Standardisation of the time-table description

Step 2: catalogues

T_Start T_End P1 … PN Q1 … QM …

T_Start T_End P1 … PN Q1 … QM …

T_Start T_End P1 … PN Q1 … QM …

T_Start T_End P1 … PN Q1 … QM …

T_Start T_End P1 … PN Q1 … QM …

AMDA,SSCWeb or

other services

CAA, CDAweb,CDPP or otherdata centres

QSAS, CL,AMDA or

other tools

Time-table:T1 – T2T1 – T2….

AMDA

USER

Computationof P1, ..,PN

CAA

Catalogue:T1, T2, P1, …, PN

Sub-database

Catalogue:T1, T2, P1, …, PN, Q1, …,QN

Future: Archiving the catalogues for availability to the community

QSASor CL

CDPP

CDAweb

DARTS

USER

Catalogues and associated tools available at the data centres

ConclusionExchange of standardised Time-Tables between data centres, services and tools should allow the researcher

- to save large amount of time and energy- to exploit more extended data amount- to increase the scientific return of the data

First step: extraction and exploitation of sub-databases corresponding to the Time-Tables

Second step: creation and exploitation of catalogues, similarly to astronomy and solar physics

This project starts with CAA, QSAS, CL and AMDA, but of course, other new participant is welcome

The specifications of this project need the input of the potential users Poster this evening in order to collect suggestions and ideas.

AMDA: contact: amda@cesr.frStable version: http://cdpp-amda.cesr.frTest version: http://iapetus.cesr.fr/AMDA_PRETEST/

Use casesUse case = simulation of studies which could be performed by the researchers in order to:

identify and analyse their needsspecify the user requirementsImagine, define and specify the system to be developed

The developers of the system need the input from the community

Obvious requirement: standardisationStandardisation of the data format

CEF, CDF, NetCDF, (FITS)But beyond that, standard way for representing data of different classes (particle , waves, Time-Tables, …)

Standardisation of the data description SPASE (Space Physics Archive Search and Extract,

http://www.spase-group.org/ )

Step 2: sub-database extractionOn CAA, CDAweb, DARTS, CDPP, AMDA, …

Requirements:

Standard format for time-tables

Interface for reading and managing the time-tables

User interface for selecting data

Optionally, basic on the fly treatment (e.g. data merging, SW propagation shifting, …)

Standardised management of the multiple file outputFilenameDescriptorProduct recovering

download

Pi-Pj/(Xi-Xj)

Step 3: plot production on CL, QSAS, AMDA

INPUT:▪ time table▪ corresponding sub-database

Manipulation of the time-tableUser edited plotLaunch in batch mode

Requirements: Standardisation (i) of the time-table and (ii) of the data Plot editor Batch mode Standardised management of the plot files

▪ Filename▪ Descriptor▪ Product recovering

AMDAAutomated Mutiple Dataset Analysis

Created by the CDPP: http://cdpp.cesr.fr/

Stable version: http://cdpp-amda.cesr.fr/Testing version: http://iapetus.cesr.fr/AMDA_PRETEST/

Contact: amda@cesr.fr

Example: event search

Old fashion: “paper” search

AE, AL ISEE-dataIMP-8

AE, AL

ISEEMAGIS

EE-el

ectro

n

IMP8

MAG

IMP8PLA

Current fashion: web

Event search takes time and energy

CDAWebSPDF

UCLA

CAA

CDPP

And others

Automated Mutiple Dataset Analysis

AMDA: the goals

Multiple Dataset: the study of the (multi-scale) dynamics of plasma objects requires to perform the integrated analysis of multi-point and multi-instrument data

Automated (or semi-automated) Analysis: help forSearch, characterisation, classification of events Extensive exploitation of the vast database (statistics)“historical” studies (example: through solar cycles)Building cataloguesBuilding virtual constellationsTime-Table (EventList) production

AMDA, fact sheet

• Built in IDL, C, FORTRAN, Javascript

• Works on standardised and “simple” data, NetCDF

• Includes models (Tsyganenko, Shock, MP, NS, …)

• Uses external data (now from CDAWeb and soon from MAPSKP, IWF, THEMIS, …)

• Forum for user feedback

• Twiki for information (development, bugs, …) exchange

• Already in use for scientific studies

AMDA DOES NOT WORK WITH InternetExplorer and Safari

AMDA is public

FunctionalitiesAutomated access to data

Access to and on-the-flight ingestion of external data (test version only)

Data merging

Data download

Data visualisation

User edited parameter computation on the data

Automated conditionnal search

Visual searchProduction of Time-Tables

LocalDatabase

AMDA

System

CDAWeb

CDPP

MAPSKP

AMDA/”External Data”

others

Needed development on AMDA

Temporal functions (mean, regression, dispersion estimator, …)

User pluggins

Continuous SW propagation shifting

Scatter plots, map plots, …

Your suggestion

Your suggestion

Your suggestion

Your suggestion

….

Conclusion and questions

CAA, QSAS, CL and CDPP teams are starting a collaboration for providing to the researchers ways to save time and energy for performing their studies

Other services and database are invited to join this effort

We need input, comments, suggestions from the potential users.