ITER Unified Data Access UDA - Nucleus Documents/Data... · 11th IAEA Technical Meeting on Control, Data Acquisition, and Remote Participation for Fusion Research 2 , Greifswald,

ITER Unified Data Access UDA

Rodrigo Castro1, Yury Makushok2, Lana Abadie3, Simon Pinches3, David G. Muir4, Jonathan Hollocombe4, Jesús Vega1, Joan Faig5, Gregorio Garcia2

1. Laboratorio Nacional de Fusion, CIEMAT. Madrid. Spain

2. SGENIA Madrid. Spain 3. ITER Organization, St Paul lez Durance, France

4. EURATOM/UKAEA Fusion Association, Culham Science Centre, Oxfordshire UK

5. INDRA Sistemas, S.A., Madrid, Spain

2 11th IAEA Technical Meeting on Control, Data Acquisition, and Remote Participation for Fusion Research,

Greifswald, 2017

Index

ITER Data archiving types

UDA Architecture

UDA Components

UDA Index DB

UDA Indexer

UDA Server

UDA Client

UDA tools


Greifswald, 2017

Simplified ITER data archiving model

Data Archiving

DAN

(STREAMS)

PON (EPICS) SDN

Monitoring Data Analysis

Simulation


Greifswald, 2017

DAN Archiving

DAN Network

HDF5 Storage Files

Archive Writer Archive

Writer

DAN Protocol

DAN Streamer

DAN Client

DAN Client

Configuration

DAN API

DAN Streamer

DAN Publisher

DAN Publisher

DAN Publisher

HDF5 Storage Files

Archive Writer Archive

Writer

DAN Archiver


Greifswald, 2017

DAN Archiving

Multichannel

Multidimensional and complex data types

Metadata

Stream level

Channel level

Datablock level

Ready for video


Greifswald, 2017

SDN Archiver

SDN Subscriber Thread (1

topic)

SDN ARCHIVER

dan_writer

thread

Topic Types XML

HDF File

SDN


topic)

dan_writer

thread

HDF File


topic)

dan_writer

thread

HDF File


Greifswald, 2017

PON Archiving based on HDF5

PON BEAUTY

CSS Archiver

Channel Access

PVs

HDF5 BackEnd

HDF5

file HDF5

file HDF5

file


Greifswald, 2017

Why HDF5 files?

Self Description Data and metadata is included

• Complete self coherent unit

Huge file sizes

Single Writer – Multiple Reader

Optimized direct data access to time segments

Data may be appended without copying or redefining its structure

File versions are supported

Platform independent

Supported from a wide set of programming languages

Complete utilities set

Robust

SWMR

Challenges in ITER Huge number of variables / file High sampling rate data


Greifswald, 2017

UDA Architecture

UDA Index DB

Data Access Client

UDA API

UDA Request

Data

UDA Indexer

UDA server UDA

Indexer UDA Indexer

Pulses DB

DAN PON SDN

Data Archiving Repository

UDA Index


Greifswald, 2017

UDA Index DB

First level catalogue of archived data

Index information is only got from archiving files

Variables

Storage location

Timestamps

File state

Data quality


Greifswald, 2017

UDA Indexer

Responsible of

Detection of data storage events

Index data extraction from archiving files

Index data injection into UDA index DB

Requirements

Online and offline indexing

Performance

• Multiple files in parallel

Scalability

• N number of indexers in parallel


Greifswald, 2017

Distribution and scalability

UDA Indexer UDA

Indexer

UDA Indexer

DAN PON SDN


UDA Indexer

UDA Index DB


Greifswald, 2017

UDA Indexer Architecture

File

system

watcher

UDA Indexer

Indexer

Thread Indexer

Thread Indexer

Thread Indexer

Thread

Unresolved

files

… UdaDBInjector

Characteristics

extractor/detector

Data Archiving

Reader

Indexer Thread

Std

Input

UDA Server


Greifswald, 2017

IDAM (Integrated Data Access Management)

Wide experience in MAST: David. G. Muir, J. Collohombe

Modularity

Extension to new data archiving formats

Ready for multiple data repositories

Powerful requests ready to be extended with

• new functionalities

• request criteria

Data transmission protocol

Multiplatform data serialization

Complex data types and multidimensional data support

MDSPlus interface


Greifswald, 2017

UDA Server Architecture

UDA Request Solver

IDAM Server

UDA Request

Interpreter

IDAM protocol

UDA Request

UDA Response

Packer Pulses

Client

Library

UDA

Index

Client

library

UDA Response

Data

Archiving

Data

Access



Greifswald, 2017

UDA Request Syntax (I)

UDA::getData(<variable>, <interval>, <optional_parameters>)

<variable>

• variable=/ITER/<variable_name><field>

<interval>

• startTime=<time>, endTime=<time>

• refTime=<time>, startTime=<relTime>, endTime=<relTime>

• shot=<pulse>

• shot=<pulse> , startTime=<relTime> , endTime=<relTime>

<optional_parameters>

• Return timestamps and data type

• Decimation

• Chunking parameters


Greifswald, 2017

UDA Request Syntax (III)

UDA::getData(“variable=/ITER/MAG-CYSI-FCT0:sensor1D-1[4], startTime=1484264684735399104 , endTime= 1484264714735399104, decSamples=10000, retType=double”)

UDA::getData(“variable=/ITER/MAG-CYSI-FCT1:event3/conv1, pulse=3542121, startTime=10M , endTime= 15M, retType=long”)


Greifswald, 2017

UDA Server Support of all ITER data types

Reasonable time limit for reading data: 3 seconds

Extensible for new repository formats

Request based on timestamp interval or pulse number

Metadata and payload

Multidimensional data and complex data types

Data chunking

Implementation in C

MDSPlus interface

Supports load-balancers

UDA Client


Greifswald, 2017

UDA Client Application

UDA Client

UDA Client API UDA Request Solver

UDA Request

IDAM Client IDAM Server

UDA Request

Interpreter

. . . IDAM protocol

UDA Request


Greifswald, 2017

UDA Client API

Ready for C, C++, Python and JAVA

Simple interface and automatic types conversions

Multithread

Supports continue data chunk processing

Examples

UdaClientReaderCpp UCR("localhost", 3090);

lastPulse = UCR.getLastPulse();

pv_list = UCR.getVariableList("MAG-CYSI-.*");

fields_sdn = UCR.getFieldsList("/ITER/MAG-CYSI-FTC0:sensor-1");

ts = UCR.getPulseTimeStamps(PulseID);

UCR.resetAll();


Greifswald, 2017

UDA Tools

Set of tools ready to use for UDA (without developing)

uda-get-last-pulse

uda-get-pulse-info

uda-get-pulses

uda-get-variables

uda-get-var-fields

uda-plot-tail

uda-python-get-data

uda-to-isotime

uda-to-nstime

uda-get-pulse-by-time

uda-itool>

First version of interactive terminal tool

Autocomplete / online help / settings

Adapted version of jScope for UDA

uda-jScope (with UDADataProvider plugin)


Greifswald, 2017

What can I find in CCS now?

Data Archiving

DAN Archiving

• DAN API

• DAN Archiver service

SDN Archiving

• SDN Archiver

PON Archiving

• BEAUTY HDF5

HDF5 utilities


Greifswald, 2017

What can I find in CCS now?

UDA

UDA Index service

• UDA Index DB

• UDA Indexer

Pulses DB and pulses lib (prototype for UDA)

UDA Server

• With MDSPlus interface

UDA Client

• API ready for DAN, PON and SDN data

• API with interface for C, C++, JAVA, Python

Thank you

Rodrigo Castro1, Yury Makushok2, Lana Abadie3, Simon Pinches3, David G. Muir4, Jonathan Hollocombe4, Jesús Vega1, Joan Faig5, Gregorio Garcia2

1. Laboratorio Nacional de Fusion, CIEMAT. Madrid. Spain

2. SGENIA Madrid. Spain 3. ITER Organization, St Paul lez Durance, France

4. EURATOM/UKAEA Fusion Association, Culham Science Centre, Oxfordshire UK

5. INDRA Sistemas, S.A., Madrid, Spain