Karabo: The European XFEL software framework

Burkhard Heisen2nd Collaboration Meeting of the European XFEL

April 24, 2013

Karabo: The European XFEL software framework


What I always start with… 2

Burkhard Heisen (WP76)

DAQdata readoutonline processingquality monitoring (vetoing)

SCprocessing pipelinesdistributed and GPU

computingspecific algorithms

(e.g. reconstruction)

Controldrive hardware and complex experimentsmonitor variables & trigger alarms

DMstorage of experiment & control datadata access, authentication authorization etc.

setup computation & show scientific results

allow some control & show hardware status

show online data whilst running

A typical use case:

Accelerator Undulator Beam Transport

Sample Injector

DM SC

ControlDAQ

Tight integration of applicationsEasy, efficient and comfortable usage of those


Outline – Follow the requirements 3


Preparation phase

Setup phase

Data taking phase

Data analysis phase

Phase Independent

Proposal phase

Publication phase

Clean up phase

Identify needs, Accept/Reject/Postpone several month before

Prepare configurations, custom developmentsseveral month – few weeks before

Verify setup, find signal, alignmentsfew days – few hours before

Data storage, sample switching/delivery

Remove custom h/w, bring s/w back to safe

Data analysis workflows, data management

Data cleanup


Phase independent Requirement

Distinction between different users with associated access restrictions

Solution Karabo is user centered, i.e. any access to

control, (meta-)data, or computing infrastructure requires login.

Karabo interacts with a central database managing all user data.

Access is controlled using a role-based system. Roles are assigned based on login context. The

context involves username, current time and current location.

The same user for example may have more rights during his beamtime and at a specific computer than at other places and during different times.

4


Central DB

1. Authorizes2. Computes context based

access role

usernamepasswordcurrentTimelocation

userIdaccessRolesession

GUI


Preparation phase 5


Requirement Setup of experiment configuration, scan macros, etc.

Solution XFEL system operators can prepare configurations and scan macros

according to information of the proposal. Configurations and macros are managed within Karabo’s central database. The control system is fully scriptable. The default language is Python. Control commands can be issued sequentially, or concurrently (e.g. move

several axes at the same time) Scripting can be done procedurally (python program) or interactively (IPython) Being Python any complicated constructs can be coded (i.e. n-D scans with

conditionals, etc.) Scripting can be done within the GUI Main API: get, set, monitor, execute, instantiate, kill


Preparation phase (special needs) 6


Requirement Implementation of new control devices and/or integration of new

algorithms to analysis pipeline Solution

Karabo is split into a core framework and extension plugins (devices). XFEL developers (or even the beamline users) can create new devices

for Karabo that can be injected into the runtime and will enable new control or analysis capabilities.

Several code templates exist for different categories of devices. Devices can be written in either C++ or pure Python (transparent to the

Karabo runtime). Devices impose a standard for self-description (properties, commands,

state-machine). GUI widgets and behavior such as auto-completion and interactive scripting is automatically generated using this information.

Devices can be deployed at any computer and are automatically made available and ready to use for any control-clients.


Setup phase 7


Requirement Verification of h/w and s/w functionality, alignments, signal search, etc.

Solution Sanity functionality within Karabo ensures correct technical setup (hosts

available, correct plugins loaded etc.) XFEL beamline scientist can use the GUI for inspecting parameters,

trendlines, histograms and images in conjunction with prepared macros to verify correct behavior

The GUI system uses the Device self-description to provide basic access to all features.

Custom panel creation needs no coding but is a process that is similar to the way you create a PowerPoint slide.

Panels can be loaded and saved per user. Sharing with other users is also possible.


Multi-purpose GUI system 8

Navigation

Custom composition area

Configuration

Notifications

Logging / Scripting console

Documentation

Kerstin Weger (WP76)


Property/Command composition 9

drag & drop

Display widget (Trend-Line)

Display widget

Editable widget



Property/Command composition 10

drag & drop

Display widget (Image View)

Display widget (Histogram)



Custom panels – PowerPoint like 11

Change between“Design/Live” mode

Open/Save panel view

Insert text, line, rectangle, …

Cut, copy, paste, remove item

Rotate, scale item

Group items

Bring to front/back



Data taking phase 12


Requirements Data collection, modification of variables, switching samples, online data

analysis Solutions

Karabo is tightly integrated into the data management system. Within the GUI, experiment runs can be started and stopped.

Whilst running, Karabo monitors and saves all (slow-)control data which can later be associated to the experiment raw data.

Raw data is streamed to the PC Layer, formatted and is saved in the online storage as HDF5 files.

Karabo scientific workflow system can directly be fed with the raw data allowing highly parallel, pipelined processing whilst data taking.

The GUI allows monitoring experiment statistics and data-analysis runs. Results of online data analysis can be used for tuning the experiment and

vetoing bad data (those won’t even travel to the offline storage)


Device (workflow) composition 13

Workflow node (device)

drag & drop

Draw connection



Data analysis phase 14


Requirements Pre-analysis on-site up to preliminary results, full analysis off-site

Solutions The same workflow system as already available during data taking can be

used in offline mode. Workflows are now fed by data as available from the offline storage (or - if still available - from the online storage).

Data analysis for the user starts with already calibrated data Data calibration is also implemented as a workflow (but maintained by XFEL

and accessing calibration parameters from Karabo’s central resources) The analysis system will provide common image processing algorithms

making also use of GPU technology Community algorithms can easily be integrated to Karabo and shared

amongst scientists. XFEL.EU plans to build up a central resource for state of the art analysis tools.


Processing workflows 15


IO whilst computing

Pixel parallel processing(one GPU thread per pixel)Notification about new data possible to obtain

GPU

CPU


Yes, it’s there! 16



Conclusions and timelines

Challenges are important to learn about detailed requirements pnCCD (combines control, DAQ, calibration and analysis) Beckhoff test stands Slice test (gives ideas about data throughput) 2-tile LPD, later ¼ Mpx LPD (imaging detector) CrystFEL integration (data analysis workflows)

First release of Karabo on 30th July Will cover what was mentioned in this talk Need more time to place the available technical solutions in

larger contexts and workflows

17



18

Thank you for your kind attention.


Documents

Karabo: The European XFEL software framework