Developments and applications of DAQ framework DABC v2 Jörn Adamczewski-Musch, Nikolaus Kurz, Sergey Linev GSI / Experiment Electronic

Development and applications of DAQ framework DABC v2

Developments and applications of DAQ framework DABC v2http://dabc.gsi.de

Jrn Adamczewski-Musch,

Nikolaus Kurz,

Sergey Linev

GSI / Experiment Electronic

CHEP 14.04.2015

1

DABC core functionality

Multi-thread, multi-node applications

Zero-copy transport approach

full support for InfiniBand VERBS

advanced sockets treatment with select()

Flexible plugin interface

MBS, HADAQ, ROOT, Go4, EPICS, DIM, FESA

http for slow control

JavaScript ROOT as web GUI

In development since 2007

v1.0 in 2009, now v2.7

14.4.2015

Sergey Linev, DABC v2, dabc.gsi.de

2

Local communication

14.4.2015


3

Receiver

Sender

memory pool

buffer

buffer

buffer

buffer

buffer

buffer

File,

socket,

PCIe,

...

Consumer

Pre-allocated memory pool

Reference counter for each buffer

Buffers queue between modules

No memcpy between sender and receiver

queue

Remote communication

14.4.2015


4

Transport

Sender

memory pool

buffer

buffer

buffer

buffer

buffer

buffer

Producer

Receiver

Transport

memory pool

buffer

buffer

buffer

buffer

buffer

buffer

Consumer

node1

node2

Same approach as in local

No memcpy for InfiniBand

Different communication patterns:

1 1, 1 N, N 1, N x M

queue

queue

InfiniBand tests with DABC

fat tree topology

~800 nodes with QDR IB

12 spine and 34 leaf switches

14.4.2015


5

ibspine01

ibswitch01

01

02

03

04

05

06

07

08

ibswitch07

49

50

51

52

53

54

55

56

ibswitch02

09

10

11

12

13

14

15

16

ibswitch03

17

18

19

20

21

22

23

24

ibswitch04

25

26

27

28

29

30

31

32

ibswitch05

33

34

35

36

37

38

39

40

ibswitch08

57

58

59

60

61

62

63

64

ibswitch09

65

66

67

68

69

70

71

72

ibswitch10

73

74

75

76

77

78

79

80

ibswitch11

81

82

83

84

85

86

87

88

ibspine02

ibspine03

ibspine04

ibswitch06

41

42

43

44

45

46

47

48

ibswitch12

89

90

91

92

93

94

95

96

all-to-all traffic pattern

1.25 TB/s or 80% of available bandwidth

Results from May 2011

PEXOR/KINPEX plug-ins

14.4.2015


6

PEXOR (KINPEX): Pci-EXpress Optical Receiver

Lattice (Kintex) FPGA

4 lane PCIe

4 high speed SFP connectors

FEBEX (ADC),

POLAND (QFW),

TAMEX (TDC)

various frontends

Detector

GOSIP protocol,

optical chains

pexorplugin::Device

InitDaq() UserReadout()

RequestToken()

StartAcquisition()...

poland::Device


explodertest::Device


subclasses

DABC

hardware

libpexor.so

user library

pexor.ko

Linux kernel module

gosipcmd

control tool

PEXOR/KINPEX plug-ins

14.4.2015


7

Use generic pexor kernel module and library

Device base class:

common functionality (gosip, DMA, interrupts)

interface methods (InitDAQ(), UserReadout() )

Device subclasses:

Frontend specific, may implement user code

Readout modes:

Free running (polling request)

Request after trigger interrupt (emulates MBS DAQ)

automatic within trigger interrupt handler

http://web-docs.gsi.de/~adamczew/dabcpexor/index.html

http::Server in DABC

Access to different data in DAQ application

JavaScript ROOT as user interface

One could integrate and access data from different sources:

DABC, ROOT, Go4, MBS, EPICS, DIM, FESA, ...

Same data could be accessed by Go4 and displayed using native ROOT graphics

One also could use command-line tools to access DAQ/analysis data via http or socket

14.4.2015


8

Heterogeneous distributed applications

14.4.2015


9

MBS DAQ

shell> mbs -dabc

mbs> @startup

mbs> open file test.lmd

mbs> sta acq

cmd

Go4 analysis

linev@lxg0538:~ $ go4analysis -stream localhost

GO4-*> Reading library: libGo4UserAnalysis

GO4-*> Close of TUserSource

GO4-*> Create default analysis with processor class TFirstStepProcessor

GO4-*> Use class TStreamEvent as output event

GO4-*> Welcome to Go4 Analysis Framework Release v4.6.3 (build 40603) !

GO4-*> Analysis: Added analysis step Analysis

GO4-*> Main: starting analysis in batch mode ...

GO4-!> Fail to open AutoSave file Go4AutoSave.root

GO4-*> Analysis LoadObjects: Failed to load from file Go4AutoSave.root

f_evt_get_open for STREAM: port=6002 timeout=1

GO4-*> Factory: Create input event for MBS

GO4-*> Event MbsEvent101 has source localhost class: TGo4MbsStream

GO4-*> Factory: Create event processor Processor

GO4-*> Create TFirstStepProcessor Processor

GO4-*> Executing ROOT script first.C

GO4-*> Factory: Create output event OutputEvent

GO4-*> Event OutputEvent has source Processor class: TFirstStepProcessor

GO4-*> Analysis -- Initializing EventClasses done.

TGo4Sniffer

socket

Publisher

HADAQ

Publisher

socket

EPICS

DABC DAQ

DABC master

dabc::Publisher

collects data from remote

build global hierarchies

version control

http

socket

mbs plugin

socket

socket

DABC agents

log

status

MBS/GOSIP web gui

14.4.2015


10

DABC/PEXOR/GOSIP web gui

14.4.2015


11

POLAND frontend web gui

14.4.2015


12

HADES detector

HADES is a versatile detector for a precise spectroscopy of e+e- pairs (di-electrons) and charged hadrons produced in proton, pion and heavy ion induced reactions in a 13.5 GeV kinetic beam energy region

14.4.2015


13

http://www-hades.gsi.de

http://blogs.helmholtz.de

IOC

IOC

IOC

HADES DAQ before

14.4.2015


14

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

small hld

hld

hld

hld

hld

IOC

EPICS GUI

Event builder

Event builder

Event builder

Event builder

server

/shm

/shm

/shm

/shm

HYDRA2

~30 TRB frontend boards

~10 KHz, max 300 MB/s

simple round-robin approach

C-based, single thread code

EPICS IOCs for DAQ slow control

file-based quality monitoring

UDP

client

DABC

HADES DAQ now

14.4.2015


15

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

hld

Combiner

output

stream

DABC on all event builders

All infrastructure remains as is

Socket data stream to online analysis

Used in production beams 2014

http

DABC

hld

Combiner

output

stream

http

DABC

hld

Combiner

output

stream

http

DABC

hld

Combiner

output

stream

http

EPICS GUI

/shm

/shm

/shm

/shm

server

client

HYDRA2

UDP

HADES DAQ upgrade

New RICH and ECAL electronic

more than 100 TRBs (30 up to now)

Use in many places FPGA TDCs

need for special calibration procedure

Potentially free-running readout for RICH

need to produce subevents afterwards

HLD data format should be preserved

14.4.2015


16

Change networking topology

14.4.2015


17

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

DABC

DABC

DABC

DABC

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

DABC

DABC

DABC

DABC

10Gb Ethernet

All data from single TRB always send to same DAQ node

Every DAQ node can perform TDC calibration and event selection immediately

Event building done afterwards with TCP/IP or IB

UDP

UDP

TCP

Next plans

FPGA TDCs calibrations in DABC

already done, including web control GUI

Event building network (BNet) for HADES

planned as v3 release, end of 2015

Accommodate C++11 features

native threads, references

probably as v4 in ~2017

14.4.2015


18

Conclusion

Flexible framework for different DAQ tasks

Web and JSROOT as user interface

Used in different test beams

Production DAQ for HADES experiment

14.4.2015


19

Documents

Developments and applications of DAQ framework DABC v2 Jörn Adamczewski-Musch, Nikolaus Kurz, Sergey Linev GSI / Experiment Electronic