MBS (Multi Branch SystemMBS (Multi Branch ) IIlnr.irb.hr/pd/daqws/images/N.Kurz_2.pdf · 2011-11-08 · GOSIP: supports data transfer between PEXOR and FEBEX / EXPLODER (front-ends

N.Kurz, EE, GSI, Zagreb 4,5-Nov-2011 1

MBS (Multi Branch MBS (Multi Branch System) II) II

The General Purpose Data Acquisition System MBSat GSI (and elsewhere)

PCI Express based SystemsData transfer via Fibers

Custom Hardware

2

FEBEX2/3: Pipeline ADC Front End BoardFEBEX2/3: Pipeline ADC Front End Board

FEBEX2: 8 ADC channels, 60MHz sampling rate, 12 bit resolution (60/12)FEBEX3: 16 ADC channels, 60MHz sampling rate, 12 bit resolution (60/12) and 50/14

ADC

TRIGGER I/O

FPGA

SFP 0

SFP 1

FEBEX

CLK INAdapter boards for singleended and differential inputs

also possible:65 MHz, 12 bit 16 (32) ch50 MHz, 14 bit 8 (16) ch

105 MHz, 12 bit 8 (16) ch100 MHz, 16 bit 8 chetc.

2 pairsoptical fibers2 Gbit/s per fiber

Purpose: Provide fast “digital electronics” to MBS community.Filter “continuous” data streams from ADCs with respect to trigger windows.In most case Pulse Shape Analysis (PSA: E, t) required.

3

EXPLODER1/2AEXPLODER1/2A

EXPLODER1: 32 LVDS input pairsEXPLODER2A: 64 LVDS input pairs

TRIGGER I/O

FPGA

SFP 0

SFP 1

EXPLODER

CLK IN

2 pairsoptical fibers2 Gbit/s per fiber 32 LVDS input pairs

interface for Nxyter front end card

interface for any detector, whichdelivers LVDS signals

Purpose: Provide a connection to all detector systems with LVDS outputs (Nxyter, GET4).Filter “continuous” data streams from detectors according to trigger windows.

4

FEBEX2FEBEX2

5

EXPLODERIEXPLODERI

6

PEXOR: Front End Data CollectorPEXOR: Front End Data Collector

TRIG. REG. I/O

PEXOR

FPGA

4 SFP Fiber pairseach 2 Gbit/s to connect 4 x 256 front end boardsat maximum

after 10/8 coding:200 MB/s payload speedper SFP

4 Lane PCI Express> 600 MB/s FPGA -> PC DRAM payload speed

7

PEXOR3PEXOR3

8

PEXORPEXOR--FEBEX2 ConnectionsFEBEX2 Connections

TRIG. REG. I/O

PEXOR

FPGA

ADC

TRIGGER I/O

FPGA

FEBEX: id 0

CLK IN

ADC

TRIGGER I/O

FPGA

FEBEX: id 1

CLK IN

ADC

TRIGGER I/O

FPGA

FEBEX: id 2

CLK INConnection via:FiberCopper cableCopper backplane

GOSIP

9

PEXORPEXOR--TRIXORTRIXOR--FEBEX Test SetupFEBEX Test Setup(FEBEX not shown)(FEBEX not shown)

10

Traces from GO4 <Traces from GO4 <-- MBS <MBS <-- PEXOR <PEXOR <-- FEBEXFEBEX

11

Data Transfer PEXORData Transfer PEXOR--FEBEX / EXPLODER GOSIP FEBEX / EXPLODER GOSIP ((GGigabit igabit OOptical ptical SSerial erial IInterface nterface PProtocol)rotocol)

ShizuShizu MinamiMinami

tS

Star – versus Chain topologies between frond-ends and concentrator/readout boards:

Star: Easy transfer protocol, limited number of frontends, hubs neededChain: (More) complicated transfer protocol, highly scalable in size and speed.

GOSIP:supports data transfer between PEXOR and FEBEX / EXPLODER (front-ends )for systems of sizes between1 front-end and 1024 front-end cards. For data speed optimization, each front-end is equipped with two SFPs.See topology in previous transparency, see GOSIP protocol below.

GOSIP supports Chain Initialization of all four PEXOR chains. During initialization each front-end gets a uniquemodule id, starting from 0 with the front-end closest to the PEXOR,

GOSIP supports Transparent Mode (read and write access) for slow control and setup issues from PEXORto each frontend with a speed of ~ 100K accesses/s:

A r/w request is send from the PEXOR upstream to front-end 0. Each frontend examines if the request is for itself.If yes, it passes the result (read: ack and data, write ack) downstream to the PEXOR. If the request is fora different front-end the request is passed upstream to the next front-end in the chain. Due to the chain topology chosen,data coming downstream a chain, can pass all front-ends, via a FIFO, without intervention, until it reaches finally the PEXOR.

GOSIP supports Token Mode for fast data transfer from front-ends to PEXOR:

In token mode the readout toke is initiated by the PEXOR, which sends the token to the first front-end. This sends its datapacket down to the PEXOR and sends afterwards the token to the next front-end in the chain. The last front-end in the chainsends the token after sending its data packet back to the token, which completes the transfer.payload speeds of 4 times 200 MB/s have been measured.

12

Fast Token Data Transfer OptionsFast Token Data Transfer OptionsFrontFront--ends ends --> PEXOR> PEXOR

1) Wait for data ready mode: GOSIP waits until front-end declared data ready before sending content of data buffer 0/1.For triggered systems!

2) No wait Token: GOSIP sends data from data buffer 0/1 immediately on token arrival time down stream.Useful for “triggerless” systems!

FEBEX / EXPLODER FPGA (per input/ADC channel)

Input ring buffer

Data buffer 0 Data buffer 1

Implementationtraces, PSA

GOSIP

token intoken outdata out

13

Fast Token Data Transfer OptionsFast Token Data Transfer OptionsPEXOR PEXOR --> PC DRAM / MBS> PC DRAM / MBS

1) Parallel token data sending on for all connected front-end chains. Sequential DMA, initiated from MBSuser readout function, from PEXOR to PC DRAM. Used for “small” data sizes.

2) Sequential token data sending for all connected front-end chains. Sequential DMA, automatically initiatedby PEXOR FPGA from PEXOR to PC DRAM. Used for “big” data sizes. Limit is PC DRAM not PEXOR memory.

PEXOR FPGA

Data in SFP 0

Data in SFP 2

FPGA MEMORY

PCIeDMA

PC DRAM

MBSSub-event

pipeData in SFP 1 PCI Express

Data in SFP 3

14

TRIXORTRIXOR

TRIXOR: - Identical functionality as TRIVA3/5/7

- Can be plugged in PCI Express or PCI slots. Takes only power.

- Works as master (trigger from input) and slave (trigger from trigger bus).

- Several TRIXOR and TRIVA can be inter-connected via the trigger bus to composesynchronous bigger MBS systems.

- Communication with the TRIXOR viaPEXOR and the Trigger Register I/Oconnector.

- Accepted trigger send a signal via theTrigger I/O connector to the PEXOR,which is then transformed into a

PCI Express interrupt to notify the PCreadout processor.

PCI Express, PCI (only power)

TRIG. REG. I/O

TRIXOR

FPGA

OUT

IN

OUT

IN

TriggerBUS

Trigger

15

TRIXORTRIXOR

16

TRIGGERINGTRIGGERING

TRIG. REG. I/O

TRIXOR

FPGAOUT

IN

OUT

IN

Trigger Dead Time Locker Priority Encoder

(VULOM, EXPLODER2A, LEVCON)

Accepted TRIGGER OUT

Raw TRIGGER IN

4 bitencoded

TRIG. REG. I/O

PEXOR

FPGA

ADC

TRIGGER I/O

FPGA

FEBEX: id 0

CLK IN

ADC

TRIGGER I/O

FPGA

FEBEX: id 1

CLK IN

ADC

TRIGGER I/O

FPGA

FEBEX: id 2

CLK IN

FEBEX Trigger and Clock Bus”2 in/out Common Clock for all FEBEX2 in Physics and Sync Trigger1 out Trigger request, OR from all FEBEX

Standard PC running MBSon LynxOSor Debian Linux

17

FEBEX2/3 FeaturesFEBEX2/3 Features

- FEBEX2: 8 channels, FEBEX3: 16 channels - 60 MHz, 12 bit, +- 1 V input signals- ADC input circular buffer (per channel)- Double signal trace buffer (per channel)

- I/O: IN (2): Physics trigger, sync. triggerIN: Common clock (for hit timing)OUT: Common clock (dedicated FEBEX as clock master, external clock master)OUT: Trigger request from hit finder. OR from all channels (see below and next slide)

- Two hit finder (or self trigger) algorithms ( 3 step, trapezoidal filter)- Dead time clear before readout (double trace buffer!)

- Data Output: Complete trace in trigger window and hit time (from common clock)

18

FEBEX2/3 Features II, Setup ParametersFEBEX2/3 Features II, Setup Parameters

All features per channel:

- Enable/Disable Channel - Enable/Disable Self trigger (hit finder) - Enable/Disable Data reduction - Set positive or negative input signals - Even/Odd channel readout

- Set pre-trigger time (0 - 34 us, in nr. of ADC samples (1./60MHz := 16.7 ns )- Set trace length (0 - 200 us, in nr. of ADC samples (1./60MHz := 16.7 ns )

- Select Self trigger method:a) 3 step b) trapezoidal filter, select one out of 4 different scanning frequencies

60 -, 30 -, 15 -, 7.5 MHz- Self trigger Threshold (in ADC counts, 0.5 mV steps)

19

FEBEX / EXPLODER Signal Input StageFEBEX / EXPLODER Signal Input Stage

FEBEX / EXPLODER input stage implemented in FPGA for each ADC/Input channel:

Input ring buffer (2048 samples)

Data buffer 0 (12288 samples)

Samples from ADC (60MHz)

Data buffer 1 (12288 samples)

Input ring -, data buffer 0/1 for each ADC channelInput ring buffer accepts ADC samples with the speed of the ADC without interruptionOn occasion of a trigger, content of Input buffer is copied with the speed of the ADC in a toggling mode into one of the data buffers

Length of input ring buffer defines maximum pre trigger window (see also next slide):100 MHz: 20 us, 60 MHz: 34 us

Length of data buffer defines the maximum trace length (for hit finders/PSA):100 MHz: 123 us, 60 MHz: 200 us

both Input ring buffer and data buffer sizes are a sensible choice. can be changed.

20

““DeadDead--timetime”” free Data Acquisition withfree Data Acquisition withTrigger WindowsTrigger Windows

Trigger window adjustable from 1 sample to 12288 samples (FEBEX)

Trigger window Tn

Trigger Tnstart copyinginput ring buffer todata buffer 0

copy time Tn

Trigger window Tn+1

Earliest time forTrigger Tn+1start copyinginput ring buffer todata buffer 1

copy time Tn+1

Trigger window Tn+2

Eaerliest time forTrigger Tn+2start copyinginput ring buffer todata buffer 0

copy time Tn+2

Note: - Trigger windows can be adjacent (no dead time) - Avoid overlapping trigger windows by setting conversion time (minimum time between two triggers) to trigger window length- Double data buffers 0/1 allow to release the system dead time before actual readout of data from FEBEX to PEXOR.- Very large data sizes (data rates > 200 MB/s) might delay dead time release in a sense, that adjacent trigger windows(dead time free) are not possible in all cases. In this case the system is not anymore dead time free. This situation is alsopresent in so called “trigger less” systems, when data rates produced in the frontends, exceeds the bandwidth of a transferchannel.

21

VULOM5 + VULOM5 + SFPsSFPsConnection VME <Connection VME <--> FEBEX / EXPLODER> FEBEX / EXPLODER

FPGA

FPGA

FPGA

VULOM5

VULOM5 SFP ADD ON P1

P2

VME

VME 2eSST: 150 MB/sVULOM5 -> RIO4

4 SFP Fiber pairseach 2 Gbit/s

after 10/8 coding:200 MB/s payload speed

22

VULOM FEBEX/EXPLODER connectionsVULOM FEBEX/EXPLODER connections

ADC

TRIGGER I/O

FPGA

FEBEX: id 0

CLK IN

ADC

TRIGGER I/O

FPGA

FEBEX: id 1

CLK IN

ADC

TRIGGER I/O

FPGA

FEBEX: id 2

CLK IN

FPGA

FPGA

FPGA

VULOM5

VULOM5 SFP ADD ON P1

P2

VME

23

Application with EXPLODER:

Readout of Nxyter chips/boards

24

NxyterNxyter readout with EXPLODER / PEXORreadout with EXPLODER / PEXORIvan Ivan RusanovRusanov

TRIGGER I/O

FPGA

SFP 0

SFP 1

EXPLODER

CLK IN ADC

Nxyter30 cm cable

- Running stable with MBS on PC with PEXOR,TRIXOR and EXPLODER / NXYTER.- EXPLODER and Nxyter Synchronization with external clock (CLK IN).- Reset of Nxyter clock with first hardware trigger to EXPLODER, armed after initialization.

Exploder + Nxyter front-end card GEMEX

25

Test Hit Pattern of two Test Hit Pattern of two NxyterNxyter chipschips

26

NxyterNxyter Internal Time DifferenceInternal Time Difference((NxyterNxyter 0 top, 0 top, NxyterNxyter 1 bottom)1 bottom)

27

Time Difference across two Time Difference across two NxyterNxyter chipschipsChannel (0Channel (0--0, 70, 7--7, 157, 15--15)15)

28

ADC ADC NxyterNxyter, Channel 0, 7, 15, Channel 0, 7, 15

29

GEMEX (256 Detector Channels)GEMEX (256 Detector Channels)

TRIGGER I/O

FPGA

SFP 0

SFP 1

GEMEX

CLK IN

Nxyter

ADCNxyter Detector

Signals

Project with Detector Lab GSI: 10000 channel GEM TPC

30

A real Experiment withPEXOR, TRIXOR, FEBEX

The Search for Element 119, 120 at GSI

Connecting VME am PCI Express Systems

31

119/129 MBS DAQ 2011/12119/129 MBS DAQ 2011/12

Accepted Trigger In

PEXOR

TRIXOR

LynxOS PC

LynxOS PC

RIO

3

TRIV

A

ADCs, Latches, …

VME

Event-Builder

Trigger Bus

TCPTCP

640 Digital electronicchannels320 Detector channels

ancillary(only if needed)

Data LoggingOnline Monitoring

32

Example Trace (pulse height: ~ 300 mV) Example Trace (pulse height: ~ 300 mV)

0 500 1000 1500 2000 2500 3000

-600

-500

-400

-300

-200

-100

0

Trace, base line restored 10:59:02 2011-09-26 Analysis/Histograms/Traces BLR/TRACE, base line restored SFP: 0 FEBEX: 0 CHAN: 0

An al ysi s/ H is to gr am s/ Tr ace s BL R/ TR A C E, b ase l i ne r es to r ed S FP : 0 FE BE X: 0 C H AN : 0

An al ysi s/ H is to gr am s/ Tr ace s BL R/ TR A C E, b ase l i ne r es to r ed S FP : 0 FE BE X: 0 C H AN : 1An al ysi s/ H is to gr am s/ Tr ace s BL R/ TR A C E, b ase l i ne r es to r ed S FP : 0 FE BE X: 0 C H AN : 2







An al ysi s/ H is to gr am s/ Tr ace s BL R/ TR A C E, b ase l i ne r es to r ed S FP : 0 FE BE X: 1 C H AN : 7An al ysi s/ H is to gr am s/ Tr ace s BL R/ TR A C E, b ase l i ne r es to r ed S FP : 1 FE BE X: 0 C H AN : 0An al ysi s/ H is to gr am s/ Tr ace s BL R/ TR A C E, b ase l i ne r es to r ed S FP : 1 FE BE X: 0 C H AN : 1
















Trace length of 3000 := 50 us chosen to:

1) Cover rapid 120 decays within a single trace2) VME dead time is always shorter than 50 us3) Decent data size/rate

33

Example Trace (pulse height: ~ 15 mV)Example Trace (pulse height: ~ 15 mV)

0 500 1000 1500 2000 2500 3000

-60

-40

-20

0

20

Trace, base line restored 11:07:06 2011-09-26 Analysis/Histograms/Traces BLR/TRACE, base line restored SFP: 0 FEBEX: 0 CHAN: 0

























34

FEBEX3, 640 Digital Channels FEBEX3, 640 Digital Channels

35

Structure for f_user.c, setup.usf and set_mo.usf identical for VME and PCI Express based systems.

MBS commands, messages, data logging, data monitoring identical for VMEPCIe systems.

setup.usf needs only a few (static) changes (controller type, etc).

set_mo.usf needs usual customization for topology and node names.

f_user.c must be customized (as usual).

36

SummarySummary

Documents

MBS (Multi Branch SystemMBS (Multi Branch ) IIlnr.irb.hr/pd/daqws/images/N.Kurz_2.pdf · 2011-11-08 · GOSIP: supports data transfer between PEXOR and FEBEX / EXPLODER (front-ends