CBM DAQ and Event Selection

CBM DAQ and Event Selection

Walter F.J. Müller, GSI, Darmstadtfor the CBM Collaboration

Topical Workshop: Advanced Instrumentation for Future Accelerator Experiments,

Bergen, Norway, 4-6 April 2005

4-6 April 2005 Topical Workshop: Advanced Instrumentation for Future Accelerator Experiments, Bergen, Norway --- Walter F.J. Müller, GSI

2

Outline

CBM (very briefly) observables setup

FEE/DAQ/Trigger requirements challenges strategies


3

CBM at FAIRSIS 100 Tm

SIS 300 Tm

U: 35 AGeV

p: 90 GeV

Compressed Baryonic MatterExperiment


4

CBM Physics Topics and Observables In-medium modifications of hadrons

onset of chiral symmetry restoration at high ρB

measure: , , e+e- (μ+ μ-) open charm: D0, D±

Strangeness in matter enhanced strangeness production measure: K, , , ,

Indications for deconfinement at high ρB

anomalous charmonium suppression ? measure: D0, D±

J/ e+e- (μ+ μ-) Critical point

event-by-event fluctuations

measure: π, K

Good e/π separation

Low cross sections→ High interaction rates→ Selective Triggers

Hadron identification

Vertex detector


5

CBM Setup

Radiation hard Silicon pixel/strip detectors in a magnetic dipole field

Electron detectors: RICH & TRD & ECAL: pion suppression up to 105

Hadron identification: RPC, RICH

Measurement of photons, π0, η, and muons: ECAL


6

CBM and HADESAll you want to know about CBM:Technical Status Report (400 p)

now available underhttp://www.gsi.de/documents/DOC-2005-Feb-447-1.pdf

http://www.gsi.de/documents/DOC-2005-Feb-447-1.pdf


7

Meson Production in central Au+AuW. Cassing, E. Bratkovskaya, A. Sibirtsev, Nucl. Phys. A 691 (2001) 745

10 MHz interaction rateneeded for 10-15 A GeV

SIS300


8

A Typical Au+Au Collision

Central Au+Au collision at 25 AGeV:URQMD + GEANT

160 p 170 n360 - 330 + 360 0 41 K+ 13 K- 42 K0

107 Au+Au interactions/sec

109 tracks/sec to reconstruct for first level event selection


9

CBM Trigger Requirements In-medium modifications of hadrons

onset of chiral symmetry restoration at high ρB

measure: , , e+e-

open charm (D0, D±) Strangeness in matter

enhanced strangeness production measure: K, , , ,

Indications for deconfinement at high ρB

anomalous charmonium suppression ? measure: D0, D± -

J/ e+e Critical point

event-by-event fluctuations

measure: π, K

offline

assume archive rate:few GB/sec20 kevents/sec

offline

offline

trigger

trigger

trigger trigger on high pt e+ - e- pair

trigger ondisplaced vertex

drives FEE/DAQarchitecture


10

Open Charm Detection Example: D0 K-+ (3.9%; c = 124.4 m) reconstruct tracks find primary vertex find displaced tracks find secondary vertex

target

first two planesof vertex detector

few 100 μm

5 cm

high selectivity because combinatorics is reduced


11

CBM DAQ Requirements Profile

D and J/Ψ signal drives the rate capability requirements D signal drives FEE and DAQ/Trigger requirements

Problem similar to B detection, like in LHCb or BTeV (rip)

Adopted approach:

displaced vertex 'trigger' in first level, like in BTeV (rip)

Additional Problem:

DC beam → interactions at random times

→ time stamps with ns precision needed

→ explicit event association needed Current design for FEE and DAQ/Trigger:

Self-triggered FEE Data-push architecture


12

Conventional FEE-DAQ-Trigger Layout Detector

Cave

Shack

FEE

Buffer

L2 Trigger L1 Trigger

DAQ

L1 A

ccep

t

L0 Trigger

fbunch

Archive

Trigger

Primitives

Especially

instrumented

detectors

Dedicated

connections

Specialized

trigger

hardware

Limited

capacity

Limited

L1 trigger

latency

Modest

bandwidth

Standard

hardware


13

Limits of Conventional Architecture

Decision time for first level trigger limited.

typ. max. latency 4 μs for LHC

Only especially instrumenteddetectors can contribute to

first level trigger

Large variety of veryspecific trigger hardware

Not suitable for complexglobal triggers like secondary

vertex search

Limits future triggerdevelopment

High development cost


14

L1 Trigger

Special

hardware

High

bandwidth

The way out .. use Data Push ArchitectureDetector

Cave

Shack

FEE

Buffer

L2 Trigger L1 Trigger

DAQ

L1 A

ccep

t

L0 Trigger

fbunch

Archive

Trigger

Primitives

Especially

instrumented

detectors

Dedicated

connections

Specialized

trigger

hardware

Limited

capacity

Limited

L1 trigger

latency

Modest

bandwidth

Standard

hardware

fclock

L2 Trigger

Timedistribution


15

L1 Trigger

Special

hardware

High

bandwidth

The way out ... use Data Push ArchitectureDetector

Cave

Shack

FEE

DAQ

Archive

fclock

L2 Trigger


16

L1 Select

Special

hardware

High

bandwidth

The way out ... use Data Push Architecture Detector

Cave

Shack

FEE

DAQ

Archive

fclock

L2 Select

Self-triggered front-end

Autonomous hit detection

No dedicated trigger connectivity

All detectors can contribute to L1

Large buffer depth available

System is throughput-limited

and not latency-limitedModular design:

Few multi-purpose rather

many special-purpose

modulesUse term: Event Selection


17

Front-End for Data Push Architecture Each channel detects autonomously all hits An absolute time stamp, precise to a fraction of the

sampling period, is associated with each hit All hits are shipped to the next layer (usually

concentrators) Association of hits with events done later using time

correlation

Typical Parameters: with few 1% occupancy and 107 interaction rate:

some 100 kHz channel hit rate few MByte/sec per channel whole CBM detector: 1 Tbyte/sec


18

Typical Self-Triggered Front-End Average 10 MHz interaction rate Not periodic like in collider On average 100 ns event spacing

0 5 10 15 20 25 30 time

ampl

itude

50

100

a: 126 t: 5.6

a: 114 t: 22.2

Use sampling ADCon each detector

channel running withappropriate clock

Time is determinedto a fraction of thesampling period

threshold


19

Toward Multi-Purpose FEE Chain

PreAmpPreAmp preFilter

preFilter ADCADC Hit

Finder

Hit

Finderdigital

Filter

digital

FilterBackend

& Driver

Backend

& Driver

Pad GEM's PMT APD's

Anti-Aliasing

Filter

Sample rate:

10-100 MHz

Dyn. range:

8...12 bit

'Shaping'

1/t Tailcancellation

Baselinerestorer

Hit

parameter

estimators:

Amplitude

Time

Clustering

Buffering

Link protocol

All potentially in one mixed-signal chipsee talk V. Lindenstruthsee talk L. Musa


20

CBM DAQ and Online Event Selection

More than 50% of total data volume relevant for first level event selection

Aim for simplicity Ansatz:

do (almost) all processing done after the build stage

Simple two layer approach:1. event building2. event processing

Other scenarios are possible, putting more emphasis on:

do all processing as early as possible

transfer data only then necessary

neededfor D needed

for J/μ usefullfor J/μ

STS, TRD, and ECAL data usedin first level event selection


21

Logical Data Flow

Concentrators:multiplex channelsto high-speed links

Time distribution

Buffers

Build Network

Processing resources forfirst level event selectionstructured in small farms

Connection to'high level'

selection processing


22

Bandwidth Requirements

Data flow:

~ 1 TB/sec

1st level selection:

~ 1014-15 operation/sec

Data flow:

few 10 GB/sec

to archive: few 1 GB/sec

Moore helps

Gilder helps

~ 100 Sub-Farms


23

Focus on CNet


24

Self-Triggered FEE – Output Format I

FEE

17 15 ...

68 34 ...

134 18 ...

135 19 ...

1234 33 ...

Output of a FEE chipis a list of hits

Each hit has a timestamp

plus other information

TimeStamp

Channeladdress

other values:amplitudes

pulse shape

!! Time Stamp values can increase forever !!

? How to express absolute time efficiently ?

Output of asingle

FEE chip


25

Handle the infinite Time Axis

Time

Epoch 1 Epoch 2 Epoch 3 Epoch 4

(2, 137 ns) (3, 314 ns)

1. Subdivide Time in Epochs

2. Express a timerelative to an epoch

3. Introduce Epoch Markers

A HitAn EpochMarker

practical epochlength about 10 μs

practical epochlength about 10 μs


26

Self-Triggered FEE – Output Format II

FEE

M 1

H 17 15 ...

H 68 34 ...

H 134 18 ...

H 135 19 ...

H 1234 33 ...

M 2

M 3

H 258 19 ...

Output of a FEE chipis a list of hits and

epoch markers

Each hit has a timestamp

plus other information

Recordtype

Hit

EpochMarker

Hit with effective timestamp (3, 258)


27

Self-Triggered FEE – Concentrators

FEE

M 1

H 17 15 ...

H 68 34 ...

H 134 18 ...

H 135 19 ...

H 1234 33 ...

M 2

M 3

H 258 19 ...

A concentrator merges

the data streams andeliminates redundant

epoch markers

FEE

M 1

H 18 2007 ...

M 2

H 589 2134 ...

M 3

H 258 2714 ...

M 1

H 17 15 ...

H 18 2007 ...

H 68 34 ...

H 134 18 ...

H 135 19 ...

H 1234 33 ...

M 2

H 589 2134 ...

M 3

H 258 19 ...

H 258 2714 ...

Seems prudent

to keep dataalways sorted

in time

time address


28

FEE Data – Clusters I

In many subsystems a particle causes correlated hits in physically neighboring detector cells (STS, TRD, ECAL)

Depending on detector subsystem the cluster pattern is 1d or 2d contained in one FEE chip or not

examples in CBM: STS-MAPS: 2d contained STS-Strip: 1d mostly contained TRD 1d mostly contained to 2d often uncontained

depending on pad geometry (varies inside→outside)

RPC t.b.d. ECAL 2d many uncontained

Note for 2d: a 16(64) channel chip has ¾(½) of channels on perimeter !


29

FEE Data – Clusters II

Usually one wants to read very low amplitude hits in the tail of a cluster low channel hit threshold might give to much noise → read only low amplitude hit if in neighborhood of a big one → how to handle clusters crossing a chip border ?

use two thresholds high threshold determines particle hit and region of interest RoI communicated to all relevant neighbors low amplitude hits in RoI are validated and send → this implies cross communication on CNet between FEE chips...

If RoI are communicated, CNet becomes a real network !!Better name

d

FNet

see talk V. Lindenstruthsee talk L. Musa


30

Focus on BNet


31

Event Building – Alternatives

Straight event-by-event approach: data arrives on ~1000 links ~100 byte per event and link 1010 packets/sec to handle...

Handle time intervals or event intervals 10 μs or 100 events seems reasonable

Very regular and fully controlled traffic pattern: data traffic can be scheduled to avoid network

congestion a large fraction of the switch bandwidth can be used


32

Networking I High-speed networking

high density connectors 2.5 Gbps SerDes now ~100 mW 480 Gbps InfiniBand switch on one chip DDR and QDR link speeds will come just wait and see

Mellanox MTA4739624 port InfiniBand switch 4x ports, 1 Gbyte/sec per port → 96 x 2.5 Gbps SERDES 480 Gbps aggregate B/W Single chip implementation 961 ball BGA 18 W power dissipation Double data rate version

(5 Gbps per link) in pipe....


33

Networking II

TODAY: Voltaire ISR 9288 switch 288 4x ports; non-blocking cost today ~120 kEUR (or ~400 EUR/port) 288 GByte/sec switching bandwidth

likely in a few years: 288 4x port QDR likely same or lower cost 1152 GByte/sec switching speeds adequate for CBM...

Conclusion: BNet switch is not a major issue


34

Focus on PNet


35

Network Characteristics

Data PushDatagram'serrors marked

but not recovered

Request/Responseand Data Push

Transactionserrors recovered


36

L1 Event Selection Farm Layout

Current working hypothesis: CPU + FPGA hybrid system (proviso follows)

Use programmable logic for cores of algorithms Use CPU for the non-parallelizable parts Use serial connection fabric (links and switches) Modular design (only few board types) FPGA


37

Network Summary

5 different networks with very different characteristics CNet

medium distance, short messages, special requirements connects custom components (FEE ASICs)

TNet broadcast time (and tags), special requirements

BNet naturally large messages, Rack-2-Rack

PNet short distance, most efficient if already 'build-in' connects standard components (FPGA, SoCs)

HNet general purpose, to rest of world Ethernet

PCIe,ASI,....

Custom

Custom

Ethernet,

Infiniband,...

Whatever the implementation is, it will be called Ethernet...

FEE Interfaces and CNet will be co-developed. Depends on clock/time distribution is done

Potentially build with CNet components

Look at emerging technologiesStay open for changes and surprises

Cost efficiency is key here !!

Probably uncritical


38

Algorithms

Performance of L1 feature extraction algorithms is essential critical in CBM: STS tracking + vertex reconstruction

TRD tracking and Pid

Look for algorithms which allow massive parallel implementation Hough Transform Tracker

needs lots of bit level operations, well suited for FPGA Cellular Automaton tracker Other approaches to be evaluated

Co-develop tracking detectors and analysis algorithms L1 tracking is necessarily speed optimized

→ more detector granularity and redundancy needed Aim for CBM:

Validate final hardware design with at least 2 trackers suitable for L1


39

Algorithms – an Example

Hough Transform assume track comes from

(close to) primary vertex map each measurement into

'Hough space' a peak in Hough space

indicates a real track is a 'global' method needs substantial amount of

calculation to fill and analyze the histograms

Many, but very simple operations

allows massively parallel implementation


40

Hough-Transform – Implementation


41

Hough-Transform – Implementation

Very suitable forimplementation in

programmable logic(FPGA's)

Other track finderapproaches, likecellular automata

tracker, also underinvestigation


42

Interim Summary Event definition has changed:

now based on time stamps and time correlation

Role of DAQ has changed: DAQ is simply responsible to transport data from producers to consumers

Role of 'Trigger' has changed: filter events delivered by DAQ 'Online Event Selection' is better term

System aspects: 'online' – 'offline' boundary blurs more COTS (commercial off the shelf) components much more modular system much more adaptable system

This is emerging technology in HEP, though baseline for ILCHowever: being used since many years in nuclear structure


43

Moore – quo vadis ?

Will price/performance of computing continue to improve ? What are limits of CMOS technology ? Where are the markets ? What are market forces ?

Technology most of the gain comes from architecture anyway conventional designs, especially x86, reach their limits

Markets end of the metal-box PC age

→ Laptops + PDA + all kind of dedicated boxes (Video, Games) end of the binary compatibility age

→ intermediate code + 'Just in Time' Compilers (JIT)

There is life after Intel x86A lot of architectural innovation ahead


44

BlueGene vs Cell Processor

CPU

Cache

Mem

Mem

IO

IO

Mem SPEMemSPE

Mem SPEMemSPE

Mem SPEMemSPE

Mem SPEMemSPE

CPU CPU

Cache Cache

IO IO IO IO

Mem

BlueGene:121 mm2; 130 nm2.8/5.6 DP GFlop STI Cell:

221 mm2; 90 nm256 SP GFlop 30 DP GFlop 25 GB/sec mem 78 GB/sec IO

Finally presentedon ISSCC 2005

SPE = Synergistic Processing ElementInternational Solid-State Circuit Conf.


45

BlueGene vs Cell Processor

Developed bySony, Toshiba and IBMMarket: VIDEOGAMESBudget: 500 M$

Developed by IBMMarket: national security science Budget: ~100 M$

High performance computing is driven now by embedded systems(games, video, ....) → Science is a spin-off, at best ...


46

STI Cell Processor 'normal' PowerPC CPU 8 Synergistic Processing

Element (SPE) each with 258 kB memory 128 x 128 bit registers 4 SP floating point units own instruction stream

32 multiply/add per clock cycle

runs at > 4 GHz

221 mm2 die sizein 90 nm


47

Game Processors as Supercomputers ?

Slide from CHEP'04 Dave McQueeney

IBM CTO US Federal


48

CPU and FPGA paradigms merge

Register

ALUCo

ntr

ol Wide Register

ALUCo

ntr

ol

ALU ALU ALU

Wide Register

ALU

Co

ntr

ol ALU ALU

ALU

ALU

ALU ALU ALU

ALU

ALU ALU

ALU ALU

ALU ALU

ALU ALU

ALU

Conventional CPU SIMD (single instruction – multiple data) CPU

Configurable Instruction Set CPU

arithmeticresources

configurableconnection

fabric

PSMPSMPSMPSMPSM

PSMPSM PSM PSM PSM


49

Configurable Instruction Set Processor

Example Stretch S5xxx Hybrid design:

conventional fixed instruction set part

plus configurable instruction set part

C/C++ compiler analyses the kernel of algorithms generates custom

instruction set generates code to use it

The promise easy of use of C/C++ performance of an FPGA

Fabric is the keyword

interconnected resources

Stretch S5 engine

from Stretch Inc. product brief


50

CPU and FPGA paradigms merge

configurablelogic

CPU

configurablelogic

CPU CPU

FPGAindustry

world view

Processorindustry

world view

A lot of innovation in the years to come

Essential will be availability of efficient development tools

Moore will go on ! There are the technologies

There are the marketsArchitectural changes

ahead


51

Summary

Self-triggered FEE: autonomous hit detection, time-stamping with ns presision sparsification, hit buffering, high output bandwidth

High bandwidth event building network handle 10 MHz interaction rate in Au-Au also cope with few 100 MHz interaction rate in p-p, p-A likely be done in time slices or event slices

L1 processor farm feasible with PC + FPGA + Moore (needed 2014) but look beyond todays PC's and FPGA's

Efficient algorithms (109 tracks/sec) co-design of critical detectors and tracking software

Quitedifferentfrom thecurrent

LHC styleelectronics

Substantial R&D

needed

RII3-CT-2004-506078


52

The End

Thanks for your attention

CBM Collaboration : 41 institutions, 15 countriesChina:Hua-Zhong Univ., WuhanCroatia: RBI, ZagrebCyprus: Nikosia Univ. Czech Republic:Czech Acad. Science, RezTechn. Univ. Prague France: IReS Strasbourg

Germany: Univ. Heidelberg, Phys. Inst.Univ. HD, Kirchhoff Inst. Univ. FrankfurtUniv. KaiserslauternUniv. Mannheim Univ. MarburgUniv. MünsterFZ RossendorfGSI Darmstadt

Russia:CKBM, St. PetersburgIHEP ProtvinoINR TroitzkITEP MoscowKRI, St. PetersburgKurchatov Inst., MoscowLHE, JINR DubnaLPP, JINR DubnaLIT, JINR DubnaMEPhi, MoskauObninsk State Univ.PNPI GatchinaSINP, Moscow State Univ. St. Petersburg Polytec. U.

Spain: Santiago de Compostela Uni.

Ukraine: Shevshenko Univ. , Kiev

Hungaria:KFKI BudapestEötvös Univ. Budapest

Korea:Korea Univ. SeoulPusan National Univ.

Norway:Univ. Bergen

Poland:Krakow Univ.Warsaw Univ.Silesia Univ. Katowice Portugal: LIP Coimbra

Romania: NIPNE Bucharest


54

FPGA – Basic Building Block

CLK

F3

F2

F1

F0XQ

X

D

C

Q

LUT

CLB

Look-up Tablejust a 4x1 RAM

D Flip-FlopUniversallogic gate

Elementarystorage unit

CLB = Configurable Logic Block


55

FPGA – Putting it together

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

PSM

PSM

PSM

PSM

PSM

PSM

PSM

PSM

PSM

ConfigurableLogic Block

Wiring

Programmableswitch matrix

I/O blocks

Modern FPGA's:>100.000 LUT 500 MHz

Documents

CBM DAQ and Event Selection