ALICE DAQ future detector readout

Filippo CostaALICE DAQ

ALICE DAQfuture

detectorreadout

October 29, 2012 CERN

Filippo Costa, CERN 2

• Current status and

evolution

• LS2 upgrade

• Architectural issues

• Development process

• Present R&D activities

29/10/2012

Outline

Filippo Costa, CERN 329/10/2012

Trigger – DAQ – HLT

GDC TDSM

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

LDCLDC

BUSY BUSY

Rare/All

Event Fragment

Sub-event

Event

File

Storage Network (8GB/s)PDS

L0, L1a, L2

L0, L1a, L2

360 DDLs

D-RORCD-RORC

EDM

LDC

D-RORC D-RORC

Load Bal. LDC LDC

D-RORC D-RORC

HLT Farm

FEPFEP

DDL

H-RORC

10 DDLs

10 D-RORC

10 HLT LDC

120 DDLs

DADQM

DSS

Event Building Network (20 GB/s)

430 D-RORC

175 Detector LDC

75 GDC30 TDSM

18 DSS60 DA/DQM

75 TDS

Archiving on Tapein the ComputingCentre (Meyrin)


DDL current status (Oct ‘12)

DETECTOR # DAQ DDL # HLT DDLACORDE 1 0

EMCAL 24 24

FMD 3 3

HMPID 14 0

MTRK 20 20

MUON-TRG 2 2

PHOS 12 12

PMD 6 0

SDD 24 24

SPD 20 20

SSD 16 16

T0 1 1

TOF 72 0

TPC 216 216

TRD 13 13

V0 1 1

ZDC 1 1

TOT DETECTORs

17

TOT # DAQ DDLs

485

TOT # HLT DDLs

353


LS2 Online Upgrade2x10 or 40 Gb/sFLP

DAQ and HLT10 or 40

Gb/s

FLPEPN

FLP

ITS

TRD

Muon

FTP

L0L1

FLPEMCal

EPN

FLPTPC

DataStorage

FLP

FLPTOF

FLP

FarmNetworkPHOS

Trigger Detectors

~ 2500 DDL3s10 Gb/s

L0

DataStorage

EPN

EPN

StorageNetwork

RORC3

RORC3

RORC3

RORC3

RORC3

RORC3

RORC3

RORC3

∞CLK


LS2 upgrade requirements (LoI)

Detector MAX R/O rate (kHz)

(pp and Pb-Pb)

Event Size Pb-Pb after ZS (MB)

ITS Continuous 0.2

TPC Continuous 1.0

TOF 200 – 400

TRD 27 – 100 0.2

EMCal 50

Muon 5The rate for heavy-ion events handled by the online systems up to permanent data storage should be increased up to 50 kHz (with a safety factor of 2) corresponding to roughly two orders of magnitude, compared to the present system.

Data compression will reduce the input peak data throughput of 1 TByte/s to an average recorded data output of 80 GB/s to a local data storage and 12 GB/s to the computing center.


Data compression 1Cluster finder (RORC3)

TriggerLevel 0,1

TriggerLevel 2

Event-Building Network

Detector

Digitizers

Front-end Pipeline/Buffer

Readout Buffer

Sub-event BufferFirst-Level Processor (FLP)

Data compression 2Event-Building

Event-building andProcessing Node (EPN)

Decision

Detector Data Link (DDL3)

Trigger & DAQ logical model

Decision

LHC Clock

Distribution of functionsas presented in the LoI

FastTrigger

Processor

DetectorElectronics

Online System

LHC Clock


TriggerLevel 0,1

TriggerLevel 2

Event-Building Network

Detector

Digitizers

Front-end Pipeline/Buffer

Readout Buffer

Sub-event BufferFirst-Level Processor (FLP)

Event-Building Data compression 2

(EPN)

Decision

Detector Data Link (DDL3)

Trigger & DAQ logical model

Data compression 1Cluster finder

Decision

Alternatives scenarios exist:e.g. cluster finder as part ofthe detector readout

LHC Clock LHC Clock

9

Initial requirements in the LoI. Carry on with functional requirements and R&D in

parallel Refine detectors functional requirements for DCS,

TRG, DAQ in view of the detector TDRs (2013) Online R&D to develop prototypes in view of

Online/Offline TDR (2014)o DDL2:

• Prototype characterization• Production for the TRD and the HLT

o DDL3• Technology selection

o Online dataflow demonstrator• Detector readout, FLP, network, EPN

Development process

09/11/2012

Filippo Costa, CERN


Higher reliabilityReduced Support

Issues well defined

Common hardware

and protocol

29/10/2012

Benefit of the commonality

So far we used a common protocol and hardware (SIU – DDL – RORC) for the read-out of all detectors.Lots of benefits:• having the same data transmission

protocol for all the detectors reduces specific debugging sessions.

• It encourages knowledge sharing between the detectors.

• It creates the “standard” in a custom protocol.

We are planning to follow the same idea in the future upgrades, using a common protocol for sending data to the DAQ system.


DDL SIU evolution

SIU1 SIU2 SIU3

1 ch @ 2 Gb/sACTEL FPGA

(CORE cost 560 CHF)

6 Gb/sXILINX / ALTERA /ACTEL

FPGA(CORE cost 0 CHF)

10 Gb/sXILINX / ALTERA / ACTEL

FPGA(CORE cost 0 CHF)

Custom DDL protocol Custom DDL protocol(same protocol but faster)

• Custom DDL 10 Gb/s• Ethernet @ 10 Gb/s• PCIe over fibre

RUN1LS1

RUN2LS2

RUN3

Det. Read-Out FPGA

SIU IP CORE

Det. Read-Out FPGA

SIU IP CORE


RORC evolution

RORC1 RORC2 (aka C-RORC) RORC3

TBD2 ch @ 2 Gb/s

PCIe gen.1 x4 (1 GB/s)ALTERA FPGA

12 ch @ 6 Gb/sPCIe gen.2 x8 (4 GB/s)

XILINX FPGA

12 ch @ 10 Gb/sPCIe gen.3

ALTERA / XILINX

Custom DDL protocol Custom DDL protocol(same protocol but faster)

• Custom DDL 10 Gb/s• Ethernet @ 10 Gb/s• PCIe over fibre

RUN1LS1

RUN2LS2

RUN3


Run 2 - DDL1 &

DDL2

Run 1 - DDL1

29/10/2012

Speed of the link

backward compatible

Same DDL protocol

No need to change the

readout hardware if not needed.

Transition DDL1 to DDL2

+


R&D DDL3

Evaluationprocess

DDL custom protocol

UDP Ethernet 10 Gb/s

PCIe over fibre

Other good options

DDL3

Data transmission protocols are under evaluation, for the time being no final decision has been taken yet.Each protocol has different pros and cons, tests started already now, soon to come a reasonable decision.


UDP Ethernet 10Gb

The evaluation for UDP Ethernet data transmission protocol has already started.Preliminary tests have been performed together with RD51 collaboration(Hans MULLER, Alfonso TARAZONA MARTINEZ).A test system has been prepared:

• 1 SRU board with a VIRTEX6 , 10 Gb IP OPENCORE.

• 1 Machine (DELL server power Edge r 720) with 10 Gb/s port embedded in the motherboard.

Continuous readout, no external trigger system, no timeout between 2 packets.


UDP Ethernet

10 Gb/s embedded in the motherboard

XILINX VIRTEX 6

DDL optical fibre

SFP+ 10 Gb/s


UDP Ethernet

0 1 2 3 4 5 6 7 8 90

200

400

600

800

1000

1200

1400

Data throughput vs. packet size

packet size (kB)

PEA

K d

ata

thro

ughput

(MB

/s)

0 1 2 3 4 5 6 7 8 90

50

100

150

200

250

300

350

400

Acquisition rate vs. packet size

packet size (kB)

Acquis

itio

n r

ate

(kH

z)

100 kHz


UDP Ethernet (pro/con)

Easy to implement in FPGA (IP cores available for 10 Gb/s).Fast and light protocol (no overhead for handshaking).Allow different configurations, point to point or network with routers.Reduce the hardware needed by the detector team to build test system, they can test their readout system using a standard Ethernet port of the PC.

Is it rad tool ? We (ALICE DAQ+RD51) are evaluating different solutions.SmartFusion2: XGXS/XAUI Extension (to implement a 10 Gbps (XGMII) Ethernet PHY interface)

Not reliable (but software checks can increase the reliability, backpressure algorithm implemented in UDP DATE).High CPU consuming, moving data from the Ethernet port to the memory, but different companies are already addressing the issue, PLDA and Solarflare.IP core license costs for 10 Gb/s can be expensive and not portable outside CERN, but OPENCORE can be a solution for that.


Det. readout electronics

29/10/2012

TTCrq/rx

All the detectors in ALICE receive the trigger messages through the TTCrx chip or the TTCrq board.Some of them will integrate the functionalities in the readout electronics.The code will be implemented in the FPGA, so there is no need anymore of the TTCrq board or TTCrx chip to be installed on each readout card, removing the problem of spare components.

TTC VHDL

FPGA


Radiation, how to beat it

Use of radiation tolerant FPGA:ACTEL (www.actel.com).

Using partial reconfiguration of FPGA reloading part of the firmware when:• SEU is detected, • periodically

CSABA SOOS “SEU effects in FPGA How to deal with them?”http://www.google.it/url?sa=t&rct=j&q=&esrc=s&source=web&cd=4&cad=rja&sqi=2&ved=0CEoQFjAD&url=http%3A%2F%2Findico.cern.ch%2FgetFile.py%2Faccess%3FcontribId%3D8%26resId%3D2%26materialId%3Dslides%26confId%3D56796&ei=zeeIUOzUFtCQswaRyIFQ&usg=AFQjCNFjtJxDqKSGVzYv6pD-x-2b4yfD4w

Thanks!

Documents

ALICE DAQ future detector readout