Data Acquisition Systems for Future Calorimetry at the International Linear Collider Matt Warren, on behalf of CALICE-UK Collaboration

Data Acquisition Systems Data Acquisition Systems forfor

Future Calorimetry Future Calorimetry at theat the International Linear Collider International Linear Collider

Matt Warren, on behalf of CALICE-UK CollaborationMatt Warren, on behalf of CALICE-UK Collaboration

29 Oct 07 Matt Warren - DAQ for Calorimetry at ILC 2

IntroductionIntroduction•Building a DAQ for multiple ILC CAL sub-detector prototypesBuilding a DAQ for multiple ILC CAL sub-detector prototypes

–Paying attention real ILC environmentPaying attention real ILC environment

–EUDET Testbeam in 2009! (EU funded DETector project shares much of CALICE) EUDET Testbeam in 2009! (EU funded DETector project shares much of CALICE)

•For economies of scale, we are attempting a generic DAQ for many (even SLHC??)For economies of scale, we are attempting a generic DAQ for many (even SLHC??)–Modular/Generic Structure:Modular/Generic Structure:

• Generic readout system as much as possibleGeneric readout system as much as possible

• Detector specific interfaces only at ends of chainDetector specific interfaces only at ends of chain

• Other ‘bespoke’ functionality in firmwareOther ‘bespoke’ functionality in firmware

–Commercial components and protocols where possibleCommercial components and protocols where possible• Readout links use standard connectors and protocolsReadout links use standard connectors and protocols

• Based on PCs with PCIe cards (“backplaneless”)Based on PCs with PCIe cards (“backplaneless”)

•Clock and Control attempts commercial hardware tooClock and Control attempts commercial hardware too–Extract clock and ‘fast’ signals from commercial signallingExtract clock and ‘fast’ signals from commercial signalling

•Software generic for all detectorsSoftware generic for all detectors–Try use something off-the-shelf …Try use something off-the-shelf …

BUT first, an introduction to the ILC CAL …BUT first, an introduction to the ILC CAL …


ILC CalorimetryILC Calorimetry• ILC Calorimetry is dense and high granularity ILC Calorimetry is dense and high granularity

– squeezed between large tracker & expensive coilsqueezed between large tracker & expensive coil– >100M channels>100M channels– No roomNo room for electronics or cooling. for electronics or cooling.

• Bunch structure Bunch structure interesting:interesting:– ~200ms gaps between bunch-trains~200ms gaps between bunch-trains– Trains 1ms long, 300ns bunch spacingTrains 1ms long, 300ns bunch spacing

• TriggerlessTriggerless – – sample data from sample data from everyevery bunch- bunch-crossing crossing

SO (the problem):SO (the problem):• 100M channels, analog signals 100M channels, analog signals

= front-end = front-end electronics inside detectorelectronics inside detector• Results in high power densityResults in high power density

– but no room for coolingbut no room for cooling• Long gap allows chips a 1% duty cycleLong gap allows chips a 1% duty cycleSolution: Solution: Power PulsingPower Pulsing

Time structure of bunches

Trains of bunches

Individual bunches

ECAL

HCAL

M. Anduze


Sub-Detector Geometries Sub-Detector Geometries (+ASICs)(+ASICs)

ASICS• Must share readout resource (daisy chain)• Bunch rate too high for instantaneous data

transfer.• Too much chip resource to store all eventsSO:• ‘‘Auto-trigger’ – store only data over-threshold Auto-trigger’ – store only data over-threshold

with pad id + (bunch-number)with pad id + (bunch-number)• <5kByte / bunch-train/ASIC<5kByte / bunch-train/ASIC

HCAL half-octant ECAL Module-0 (reduced-Z octant)

L = 150 cm

ASIC (>100 in total!)

Detector Unit(e.g. ECAL Slab)

M. Anduze


DAQ architecture DAQ architecture Detector Unit:Detector Unit: Sensors & ASICs Sensors & ASICs DIF:DIF: Detector InterFace -connects generic DAQ and services Detector InterFace -connects generic DAQ and servicesLDA:LDA: Link/Data Aggregator – fanout/in DIFs & drive link to ODR Link/Data Aggregator – fanout/in DIFs & drive link to ODRODR:ODR: Off Detector Receiver – PC interface for system. Off Detector Receiver – PC interface for system.C&C:C&C: Clock & Control: Fanout to ODRs (or LDAs) Clock & Control: Fanout to ODRs (or LDAs)

LDA

LDAHost PC

PC

Ie

ODR

Host PC

PC

Ie

ODR

DetectorUnit

DIF

C&C

DetectorUnit

DIF

DetectorUnit

DIF

DetectorUnit

DIF

Storage

1-3Gb Fibre50-150 Mbps HDMI

cabling

10-100m0.1-1m

Det

ecto

r

Co

un

tin

g R

oo

m


DIF (DIF (DDetectoretector I InternterFFace)ace)• FPGA + detector hardware connected to Detector FPGA + detector hardware connected to Detector

Unit.Unit.

• Two halves – Generic DAQ and Specific DetectorTwo halves – Generic DAQ and Specific Detector– 3 detectors: ECAL, AHCAL, DHCAL3 detectors: ECAL, AHCAL, DHCAL

– 1 DAQ Interface!1 DAQ Interface!

Focusing on the DAQ side:Focusing on the DAQ side:

• From LDA, receive, decode/regenerate and From LDA, receive, decode/regenerate and distribute clocks, fast commands, config data and distribute clocks, fast commands, config data and slow controls.slow controls.

• From ASICs, receive, buffer, package and From ASICs, receive, buffer, package and forward data to LDAforward data to LDA

– ASICs power-up and read-out power-down in turnASICs power-up and read-out power-down in turn

• ALSO: USB interfaceALSO: USB interface– Hardware designers already have oneHardware designers already have one

– DAQ plans to integrate for stand-alone testingDAQ plans to integrate for stand-alone testing

DetectorUnit

DIF

USB

DAQ

e.g. ECAL M. Goodrick


DIF-LDA linkDIF-LDA link• Serial links running at multiple of Serial links running at multiple of

machine clockmachine clock

• ~50Mbps (raw) bandwidth minimum~50Mbps (raw) bandwidth minimum

• robust encoding (8B/10B)robust encoding (8B/10B)

• HDMI cables/connectors interface.HDMI cables/connectors interface.– Commercially available cablesCommercially available cables

– Rated >300MbRated >300Mb

– Even halogen free availableEven halogen free available

• Signals (ideally just TX/RX but …): Signals (ideally just TX/RX but …): Clock (diff)Clock (diff)

Control/Fast (diff)Control/Fast (diff)

Very Fast (diff)Very Fast (diff)

Data (diff)Data (diff)

single ended aux x2 (or UTP)single ended aux x2 (or UTP)

• LDAs serve even/odd DIFs for LDAs serve even/odd DIFs for redundancyredundancy

LDA

LDA

DetectorUnit

DIF

DetectorUnit

DIF

DetectorUnit

DIF

DetectorUnit

DIF


LDA (LDA (LLink/ink/DDataata A Aggregator)ggregator)• Located as close as possible to DIFsLocated as close as possible to DIFs

– Shortest cables, but convenient location (space, cooling)Shortest cables, but convenient location (space, cooling)

• Supports as many DIFs as possible considering bandwidth Supports as many DIFs as possible considering bandwidth and physical constrainsand physical constrains

– Ideally 50 (20Mbps/DIF)Ideally 50 (20Mbps/DIF)

– Prototype will have 10Prototype will have 10

• Aggregates front-end data and sends it off-detectorAggregates front-end data and sends it off-detector– Fibre optic link. 1-3Gbps, with SFP (see next slide)Fibre optic link. 1-3Gbps, with SFP (see next slide)

• Fanout C+C+C to DIFsFanout C+C+C to DIFs

• USB interface for stand-alone/top-of-chain testingUSB interface for stand-alone/top-of-chain testing

LDA

LDA

• Currently using a commercial FPGA dev-board:Currently using a commercial FPGA dev-board:Enterpoint Broaddown2 – Xilinx Spartan3-200Enterpoint Broaddown2 – Xilinx Spartan3-200• With add-on boards for our needsWith add-on boards for our needs

– SPF+SerDes for ODR link SPF+SerDes for ODR link – 10 HDMI connectors with clock fanout10 HDMI connectors with clock fanout


ODR ODR ((OOff ff DDetector etector RReceiver) + eceiver) + LinkLink

Host PC

PC

Ie

ODR

Storage

C&C

Hardware:Hardware:

• Using commercial FPGA dev-board:Using commercial FPGA dev-board:– PLDA XPressFX100PLDA XPressFX100

– Xilinx Virtex 4, 8xPCIe, 2x SFP (3 more with expansion Xilinx Virtex 4, 8xPCIe, 2x SFP (3 more with expansion board)board)

• Our own firmware and Linux driver softwareOur own firmware and Linux driver software

SFPs for optic link

Expansion (e.g. 3xSFP)

• Receives module data from LDAReceives module data from LDA– PCI-Express card, hosted in PC. PCI-Express card, hosted in PC.

– 1-4 links/card (or more), 1-2 cards/PC1-4 links/card (or more), 1-2 cards/PC

– Buffers and transfers to store as fast as possibleBuffers and transfers to store as fast as possible

• Fibre optic link to detector via SFP modules (std networking hw)Fibre optic link to detector via SFP modules (std networking hw)– Currently GigE (1.25Gb), but could higher and use different proto.Currently GigE (1.25Gb), but could higher and use different proto.

• Sends controls and config to LDA for distribution to DIFsSends controls and config to LDA for distribution to DIFs

• Interfaces to C+C for synchro runningInterfaces to C+C for synchro running– Goal to send clock and prompt controls over optic link tooGoal to send clock and prompt controls over optic link too

– Reset and reprog FPGAsReset and reprog FPGAs

• Performance studies & optimisation on-going (see next slide):Performance studies & optimisation on-going (see next slide):– Bottleneck in writing data to disk.Bottleneck in writing data to disk.


ODR Throughput MeasurementsODR Throughput Measurements

0

50

100

150

200

250

300

350

400

32 410 1024 1800 2900 4200 6500

data size [bytes]

tra

ns

fer

rate

[M

B/s

]

Net, no disk

Net, disk write

NDG

IDG, no disk

IDG, disk write

DMA function, no datawrite

Ethernet framesize

NDG – Network Data GeneratorIDG – Internal (ODR) Data Generator

• All measurementsAll measurements: single requester thread, single IO thread (disk write),: single requester thread, single IO thread (disk write),

• Each event fragment written to a separate file. Data written to the localdisk (fs: ext3)Each event fragment written to a separate file. Data written to the localdisk (fs: ext3)

e.g. e.g. WORST CASE!WORST CASE! ` `

NDG plot – between two separatemachines, Gigabit, copper Eth on both sides

Problem with test!

A. Misiejuk


Clock & ControlClock & Control• C&C unit provides machine clock and fast signals C&C unit provides machine clock and fast signals

to 8x ODR/LDA.to 8x ODR/LDA.• Logic control (FPGA, connected via USB)Logic control (FPGA, connected via USB)

– Command encodersCommand encoders

– Remote signal enable, clock selectionRemote signal enable, clock selection

– But capable of stand-alone, dumb modeBut capable of stand-alone, dumb mode

• Provision for async scintillator type signals (VFast)Provision for async scintillator type signals (VFast)• LDA provides next stage fanout to DIFsLDA provides next stage fanout to DIFs

– Eg C&C unit -> 8 LDAs -> 10 DIFs = 80 DUs. Eg C&C unit -> 8 LDAs -> 10 DIFs = 80 DUs.

• Signalling over same HDMI type cablingSignalling over same HDMI type cabling• Facility to generate optical link clock (~125-Facility to generate optical link clock (~125-

250MHz from ~50MHz machine clock)250MHz from ~50MHz machine clock)

• Commercial systems are not ideal here.Commercial systems are not ideal here.– Looking at custom protocol on fibre optic linkLooking at custom protocol on fibre optic link

– Prompt signals and low jitter clock recovery needs further investigationPrompt signals and low jitter clock recovery needs further investigation

LDA

LDA

Host PC

PC

Ie

ODR

C&C

Host PC

PC

Ie

ODR

Machine

Run-Control


Software and OperationSoftware and OperationSoftware:Software:

Looking for OTS software to cover both slow and fast controls:Looking for OTS software to cover both slow and fast controls:

Early days – examining EPICs, ACE and DOOCS (the current favourite)Early days – examining EPICs, ACE and DOOCS (the current favourite)

- DOOCS is open source, actively developed, slow and fast controls, and - DOOCS is open source, actively developed, slow and fast controls, and already used by ILC communityalready used by ILC community

Operation:Operation:

Two modes:Two modes:

1) Configure: PCs controlled over network to send configuration to LDAs 1) Configure: PCs controlled over network to send configuration to LDAs and DIFsand DIFs

2) Run:2) Run:– LDA/DIF set to data-taking modeLDA/DIF set to data-taking mode

– ODR configured for data reception, control handed to central.ODR configured for data reception, control handed to central.

OROR ODR needs no control – simply waits ODR needs no control – simply waits

– Bunch-train starts/stop signals sent to LDAs control data flow.Bunch-train starts/stop signals sent to LDAs control data flow.

Fairly autonomous system (i.e. no trigger!)Fairly autonomous system (i.e. no trigger!)


Summary (an example – Summary (an example – AHCAL)AHCAL)

LDA

DIF

Detector Unit

Off-Detector

DAQ

P. Göttlicher, DESY

Documents

Data Acquisition Systems for Future Calorimetry at the International Linear Collider Matt Warren, on behalf of CALICE-UK Collaboration