1

DAQ and Trigger upgrade

U. Marconi, INFN BolognaFirenze, March 2014

DAQ and Trigger TDR• TDR planned for June 2014• Sub-systems involved:– Long distance optical fibres–Readout boards (PCIe40) and

Event Builder Farm– The Event Filter Farm for the HLT– Firmware, LLT, TFC

2

3

Readout System Review• Held on Feb 25th, 9:00 – 16:00• 4 reviewers– Christoph Schwick (PH/CMD – CMS DAQ)– Stefan Haas (PH/ESE)– Guido Haefeli (EPFL)– Jan Troska (PH/ESE)

• https://indico.cern.ch/event/297003/• All documents can be found in EDMS

https://edms.cern.ch/document/1357418/1

4

Trigger LHCb TB MeetingLHCb Technical Board Meeting 18/03/2014

Full software trigger,software LLT

R. Legac, Trigger Coordinator

5

Long distance fibres

The distance to cover with 850 nm OM multimode optical cables, from underground to the surface, is 300 m.

Locate the Event Builder Farm and the Event Filter Farm for the HLT outside the cavern.

• Minimum required bandwidth: 32 Tbit/s – # of 100 Gigabit/s links > 320, – # of 40 Gigabit/s links > 800, – # of 10 Gigabit/s links > 3200 – # of 5Gbit GBT links (4.5 Gbit effective):

> 10000– Current estimate O(15000), DAQ, Controls

and Timing system. Spares included

6

Long distance fibres (II)

• 144 fibres per cable. A total of 120 such cables.

• 3 patch panels (breakpoints) foreseen:expected attenuation ~ 3dB

• 4.8 Gbit/s signal produced on the detector by Versatile Link transmitters.

• VTTx to MiniPod for data acquisition.• MiniPod to VTRx for control, configuration.

MiniPod

7

Optical fibres studies

Measurement of BER vs. receive OMA(*) on different OM3 and OM4 fibres.(*) optical modulation amplitude

Transmission test at 4.8 Gb/s

The target value of 3 dB on OM3 using the Versatile Link and MP is reachable.

8

Optical linksOM3 vs OM4 A shortest path seems possible

LHCb DAQ today

2 x F10 E1200i

56 sub-farms

Readout boards: 313 x TELL1

9

Push-protocol with centralized flow-control

# links (UTP Cat 6) ~ 3000

Event-size (total – zero-suppressed)

65 kB

Read-out rate 1 MHz

# read-out boards 313

output bw / read-out board up to 4 Gbit/s (4 Ethernet links)

# farm-nodes 1500 (up to 2000)

max. input bw / farm-node 1 Gbit/s

# core-routers 2

# edge routers 56

DAQEvent builder

High speed network

HLT Event filter

10

PCIe40

PCIe40

PCIe40

PCIe40

PCIe40

PCIe40

PCIe40

PCIe40

16-lane PCIe-3 16-lane PCIe-3

EBF

EFF

Readout board review• Basic question to reviewers:

“We propose a change of baseline from ATCA/AMC to PCIe. Do the reviewers support the choice of baseline (PCIe) and back-up (ATCA)?”

• Answer:“Given the listed advantages the review committee endorses the choice of the PCIe based design as a baseline.”

11

12

PCIe Gen3 based readout A main FPGA manages the input streams and transmits data

to the event-builder server using PCIe Gen3. PCIe Gen3 throughput: 16-lane × 8 Gb/s/lane = 128 Gb/s The readout version of the board uses two de-serializers.

16-lane PCIe-3 edge connectorto the motherboard of the host PC

24 optical linksfrom the FEE DMA over 8-lane

PCIe-3 hard IP blocks

U. Marconi et al. The PCIe Gen3 readout for the LHCb

upgrade, CHEP2013

13

PCIe-3 test board• ALTERA development board, equipped with a Stratix V GX FPGA, model

5SGXEA7K2F40C2N

http://www.altera.com/literature/manual/rm_svgx_fpga_dev_board.pdf“The Stratix V GX FPGA development board is designed to fit entirely into a PC motherboard with a ×8 PCI Express slot that can accommodate a full height short form factor add-in card. This interface uses the Stratix V GX FPGA device's PCI Express hard IP block, saving logic resources for the user logic application”.

8-lane edge-connector

14

The PCIe-3 DMA test setup

GPU used to test 16-lane PCIe-3data transfer between the device and the host memory

The ALTERA development board8-lane PCIe-3

15

The DMA PCIe-3 effective bandwidth

DMA

over

8-la

ne

PCIe

-3 h

ard

IP b

lock

sAL

TERA

Str

atix

V

16

Test of PLX bridge

• Long-term test using GTX690 card / PLX 8747 bridge.

• Zero impact of using a bridge and two independent PCIe targets pushing data into a PC. Consistently around 110 Gb/s over long-term.

• No load balancing issues between the two competing links observed.

• Details on: https://lbonupgrade.cern.ch/wiki/index.php/I/O_performance_of_PC_servers#Upgrade_to_GTX690

10 h

110 Gb/s

PLXGPU

1GPU

2

PCIe 16

PLX

8-lane

16-lane

PCIe switch

Event builder node

17

PCIe40Event BuildingNetwork Interface data from

the detector~ 100 Gb/s

to theevent builderDual-port IB FDR – 110 Gb/s

from theevent builder

events that are beingbuilt on this machine

opportunity for doingpre-processing of full event here

to the HLT

empty

DDR340-50 GB/sHalf duplex

2x50 GB/sFull duplex

DDR340-50 GB/sHalf duplex

18

EVB performance

• Event Builder execution requires around 6 logical cores• 18 instances of the HLT software

CPU consumption Memory I/O bandwidth

• Event Builder performs stably at 400 Gb/s

PC sustains 100 Gb/s Event Builder todayIvy Bridge Intel dual CPU 12 cores

We currently see 50% free resources for opportunistic triggering on EB nodes

19

Fast Control

Costing (preliminary)

• Long-distance fibres:~ 1.6 MCHF 15000 fibres OM3, 300 m.– Excluding patch-cords to

detector and TELL40, cable-ducts, but including patch-panels, installation and testing

– No contingency, but several quotes.

• PCIe40:DAQ version: 5.8 kCHF ECS/TRG version: 7.9 kCHF– includes 15% contingency

• Readout network,Event-building (PCIe40) ~ 3.6 MCHF – Including event-builder PCs and event-

filter network, excluding farm-servers– Model based on InfiniBand FDR (2013

quotes)• Event-building (AMC40)

~ 9 MCHF – including event-filter network– Model based on 40G and 10G

Ethernet (on 2013 quotes)

20

21

Running conditions• Visible rate and mean visible interactions per crossing• At 2.×1033 , 27 MHz, 5.2 interaction per crossing

Mean visible interactions per crossing

22

LHCb upgrade: HLT farm• Trigger-less system at 40 MHz:

A selective, efficient and adaptable software trigger. • Average event size: 100 kB• Expected data flux: 4 TB/s• Total HLT trigger process latency: 14 ms

– Tracking time budget (VELO + Tracking + PV searches): 50% – Tracking finds 99% of offline tracks with pT >500 MeV/c

• Number of running trigger process required: 4.×105

• Number of core/CPU available in 2018: ~ 200– Intel tick-tock plan: 7nm technology available by 2018-19, the n.

of core accordingly scales as 12. × (32 nm/ 7 nm)2 = 250• Number of computing nodes required: ~ 1000

23

LLT

Readout reviewers’ opinion

… only the software option should bepursued if any.

24

LLT performance

Person-power

• Fibres: (CERN)– Tests and preparation of installation 0.5 py (person-year)– Installation & testing by external company 0.5 py for supervision and follow-up

• PCIe40: (Marseille, Bologna)– design, production, commissioning ~ 18 py

• LLT: (Marseille, Annecy, Clermont Fd., LAL) ~ 17 py • Firmware: Global framework, TFC, DAQ (Marseille, Bologna, CERN, Annecy) ~ 18

py (estimated)• DAQ: (CERN, Bologna)

– ~ 15 py (excluding ECS and high-level trigger infrastructure)• Overall person-power in involved institutes is sufficient, but does not allow for any

“luxuries”, as many people are also involved in the operation of LHCb during Run 2

25

N. Neufeld

Schedule TELL40

26

Schedule readout board

PCIe40 board plan: as INFN we should develop the PCB and prototypes, to qualify our production: be ready to produce the boards.

Sblocco 25 kE s.j.

27

Schedule DAQ

• Assume start of data-taking in 2020– System for SD-tests ready whenever needed minimal

investment• 2013 – 16: technology following (PC and network)• 2015 – 16: Large scale network IB and Ethernet tests• 2017: tender preparations • 2018: Acquisition of minimal system to be able to read out

every GBT– Acquisition of modular data-center

• 2019: Acquisition and Commissioning of full system– starting with network– farm as needed

N. Neufeld

28

Schedule: firmware, LLT, TFC

• All these are FPGA firmware projects• First versions of global firmware and TFC ready

now (for MiniDAQ test-systems)– then ongoing development

• LLT – Software 2014 - 2015– Hardware 2016 – 2017 (?)

29

Schedule long-distance fibres• Test installation 2014

– validate installation procedure and pre-select provider

• Long-term fibre test with AMC40/PCIe40 on long-distance 2014/2015

• Full installation in LS1.5 or during winter-shutdowns / to be finished before LS2

• Assumptions:– Installation can be done

without termination in UX (cables terminated on at least one end), if blown, fibres can be blown from bottom to top