1

Latest ideas in DAQ development for LHC

B. Gorini - CERN

2

Guiding principles in latest studies

• Replace obsolete modular electronics technologies (e.g. VME)

• Whenever possible opt for COTS components

• Favor S/W over F/W when performance requirements allows it

3

Modular electronics replacement

• A clear choice for VMS replacement: ATCA

• ATCA is rather a packaging standard– High availability– Protocol agnostic backplane–… It offers less than VMS in terms of common

development environment

• A DAQ specific common platform is useful– E.g. the SLAC RCE for ATCA system

deployments in high energy physics

4

The RCE

• Modular DAQ based on generic ubiquitous building blocks:– Reconfigurable Cluster Elements (RCE)

• Mezzanines based on large FPGAs and System on Chip (now ARM processor)

• very high data bandwidth at input and multi-Gbps Ethernet output.

– Cluster Interconnects (CI) • On a generic ATCA carrier,

the Cluster On Board (COB)

5

• ATLAS, ALICE and LHCb are considering to move away from modular electronics for their back-end– The plan is to receive front-end links on

PCIe cards housed on PCs

Latest ideas: less modular electronics?

Detector F/E

Boards

Detector F/E

Boards

Detector F/E

Boards

Detector F/E

Boards

Detector F/E

Boards

Detector F/E

Boards

FPGAFPGA

FPGAF/E Links

6

LHCb original plan

7

LHCb current plan

8

The PCIe40 Board

9

FE FE FE FE FE

Today 2015 2017 2019 2021 2023 2025Run 4

PresentATLAS DAQ:

Custom electroniccomponentsROD ROD ROD ROD ROD

Custom point-to-point links

ReadOutDriver

HLTPUHLTP

UHLTPUHLTP

U

HLTPUHLTP

UHLTPUHLTP

U

HLTPUHLTP

UHLTPUHLTP

U

HLTPUHLTP

UHLTPUHLTP

U

HLTPUHLTP

UHLTPUHLTP

U

Ethernet

PCs

High-Level Trigger Farm

~1500nodes

~100machines

ROS ROS ROS ROS ROS

Point-to-point S-links

ReadOutSystem (Data Buffer)

100 kHz /~200 GB/s ~1800

links

40 MHz

10Run 4

FE FE FE FE FE

Readout

Readout

Readout

Readout

Readout

HLTPUHLTP

UHLTPUHLTP

U

HLTPUHLTP

UHLTPUHLTP

U

HLTPUHLTP

UHLTPUHLTP

U

HLTPUHLTP

UHLTPUHLTP

U

HLTPUHLTP

UHLTPUHLTP

U

HPC Network

Versatile Link, GBT, LpGBT

PCs

2023

High-Level Trigger Farm

HPC NetworkCOTS network technology

2015 2017 2019 2021 2023 2025

ProposedATLAS DAQ:

40 MHz

~10,000 links

~200 systems

less than10 TB/s

11

Dynamic routing rules, automatic failover and load balancing

Support multiple GBT modes, virtual e-links

Bi-directional connection with F/E

Trigger signal and LHC clock synchronous distribution

Routing of multiple traffic types: physics events, detector control, configuration, calibration, monitoring

FELIX functionality

Multi-cast, cloning, QoS

GBT: Rad-hard F/E link technology developed at CERNE-link: variable-width logical link on top GBT. Can be used to logically separate different streams on a single link.

12

Advantages of PCIe solutions

• Less custom electronics, more COTS components– Possibility of separate upgrade of FPGA, NICs, CPUs– Can follow processor computing power evolution

• Fully programmable F/E-B/E connectivity– Not limited to crate/shelf backplane capability

• Scalable architecture– Can add readout nodes as needed

• Dynamic load balancing– Programmable data routing algorithms

• Increased fail safety– Automatic recovery of readout node crashes

• Main questions:– Probably increased latency– Most data processing in processors rather than FPGAs

13

FELIX Demonstrator System

FELIX

FPGA Card

Memory

Large Buffers per group of elinks

CPU

Custom DeviceDriver

FELIXApplication

PCIe Gen-38 lanes

DMA

Config

GBT

Optical Links

MSI-X

DMA

TTC FMC 64 Gb/s

NIC

Optical Links

64 Gb/s

24 – 48links

2 – 440-Gb/sports

Elink router

“TTC” is the Timing, Trigger and Control System

PCIe Gen-38 lanes

14

FELIX development Platform

HiTech Global PCIe development board with TTCfxXilinx Virtex-7PCIe Gen-2/3 x824 bi-directional linkshttp://hitechglobal.com/Boards/PCIE-CXP.htm

SuperMicro X10DRG-Q2x Haswell CPU, up to 10 cores6x PCIe Gen-3 slots64 GB DDR4 Memoryhttp://supermicro.com/products/motherboard/Xeon/C600/X10DRG-Q.cfm

Mellanox ConnectX-3 VPIFDR/QDR Infiniband2x10/40 GbEhttp://www.mellanox.com/page/products_dyn?product_family=119&mtag=connectx_3_vpi

15

Perspectives• LHCb is advancing in the design of their custom PCIe board

(PCIe40) – Were expecting first prototype by last December

• ALICE plan is to use LHCb PCIe H/W • ATLAS will complete the implementation of a demonstrator based

on a commercial board– Initial design review planned after completion– Decision whether to develop custom board or stay with a commercial one

will follow

• Commercial market is becoming latency aware– Hedge funds are driving (at least) a big computer manufacturer to develop

similar FPGA/PCIe card to support high frequency trading requirements

• There is room for considering a common solution – Ideally purely based on COTS (including PCIe boards) – Ideally only software development (F/W will be needed only for readout link

interface) – Following development (Gen 4) or replacement (?) of PCIe for higher

throughput reach