2
Guiding principles in latest studies
• Replace obsolete modular electronics technologies (e.g. VME)
• Whenever possible opt for COTS components
• Favor S/W over F/W when performance requirements allows it
3
Modular electronics replacement
• A clear choice for VMS replacement: ATCA
• ATCA is rather a packaging standard– High availability– Protocol agnostic backplane–… It offers less than VMS in terms of common
development environment
• A DAQ specific common platform is useful– E.g. the SLAC RCE for ATCA system
deployments in high energy physics
4
The RCE
• Modular DAQ based on generic ubiquitous building blocks:– Reconfigurable Cluster Elements (RCE)
• Mezzanines based on large FPGAs and System on Chip (now ARM processor)
• very high data bandwidth at input and multi-Gbps Ethernet output.
– Cluster Interconnects (CI) • On a generic ATCA carrier,
the Cluster On Board (COB)
5
• ATLAS, ALICE and LHCb are considering to move away from modular electronics for their back-end– The plan is to receive front-end links on
PCIe cards housed on PCs
Latest ideas: less modular electronics?
Detector F/E
Boards
Detector F/E
Boards
Detector F/E
Boards
Detector F/E
Boards
Detector F/E
Boards
Detector F/E
Boards
FPGAFPGA
FPGAF/E Links
9
FE FE FE FE FE
Today 2015 2017 2019 2021 2023 2025Run 4
PresentATLAS DAQ:
Custom electroniccomponentsROD ROD ROD ROD ROD
Custom point-to-point links
ReadOutDriver
HLTPUHLTP
UHLTPUHLTP
U
HLTPUHLTP
UHLTPUHLTP
U
HLTPUHLTP
UHLTPUHLTP
U
HLTPUHLTP
UHLTPUHLTP
U
HLTPUHLTP
UHLTPUHLTP
U
Ethernet
PCs
High-Level Trigger Farm
~1500nodes
~100machines
ROS ROS ROS ROS ROS
Point-to-point S-links
ReadOutSystem (Data Buffer)
100 kHz /~200 GB/s ~1800
links
40 MHz
10Run 4
FE FE FE FE FE
Readout
Readout
Readout
Readout
Readout
HLTPUHLTP
UHLTPUHLTP
U
HLTPUHLTP
UHLTPUHLTP
U
HLTPUHLTP
UHLTPUHLTP
U
HLTPUHLTP
UHLTPUHLTP
U
HLTPUHLTP
UHLTPUHLTP
U
HPC Network
Versatile Link, GBT, LpGBT
PCs
2023
High-Level Trigger Farm
HPC NetworkCOTS network technology
2015 2017 2019 2021 2023 2025
ProposedATLAS DAQ:
40 MHz
~10,000 links
~200 systems
less than10 TB/s
11
Dynamic routing rules, automatic failover and load balancing
Support multiple GBT modes, virtual e-links
Bi-directional connection with F/E
Trigger signal and LHC clock synchronous distribution
Routing of multiple traffic types: physics events, detector control, configuration, calibration, monitoring
FELIX functionality
Multi-cast, cloning, QoS
GBT: Rad-hard F/E link technology developed at CERNE-link: variable-width logical link on top GBT. Can be used to logically separate different streams on a single link.
12
Advantages of PCIe solutions
• Less custom electronics, more COTS components– Possibility of separate upgrade of FPGA, NICs, CPUs– Can follow processor computing power evolution
• Fully programmable F/E-B/E connectivity– Not limited to crate/shelf backplane capability
• Scalable architecture– Can add readout nodes as needed
• Dynamic load balancing– Programmable data routing algorithms
• Increased fail safety– Automatic recovery of readout node crashes
• Main questions:– Probably increased latency– Most data processing in processors rather than FPGAs
13
FELIX Demonstrator System
FELIX
FPGA Card
Memory
Large Buffers per group of elinks
CPU
Custom DeviceDriver
FELIXApplication
PCIe Gen-38 lanes
DMA
Config
GBT
Optical Links
MSI-X
DMA
TTC FMC 64 Gb/s
NIC
Optical Links
64 Gb/s
24 – 48links
2 – 440-Gb/sports
Elink router
“TTC” is the Timing, Trigger and Control System
PCIe Gen-38 lanes
14
FELIX development Platform
HiTech Global PCIe development board with TTCfxXilinx Virtex-7PCIe Gen-2/3 x824 bi-directional linkshttp://hitechglobal.com/Boards/PCIE-CXP.htm
SuperMicro X10DRG-Q2x Haswell CPU, up to 10 cores6x PCIe Gen-3 slots64 GB DDR4 Memoryhttp://supermicro.com/products/motherboard/Xeon/C600/X10DRG-Q.cfm
Mellanox ConnectX-3 VPIFDR/QDR Infiniband2x10/40 GbEhttp://www.mellanox.com/page/products_dyn?product_family=119&mtag=connectx_3_vpi
15
Perspectives• LHCb is advancing in the design of their custom PCIe board
(PCIe40) – Were expecting first prototype by last December
• ALICE plan is to use LHCb PCIe H/W • ATLAS will complete the implementation of a demonstrator based
on a commercial board– Initial design review planned after completion– Decision whether to develop custom board or stay with a commercial one
will follow
• Commercial market is becoming latency aware– Hedge funds are driving (at least) a big computer manufacturer to develop
similar FPGA/PCIe card to support high frequency trading requirements
• There is room for considering a common solution – Ideally purely based on COTS (including PCIe boards) – Ideally only software development (F/W will be needed only for readout link
interface) – Following development (Gen 4) or replacement (?) of PCIe for higher
throughput reach