Design and Evaluation of LAr Trigger DigitizerBoard in ATLAS Phase-I Upgrade

Kai Chen∗, Hucheng Chen∗, Mauro Citterio‡, Herve Deschamps†, Aude Grabas†, Stefano Latorre‡,Massimo Lazzaroni‡, Hongbin Liu∗, Phillippe Schwemling†, Stefan Simion§

, Hao Xu∗ and Heling Zhu¶,∗Physics Department, Brookhaven National Laboratory, Upton, NY, USA

†CEA-Saclay, Paris, France‡INFN Milano, Milan, Italy§LAL, Orsay, France

¶University of Science and Technology of China, Hefei, China

Abstract—The LHC upgrade is planned to enhance the instan-taneous luminosity during Run 3 from 2021 through 2023. ThePhase-I upgrade of the trigger readout electronics for the ATLASLiquid Argon (LAr) Calorimeters will be installed during thesecond long shutdown of LHC in 2019-2020. In this upgrade, theso-called super cells are introduced to provide higher granularity,higher resolution and longitudinal shower shape informationfrom the LAr calorimeters to the level-1 trigger processors. Anew LAr Trigger Digitizer Board (LTDB) will manipulate anddigitize 320 channels of super cell signals, and transmit it via40 fiber optical links to the back end where data are furtherprocessed and transmitted to the trigger processors. Five pairsof bidirectional GBT links are used for slow control from theFront-end LInks eXchange (FELIX) in the ATLAS TDAQ system.LTDB also outputs 64 summed analog signals to the currentTower Builder Board via the new baseplane. A test system isdeveloped to test all functions of the LTDB and carry out theperformance measurement. A back end PCIe card is designedwhich has the circuit to interface to the ATLAS trigger, timeand control system. It can control the generation of injectionsignals to the LTDB for performance test. It also configures andcalibrate all ASICs on the LTDB.

I. INTRODUCTION

ATLAS Liquid Argon (LAr) calorimeter consists of theelectromagnetic barrel, the electromagnetic end-cap, thehadronic end-cap and the forward calorimeter[1][2]. The posi-tion of these calorimeters are shown in the Figure 1. In currentLAr trigger readout, for each area of size ∆η×∆φ = 0.1×0.1,the Layer Sum Board (LSB), as mezzanines on the front-endboard will sum and get the energy deposition across each ofthe four longitudinal layers in the calorimeter. As depicted inFigure 2, the so called Tower Builder Board will further sumthese four energies together, and form a trigger tower with thegranularity of 0.1 × 0.1.

The second long shutdown of LHC is scheduled in 2019-2020. For LAr calorimeters of the ATLAS detector, the Phase-I upgrade of the trigger readout electronics will be installed.The objective of this upgrade is to provide higher granularity,higher resolution and longitudinal shower information[3]. Asshown in the Figure 2, after the upgrade, the level-1 triggergranularity will be improved. One current trigger tower willhas 10 so called super cells. The information from each layer

Fig. 1: ATLAS Liquid Argon calorimeter

is retained, and the granularity can be fine up to ∆η × ∆φ =0.025 × 0.1. There will be about 34000 super cells in total.All of them will be sampled at every LHC bunch-crossing ata frequency of 40 MHz.

Technical Design ReportDecember 2, 2013

ATLASLiquid Argon Calorimeter Phase-I Upgrade

Table 1. Comparison of the current Trigger Tower granularity vs. the proposed Super Cell granularity in theLAr EM barrel calorimeter, in terms of both elementary cells and ∆η and ∆φ. The number of elementary cellsgrouped for the trigger readout in η and φ are indicated by nη and nφ, respectively.

Elementary Cell Trigger Tower Super CellLayer ∆η×∆φ nη×nφ ∆η×∆φ nη×nφ ∆η×∆φ

0 Presampler 0.025 × 0.1 4 × 1

0.1 × 0.1

4 × 1 0.1 × 0.11 Front 0.003125 × 0.1 32 × 1 8 × 1 0.025 × 0.12 Middle 0.025 × 0.025 4 × 4 1 × 4 0.025 × 0.13 Back 0.05 × 0.025 2 × 4 2 × 4 0.1 × 0.1

FrontSC_layer=1SC_region=0SC_eta=0...55 [ =0.025]SC_region=1SC_eta=56..58 [ =0.025]SC_phi=0...63 [ =0.1]

BackSC_layer=3SC_region=0SC_eta=0...12 [ =0.1]SC_eta=13 [ ~0.05]SC_phi=0...63 [ =0.1]

60 cells per Trigger Tower; all layers summed

EM layer 0Presampler: 4x1 x =0.025x0.1)

EM layer 1Front: 32x1 x =0.003125x0.1)

EM layer 2Middle: 4x4 x =0.025x0.025)

EM layer 3Back: 2x4

x =0.05x0.025)

LAr EM Barrel

PresamplerSC_layer=0SC_region=0SC_eta=0...13 [ =0.1]SC_region=1SC_eta=14(15) [ ~0.1(0.12)]SC_phi=0...63 [ =0.1]

Phase-I Upgrade Level-1 Trigger Granularity (Super Cells)

EM layer 0Presampler: 4x1 x =0.025x0.1)

EM layer 1Front: 8x1 x =0.003125x0.1)

EM layer 2Middle: 1x4 x =0.025x0.025)

EM layer 3Back: 2x4 x =0.05x0.025)

MiddleSC_layer=2SC_region=0SC_eta=0...55 [ =0.025]SC_region=1SC_eta=56 [ c0.075]SC_phi=0...63 [ =0.1]

Level-1 Trigger Granularity (Trigger Towers)10 Super Cells per Trigger Tower

Existing System

Figure 3. Geometrical representation in η,φ space of an EM Trigger Tower in the current system, wherethe transverse energy in all four layers are summed (left) and of the Super Cells proposed for the Phase-Iupgrade, where the transverse energy in each layer is retained in addition to the finer granularity in the frontand middle layers (right). Each square represents an area of size ∆η×∆φ = 0.1×0.1.

end and back-end electronics is described in detail in Sec. 4 and 5 respectively.To provide high-granularity and high-precision information to upgraded trigger processors called

Feature EXtractors (FEXs) [4], new LAr Trigger Digitizer Boards (LTDB) are installed in the availablespare slots of the Front-End crates. The upgrade of the layer sum boards and of the baseplanesallows the LTDBs to digitize information with granularity up to ∆η×∆φ = 0.025×0.1 in the front andmiddle layers of the EM calorimeters. The LTDB also recreates the 0.1×0.1 analog sums and feedsthem back to the Tower Builder Board (TBB) to maintain the “legacy” system as fully operational.The digitized signals are processed remotely by the LAr Digital Processing System (LDPS) modules,which convert the samples to calibrated energies in real-time and interface to the FEX processors.

4 Chapter 1: Overview of the Phase-I LAr upgrade project

Fig. 2: Geometrical representation in η,φ space of an elec-tromagnetic trigger tower in the current system, where thetransverse energy in all four layers are summed (left) and ofthe super-cells for the Phase-I upgrade, where the transverseenergy in each layer is retained in addition to the finergranularity in the front and middle layers (right). Each bigsquare here represents an area of size ∆η×∆φ = 0.1×0.1.[3]

arX

iv:1

806.

0804

6v2

[ph

ysic

s.in

s-de

t] 2

2 Ju

n 20

18

As the LHC luminosity increases above the LHC designvalue, the improved calorimeter trigger electronics will allowATLAS to deploy more sophisticated algorithms, enhancingthe ability to measure the properties of the newly discoveredHiggs boson and the potential for discovering physics beyondthe standard model.

ATLAS

LiquidA

rgonC

alorimeterP

hase-IUpgrade

TechnicalDesign

Report

Decem

ber2,2013

SCA

MUX/SerializerOptical Links

Front-End Board

Preampl.

NewLayer Sum

Boards[LSB]

ADC

SCA

SCA

SCA

TimingTriggerControl

RCxSCA

Controller

Linear Mixer

Shaper

Baseplane

ROD

Optical ReceiverDeserializer Channel

De-multiplexerINPUT FPGA

Timing Trigger

Control Rx

PedSub

PedSub

PedSub

PedSub

E,t N-tap FIR

E,t N-tap FIR

E,t N-tap FIR

E,t N-tap FIR

DSP

DAQOutputFPGA

Controller Board Timing Trigger Control DistributionFixed Latency (~3.0us max)

80-100m fibers

TTC Partition Master

ADC

ADC

Optical Links

ADC

MUX/Serializer(FPGA)

ADC

LAr Digital Processing System (LDPS)

Optical ReceiverDeserializer

Timing Trigger

Control Rx

FPGA

SDRAM

Feature Extractor

[FEX]

PedSub

PedSub

PedSub

PedSub

E,t N-tap FIR

E,t N-tap FIR

E,t N-tap FIR

E,t N-tap FIR

S(t)

Receiver

480Gbps/module1.92 Tbps/board

~250 Gbps/board

Level-1 Calorimeter Trigger System

Current L1Calo

Processors

LAr Trigger Digitizer Board (LTDB)

Tower Builder Board [TBB]

iS(t- i)

Trigger Tower Sum and Drivers

PZ+Dly

Crate Monitoring

OTxCLK & Cfg.

CLK FanoutORx

Figure 4. Schematic block diagram of the Phase-I upgrade LAr trigger readout architecture. The new components are indicated by the red outlines and arrows.

Chapter1:

Overview

oftheP

hase-ILArupgrade

project5

Fig. 3: Schematic block diagram of the upgraded LAr triggerreadout architecture: with new components indicated by thered outlines and arrows.

As shown in the architecture of the Figure 3, LSBs need tobe upgraded to output the super cell signals. The new LArTrigger Digitizer Boards (LTDB) will process and digitize thesuper cell signals, and send the processed data to the back-end electronics LAr Digital Processing System (LDPS), wheredata are then transmitted to the trigger processors. Each LTDBwill be able to process up to 320 super cell signals, which willbe digitized by 80 of 12-bit quad-channel NEVIS ADCs[4] onthe LTDB. Twenty serializer ASICs LOCx2 will receive theADC data, and output 40 of 5.12 Gb/s data via fiber opticallinks to the LDPS[5]. The LDPS will convert the samplesto calibrated energies in real-time and interface to the FEXprocessors. With a total of 124 LTDB boards in the system,the total rate to the back end electronics is approximately 25Tbps. The LTDB will also output 64 summed signals to thetower builder board, each of the signal is sum of 4 input inthe same middle or front layer.

The control and monitoring of LTDB is realized via 5 GBTlinks connected with the Front-End LInks eXchange (FELIX)in ATLAS TDAQ system[6][7]. FELIX will distribute the TTC(Time, Trigger and Control) clock and BCR (Bunch CounterReset) signal via down-links to the LTDB. Besides TTCinformation, FELIX will also control GBTx and all ASICsvia the SCA (Slow Control Adapter) ASIC on the LTDB.

II. DESIGN AND TEST

A. Design of the LTDB

The LTDB design have been split into three stages. Inthe demonstrator stage, forty 8-channel TI ADC ADS5272digitize the 320 super cells. Four Kintex-7 FPGA are used topacking the data and send it to the back-end via 40 of 4.8Gb/s links. Radiation tolerance of COTS ADCs and powerconverters are researched[8][9]. For the pre-prototype, 80 ofthe NEVIS ADCs are used, 10 of Xilinx Artix-7 FPGA areused for data packing and transmission. From the prototypestage, all of the ADC, serilizer, optical-electric converters arecustom radiation-hard ASICs. Diagram of the LTDB prototypeis shown in the Figure 4.

Fig. 4: Diagram of the LTDB prototype

The prototype board is shown in Figure 5. Analog sectionis at the bottom half of board. Digital part is at the top side.

Fig. 5: LTDB prototype

The slow control between FELIX and LTDB are realizedvia 5 GBT links. The GBT link is connected to GBTx viathe MTRx module. GBTx supports to output two kinds ofrecovered clock: DCLK with better quality and CLKDES withprogrammable phase in step of 48.8 ps. For 40MHz, the jitter

in a frequency range from 100 Hz to 5MHz is about 4 psfor DCLK. For CLKDES, it is about 10 ps. The high qualityDCLK is used as the ADC input clock. On the prototype board,LOCx2 also uses DCLK. On the pre-production board, theCLKDES[10] is used to support the phase calibration requiredby the LOCx2.

B. Test Setup and Results

To test the LTDB boards, as shown in the test setup ofFigure 6, the PCIe card BNL-711 is developed. This 16-laneGen 3 PCIe card can interface the ATLAS TTC system, anddecode the TTC clock and TTC information like BCR. Thereare 48 bi-directional optical links which can run up to 14Gb/s. The two DDR4 modules can support 32 GB buffer witha speed of up to about 270 Gbps.

GBTxx5

BNL-711v1.5

LTDBPrototype

flx, cmem, rcc driverflxcard API

flx tools (fdaq, etc...)LTDB Test GUI

pepoBNL IC/EC API

GBT-SCAx5

ADCx80

Power MTRX x5

AnalogPart

ZC706DAC3484

InjectionBoards

4

ScopeSumming

MTX x20

GBT-FPGA

x6

LOCx2Decoder

x40

DDR4SODIMM

x2

Wupper PCIe Engine

PEX8732

LOCx2x20

Temperature

TTCSystem

TTC Decoder & ClockManager

Fig. 6: Test setup for LTDB evaluation

In the test setup, the firmware has 6 bidirectional GBTlinks and 40 GTH receiver for the data links from the LTDB.The low-level software tools from the FELIX project areused to read and write registers in the Kintex UltrascaleFPGA on BNL-711. An optimized low latency GBT-FPGAcore is developed to support multi-channel GBT links withLTDB[11]. The IC/EC API is developed to control the GBTxand communicate with GBT-SCA chips via the GBT links[12].With this API, all ASICs on the LTDB can be configuredand calibrated. The high level software is designed to monitorvoltage, current and temperature of the LTDB. Beside thefive GBT links for LTDB, one extra link is connected to theXilinx ZC706 evaluation board. This link is used control theDAC3484 evaluation card, which can generate test super cellwaveform for LTDB. The Xilinx all digital VCXO PICXO[13]is implemented in ZC706 to synchronize the GBT link toBNL-711 with system clock from BNL-711. The BNL-711PCIe card are shown in Figure 7.

Fig. 7: BNL-711 PCIe card for the LTDB test

With this test setup, all functions of the LTDB pre-production are verified. With the successful testing, two LTDBpre-production boards are installed on the detector in early2018. The pedestal and noise distribution of all channels onone board are shown in the Figure 8 and 9.

850 900 950 1000 1050Baseline / (ADC code)

0.000

0.002

0.004

0.006

0.008

0.010

0.012

µ= 959.49

σ= 33.63

Fig. 8: Pedestal distribution of the 320 channels

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9Std noise / (ADC code)

0

1

2

3

4

5

6

7

8

µ= 0.53

σ= 0.09

Fig. 9: Std noise distribution of the 320 channels

III. SUMMARY AND OUTLOOK

The new LAr calorimeter trigger readout system is beingdesigned for the Phase-I upgrade. LTDB is the kernel partin the front-end. Two LTDB pre-production boards have beeninstalled on the detector in 2018 successfully. The preliminarytest results show that the total noise level of the crate with

LTDB installed is at the same level of other crates. Thepedestal and noise level of super cells are same with test atlab. The data taking is ongoing for the LHC Run 2 with theBNL-711 PCIe card. Full integration with LDPS, FELIX andlevel 1 calorimeter trigger system is scheduled in summer of2018.

REFERENCES

[1] ATLAS Collaboration, The ATLAS Experiment at the CERN LargeHadron Collider, 2008 JINST 3 S08003

[2] ATLAS Collaboration, ATLAS Liquid Argon Calorimeter Technical De-sign Report, CERN-LHCC-96-041, https://cds.cern.ch/record/331061

[3] ATLAS Collaboration, ATLAS Liquid Argon Calorimeter Phase-I Upgrade Technical Design Report, CERN-LHCC-2013-017,https://cds.cern.ch/record/1602230

[4] J. Kuppambatti et al., A radiation-hard dual channel 4-bit pipeline for a12-bit 40 MS/s ADC 188 prototype with extended dynamic range for theATLAS Liquid Argon Calorimeter readout 189 electronics upgrade at theCERN LHC, 2013 JINST 8 P09008

[5] L. Xiao et al., LOCx2, a low-latency, low-overhead, 2 5.12-Gbps trans-mitter ASIC for the ATLAS Liquid Argon Calorimeter trigger upgrade,2016 JINST 11 C02013

[6] J. Anderson et al., FELIX: a High-Throughput Network Approach forInterfacing to Front End Electronics for ATLAS Upgrades, Journal ofPhysics: Conference Series, Volume 664, Online computing, 082050

[7] J. Anderson et al., FELIX: a PCIe based high-throughput approachfor interfacing front-end and trigger electronics in the ATLAS Upgradeframework, 2016 JINST 11 C12023

[8] K. Chen et al., Evaluation of commercial ADC radiation tolerance foraccelerator experiments, 2015 JINST 10 P08009

[9] H. Liu et al., Development of an ADC Radiation Tolerance Characteriza-tion System for the Upgrade of the ATLAS LAr Calorimeter, 2017 ChinesePhysics C 41 026101

[10] CERN GBT Project: GBTX Manual, https://espace.cern.ch/GBT-Project/GBTX/Manuals/gbtxManual.pdf

[11] K. Chen et al., Optimization on fixed low latency implementation of theGBT core in FPGA, 2017 JINST 12 P07011

[12] K. Chen. HDLC-ICEC: the first release, 2016 Zenodo,http://doi.org/10.5281/zenodo.595041

[13] Xilinx. All Digital VCXO Replacement for Gi-gabit Transceiver Applications (7 Series/Zynq-7000)https://www.xilinx.com/support/documentation/application notes/xapp589-VCXO.pdf