R&D for Readout, Trigger & DAQ · 3/2/2018 · technologies to efficiently provide integrated solutions for future experiments Detectors Front-End Readout LAr Front-End Trigger Liquid

Hucheng Chen 2018/03/02 - BNL-Yale Meeting

R&D for Readout, Trigger & DAQ

Hucheng Chen

Hucheng Chen 2018/03/02 - BNL-Yale Meeting 2

R&D Goal: continuously develop expertise in select detectors and electronics technologies to efficiently provide integrated solutions for future experiments

Front-End ReadoutDetectors

LAr Front-End Trigger Digitizer Board (LTDB)

Liquid Argon Electromagnetic Calorimeter (LAr)

Front-End Link Exchange (FELIX)Global L1 Calorimeter

Trigger (gFEX)

Anode Plane Array (APA) for ProtoDUNE-SP Time Projection

Chamber (TPC)

Front-End Link Exchange (FELIX)at ProtoDUNE-SP

Low Gain Avalanche Detectors (LGAD)

FELIX & CaRIBOu readout boards in a

CERN test beam

Control & Readout Board (CaRIBOu)

Signal Proc.

Sensor Front-End Board

Cold Front-End for ProtoDUNE -SP TPC


Front-End Signal Processing

● BNL focuses on overall system optimization for integrated detector readout

● LAr Trigger Digitizer Front-End Board○ ATLAS Phase-I LAr Calorimeter Front-End (for 2021)

3LTDBLAr Calorimeter FELIXgFEX

FELIX

LTDB

○ combined analog & digital board■ analog and digital parts designed

by different institutes⇒ Gerber files merged together

■ BNL leads architectural design■ collaboration with Saclay & Milan

○ 320 calorimeter signals (supercells)■ 80 custom 12-bit ADCs @ 40 MSPS digitization■ 20 custom MTx devices for data processing■ 5 GBTx/GBT-SCA for clock distribution, control & monitoring■ 40 optical links @ 5 Gb/s ⇒ 200 Gb/s output

○ pre-production module installed in ATLAS last month


● Preamplifier & Shaper ASIC (HLC ASIC)○ ATLAS HL-LHC Upgrade LAr Calorimeter Front-End (for 2026)

○ fully differential low-noise amplifier■ simultaneous readout of two gain settings■ built-in driver to interface COTS or ASIC ADCs for testing■ built-in pulse generator & mask bit for individual channels

to calibrate electronic response and perform crosstalk studies

○ 65 nm CMOS process (TSMC)

○ engineered by BNL Instrumentation Division■ development continued

from VMM ASIC for ATLAS Phase-I UpgradeMuon New Small WheelDetector Front-End

4

Front-End Signal Processing

LTDBLAr Calorimeter FELIXgFEX

ZC706

FETB

test board

HLC1


High-Throughput Trigger

● BNL is designing electronics that maximizes throughput while also runningfast trigger algorithms:○ FPGA with many transceivers satisfies high-bandwidth, fixed-latency & processing requirements

● Global Feature Extractor Module (gFEX)○ ATLAS Phase-I Level-1 Calorimeter Trigger Module (for 2021)

■ single board system receives data from full calorimeter■ hardware designed by BNL

○ ATCA blade:■ 30 layer Megtron-6 PCB■ 3 VU9P FPGA and 1 ZU19 MPSoC■ 128 Gb DDR4 RAM■ 420 optical fibers in 35 miniPODs

○ capacity:■ input: 3.9 Tb/s on 312 fibers■ output: 1.4 Tb/s on 108 fibers■ inter-FPGA: 1.6 Tb/s (high-speed & low-speed)

○ pre-production module to be installed in ATLAS in March 2018


25.6 Gb/s in PCB

Collaboration between BNL and Chicago, Indiana, Lund, Oregon, Pittsburgh, Stockholm


High-Throughput Trigger

● Global Common Module○ ATLAS HL-LHC Trigger/DAQ Upgrade (for 2026)

○ Global Trigger concentrates data for a full event from detector & trigger systems onto a single processor for analysis■ critical part of the TDAQ Upgrade■ up to ~100 Tb/s into Global Trigger■ exploits data aggregation and

serial-to-time multiplexing■ processes data with low latency (<4 s)

○ Global Common Module (GCM) is centerpiece of the Global Trigger■ single electronics module — based on gFEX —

used for all elements■ differences in functionality implemented via firmware



High-Bandwidth Readout

● BNL developing electronics for high-bandwidth detector readout○ factorize front-end electronics from data handling with compact,

high-density, scalable, low maintenance, easily upgradeable, commodity-based solution

○ reduces amount of custom hardware in favor of scalable detector-independent COTS hardware & software

○ paradigm shift adopted by ALICE, ATLAS, LHCb,ProtoDUNE-SP, DUNE and sPHENIX

7

custom commodityFELIX-711 used byProtoDUNE-SP

● Front End LInk eXchange (FELIX)○ baseline for ATLAS Phase-I and HL-LHC Upgrades○ ProtoDUNE-SP for readout of one APA○ expected baseline for DUNE readout○ sPHENIX for TPC and MVTX readout○ NSLS-II with Vertically Integrated Photon Imaging Chip (VIPIC)○ proposed for Belle II, JLab SoLID, test beams, eRHIC, ANL Light Source

TPC

FELIXCold Front-End


High-Bandwidth Readout● Front End LInk eXchange (FELIX)

○ developed for ATLAS Phase-I Trigger/DAQ Upgrade (for 2021)■ BNL designed hardware & co-developed firmware

○ generic PCIe card with Kintex Ultrascale FPGA ■ 48-channels Tx & Rx links in 8 miniPODs■ PCIe Gen3 x16 lanes interface to host■ supports versatile line rates & timing systems

● TTC; TTC-PON; White Rabbit…

○ capacity:■ 460 Gb/s input/output via optical fiber■ 128 Gb/s to host

○ planned upgrades for ATLAS HL-LHC:■ Ultrascale+ FPGA■ PCIe Gen4 x16 lanes

8

Collaboration between BNL and ANL, CERN, Irvine, NIKHEF, UCL, Weizmann [with FNAL (art_daq)]



● Modular readout system for CMOS sensor R&D (CaRIBOu)○ open architecture with ZYNQ SoC to simplify firmware & software○ easily adapted to various sensors under development○ carefully defined interface to minimize design revisions○ used in several test beams with FELIX readout at CERN

■ trigger rate reaches 60+ kHz — much faster than previously available readout system (4 kHz)

○ outgrowth of FELIX & LDRD efforts at BNL

9

Control and Readout

LGAD CaRIBOuSensor FE

CaRIBOu& FELIX

Control & ReadoutBoard (CaRIBOu)

Sensor Front-End BoardMounted Sensor & Readout ChipTelescope at CERN Test Beam with CaRIBOu & FELIX

Hucheng Chen 2018/03/02 - BNL-Yale MeetingHucheng Chen 2018/03/02 - BNL-Yale Meeting

ATLAS HL-LHC ITk Demonstrator Readout with FELIX● CaRIBOu readout with FELIX adapted for

ATLAS HL-LHC ITk Pixel Demonstrator○ Important for final development of pixel readout chain

with FELIX ○ Setup for pixel demonstrator readout development○ Successful integration test of IBL stave0b + FELIX in

Wuppertal in November 2017

● BNL is a major contributor to the ATLAS ITk Strip detector○ Expertise in both ITk strip and DAQ systems

● An ITk Strip Demonstrator is being built at BNL with FELIX readout○ Successful integration test of the Strip Hybrid module

with FELIX readout○ Benefit from previous FELIX based CaRIBOu and pixel

demonstrator readout development○ Aim to expand FELIX readout for ITk Strip module/stave

production test stand



sPHENIX Detector Readout with FELIX● sPHENIX TPC readout plans to use FELIX

as DAM (Data Aggregation Module) in DAQ system○ TPC test stand with FELIX readout has been built at

BNL○ TPC module with FEE prototype is being read out by

FELIX○ 24 FELIX modules will be needed to instrument the

full TPC for sPHENIX

● A MAPS based Vertex Detector (MVTX) has been proposed for sPHENIX○ FELIX will serve as BEE (Back End Electronics) for

MVTX readout○ MVTX test stand with FELIX readout has been built at

LANL○ FELIX has been used in the readout of MVTX

prototype at Fermilab test beam facility○ 8 FELIX modules will be needed to instrument the full

MVTX for sPHENIX

11

TPC module

FELIX server for TPC

FELIX server for MVTX @ Fermilab


CaRIBOu& FELIX


● BNL pioneered LAr-based detector technology in 1974○ expertise in detector physics & engineering essential to many

experiments, ○ Close collaboration between Physics & Instrumentation○ unique experience in cryogenic electronics and low-noise

micro-electronics

● R&D → Experiments → R&D○ designs developed well ahead of project funding in many

cases○ basic R&D funding enables us to take leadership roles in

these projects○ a strong cold electronics team is built up as a core BNL

competence

● Strong connections with University & Laboratory partners○ MicroBooNE: Columbia, FNAL, MIT, SLAC, Syracuse, Yale ○ SBND: FNAL, Chicago, Columbia, LANL, Syracuse, Yale, Bern,

CERN, Liverpool, Manchester, Sheffield○ DUNE: Chicago, CSU, Duke, FNAL, LLNL, Penn, Princeton,

SLAC, SMU, Stony Brook, Syracuse, Wisconsin, Yale, CERN, Bern, ...

12

HELIOS

DZero

NA48/62

ATLAS

BooNE

SBND

proto-DUNE

DUNE

Nob

le L

iqui

d R

&D

at

BN

LR806

1974

1980

1990

2000

2010

2020

Long History of Liquid Argon R&D at BNL

JFET FE

JFET FE

CMOS FE

CMOS FE+ADC+FPGA

CMOS FE+ADC+FPGA

CMOS FE+ADC+COLDATA

Advancement of Cold Electronics

TPC

FELIXCold Front-End


● Front-End ASIC: ○ ~5.5 mW/channel (input MOSFET 3.9 mW)○ ~15,000 MOSFETs○ 16 channels, programmable○ adjustable gain & filter time constant○ build in pulse generator with 6-bit DAC

● ADC ASIC○ 16 channels, programmable○ sample/hold○ 12-bit ADC at 2MS/s sampling rate○ current-mode domino architecture○ FIFO 192s bit wide x 32 bits deep○ multiplexer 8:1 or 16:1○ serializer 12:1○ adjustable offsets○ ~4.5 mW/ch.○ ~300,000 MOSFETs

● Full cold readout chain with FE ASIC, ADC ASIC and cold FPGA is being used to equip the ProtoDUNE-SP LAr TPC

13

● R&D of CMOS cold electronics started in 2008○ designed for 77K-300K operation○ designed for long lifetime○ tech. CMOS 180 nm, 1.8 V, 6M, MIM,

SBRES● Cold FPGA to format digitized signal and

drive data out of cryostat through serial links

Front End ASIC for Cold Electronics at BNL

TPC

FELIXCold Front-End


Single Phase LAr TPC Development towards DUNE

BNL is leading TPC readout electronics SYSTEM design for MicroBooNE, SBND and ProtoDUNE-SP

14TPC

FELIXCold Front-End


Integral System Design Concept

Cold electronics module and its attachment to the APA frame

15

A necessary (but not sufficient!) condition to achieve a good system performance, the integral design concept of APA + CE + Feed-through, plus Warm Interface Electronics with local diagnostics and strict isolation and grounding rules will have to be followed

ProtoDUNE-SP

TPC

FELIXCold Front-End


● BNL designed front end readout electronics system for MicroBooNE experiment● Analog front end ASIC designed in 180nm is running in LAr (~89 K) to achieve optimum signal

to noise ratio● MicroBooNE is the first experiment instrumented with cold CMOS ASICs

16

MicroBooNE Front End Electronics

TPC

FELIXCold Front-End


MicroBooNE Front End Electronics

17

H. Cold Mother Board

V. Cold Mother Board

ASIC Configuration Board

Intermediate Amplifier

Receiver ADC Board

Service Board

Signal Feed-through Assembly

TPC

FELIXCold Front-End


● Total: 11,264 channels● 704 FE ASICs, 11,264 ADC channels, 88 Cold

FPGAs● 88 Front End Mother Board assemblies● 4 sets of cold cable bundles, 4 sets of signal

feed-throughs● ~28 warm interface boards● Lifetime study of COTS ADC is being carried out

at BNL○ Sample has been stress tested for > 700 hours with 5.25V in

LN2○ Preliminary lifetime of COTS ADC extrapolated from past

study would be more than 108 years

18

TPC Readout Electronics

Warm Interface Electronics

Cold Electronics

SBND TPC Electronics

Sigma of DNL with 5.25V @ 2Msps

TPC

FELIXCold Front-End


SBND Front End Electronics Prototypes

19

Signal Feed-through & WIEC Assembly

FEMB Flange Board WIB

PTB

MBB

Cold Cable

TPC

FELIXCold Front-End


● 15360 channels● 960 FE ASICs/960 ADC ASICs/120 Cold FPGAs● 120 Front End Mother Board assemblies● 6 sets of cold cable bundles, 4 sets of signal

feed-throughs● 6 warm interface electronics crates● ~36 warm interface boards● Share many common development of front end

readout electronics with SBND

20

FEMB+CE box assemblies have been installed on three APA @ CERN as of today

ProtoDUNE-SP TPC Front-End Readout Electronics

TPC

FELIXCold Front-End


ProtoDUNE-SP Front End Electronics

21Cold Side Warm Side

7m Cold Cables

20 CE boxes on APA

FEMB (inside CE box)

Signal Feed-through Assembly

TPC

FELIXCold Front-End


● BNL focuses on overall system optimization for integrated detector readout○ inspired by experimental requirements and our physics interests

■ common considerations: flexible, modular, scalable, high-density, high-throughput

○ front-end ASIC design optimized for detector technologies

○ front-end board design tailored for difficult experimental environments■ e.g., cryogenic temperatures, high-radiation environments

○ trigger/DAQ designs that exploit COTS solutions■ eg., high-speed optical links, DSP in FPGA firmware, SoC (ZYNQ+)

Summary

22



CaRIBOu& FELIX

TPC

FELIXCold Front-End

Hucheng Chen 2018/03/02 - BNL-Yale Meeting 23

Backup Slides


Silicon Sensors● only U.S. Laboratory pursuing both

HV-CMOS Monolithic Active Pixel Sensors (MAPS) and Low Gain Avalanche Detectors (LGAD)

● depleted CCDs & single-photon detectors● Energy, Intensity, & Cosmic Frontier

experiments and Photon Sciences, Medical Imaging

● collaboration between Physics, Instrumentation, and Photon Sciences with ANL, Bern, Cambridge, CERN, CNM, Geneva, IFAE Barcelona, NYU, Pennsylvania, RAL, UC Santa Cruz, Stony Brook, Toronto, Wellesley, Yale

● supported by BNL LDRDs and DOE Early Career Award

24

The Generic Detector R&D research program relies on the unique capabilities and synergies available at BNL

Microelectronics● world-class development ASIC

development● low-power low-noise front ends● cold electronics enabling large

cryogenic detectors● high-functionality front-end ASICs● radiation-tolerant electronics● Energy & Intensity Frontier

experiments● collaboration between

Instrumentation & Physics with Arizona, Brigham Young, CEA Saclay, FNAL, LBNL, IFIN-HH Romania, IN2P3-LAL, INFN Milano, Michigan, NRL/NASA, Pennsylvania, Sao Paulo, SBU, SMU, Toronto, Washington Univ.

● supported by BNL LDRDs Data Acquisition & Trigger● readout for noble liquid environments● high-throughput data acquisition● high-bandwidth trigger processors● Energy & Intensity Frontier, Nuclear

Physics, Photon, Light Sources● collaboration between Physics &

Instrumentation and ANL, Bern, CERN, Chicago, Cincinnati, Columbia, FNAL, INFN Bologna, Indiana, UC Irvine, LANL, Lund, MSU, Oregon, NIKHEF, Pennsylvania, PNNL, Pittsburgh, SMU, Stockholm, Syracuse, UT Arlington, Weizmann, Wuppertal, Yale

Liquid Argon Detectors● BNL pioneered LAr-based detector

technology● precision sampling calorimeters● time-projection chambers● LAr properties● Energy & Intensity Frontier experiments● close collaboration between

Instrumentation & Physics with Bern, CERN, Chicago, Columbia, CSU, Duke, FNAL, LANL, LLNL, Liverpool, Manchester, MIT, Pennsylvania, Princeton, SLAC, Sheffield, SBU, SMU, Syracuse, Wisconsin, Yale

● supported by BNL LDRDs andtwo DOE Early Career Awards

Water Based Liquid Scintillator● unique facility developed by BNL● LZ, PROSPECT, 0νββ decay (SNO+),

Medical Imaging, Nonproliferation● collaboration between Chemistry &

Physics with Australia National, UC Berkeley, UC Davis, LBNL, Oxford, Penn, SBU, Tsinghua, Yale

● supported by BNL LDRDs


Microelectronics at BNL

● ASIC development successful because of concentration of resources & expertise and close collaboration with experimental teams○ advances in radiation detectors tightly

coupled to advances in front-end ASICs○ front-end ASICs rapidly becoming very

high functionality systems-on-chip (SOC)○ system integration is crucial!

25

ASIC for ATLAS Upgrade

Cryogenic Analog Low-Noise Front-End ASIC

adopted by MicroBooNE, Argontube, CAPTAIN, LArIAT, SBND, 50 L

ICARUS,DUNE 35 ton, proto-DUNE, DUNE

Cryogenic Front-End(8 FE ASICs, 8 ADC ASICs, 1 FPGA)


Radiation-Tolerant Electronics

26

● Extensive studies on ADCs & FPGAs○ performance, survivability, single- and

double-event upsets○ commissioned 14 MeV neutron source @

BNL for initial component screening○ further tests at other facilities

● Critical need for HEP experiments○ many industrial components not extensively tested by manufacturers○ space- and military-qualified components are expensive○ significant interest from other fields: Silicon Engineering,

Nuclear Physics, Photon Sciences, Medical

● Systematically explore impact of radiation○ close collaboration with university and industry partners:

BYU (M.J. Wirthlin), MSU, SMU, CERN, INFN MilanoAltera, Analog, Hittite, Linear, Microsemi, TI

radiation testing of ABC130 ASIC

at BNL-SSIF

Documents

R&D for Readout, Trigger & DAQ · 3/2/2018 · technologies to efficiently provide integrated solutions for future experiments Detectors Front-End Readout LAr Front-End Trigger Liquid