Belle Trigger/DAQ Workshop 2005- Workshop Summary -

R.Itoh, KEK

Shinshu University, 2/17-18/2005

28 participants

Shinshu Univ. Matsumoto Catsle and Mountains

Workshop Goals

1. Confirm the feasibility and compatibility of the near-term upgrade plans between Trigger/DAQ and detector groups.

* We need prompt upgrade of DAQ to cope with the luminosity increase in coming years. * The upgrade should be as much as compatible with the upgrade for Super B.

2. Discuss readiness of our design for Super KEKB with L=5x1035 cm-2sec-1.

* Previous design was for L=1x1035 cm-2sec-1 with rather pessimistic assumption on trigger rate. * Update trigger rate estimation using the latest data and verify our design.

CDS Agenda was used to manage program and talk slides!

You can reach the page from http://belle.kek.jp/workshops/TDAQ2005

Current Status

DAQ dead time during data taking (2/19-20/2005)

w/o injection

w/ injection

Average dead time during data taking = 7.2 % @ 500Hz L1 rate

= ~4% (readout) + ~2% (Inj. VETO) + ~1% (CDC current limit.)

Dead Time Fraction

~3.5% deadtimeadded due toinjection VETO

Estimation of dead time with increased luminosity

Possible DAQ condition until SuperB upgrade

- L is ~1.5 x 1034 in 2005 Ave. trigger rate ~ 500 Hz => intrinsic DAQ deadtime ~ 3-4% can live with current DAQ- L will increase up to 2~5 x 1034 in 2006-20xx (with crab cavity installed) Ave. trigger rate ~ 1-2kHz => intrinsic DAQ deadtime >20% needs effort to reduce deadtime and to increase processing power for event builder/L3 farm- L > 1035 in 20xx (xx can be 08~11?) by SuperB upgrade Ave. trigger rate >10kHz (max. 30kHz) => pipelined DAQ is really necessary >10 x CPU power for EB/L3 farm

“ Clear and Present Danger” by Tom Clancy

Trigger rate (@L=5x10Trigger rate (@L=5x103535))● According to present knowledge,

– Luminosity term is quite large.• 200Hz x Luminosity ( unit:1034cm-2sec-1)• 10kHz at L = 5x1035cm-2sec-1

– To reduce it, better trigger system is necessary.– To keep tau events and to suppress two-photon events

– HER beam dominantly affects the background trigger rate.● Trigger rate P x IHER I2

HER ● 200Hz at IHER =1.27A 0.2x(4.1/1.27)2 = 2kHz● One worry : Lower y* 6mm 3mm● Good point : More bumps at Tsukuba straight section to

reduce the vacuum pressure.

Uno

Iwasaki

Trigger Rate Readout deadtime Total efficiency A) 500Hz 4 % ~ 91 %

B) 1KHz ~20 % ~ 75 %

C) 2KHz >50 % <45 %

D) 10 KHz ~100 % ~ 0 %

SuperKEKB

Currentcondition

* We need to manage B) and C) even before SuperKEKB upgrade without interrupting scheduled accelerator running.

1. Upgrade of readout system to pipeline-based system

Needs some upgrade!

Expected performance without upgrade of frontend readout

* Another possibility is further sub-division of slow detector readout (like ECL), but LeCroy does not support FASTBUS TDCs any longer.....

2. Upgrade of Event Builder/RFARM also.* To remove 650Hz barrier

* Replace existing FASTBUS TDC system with COPPER based pipelined TDC system(no dead time) detector by detector during scheduled accelerator shutdown time (Summer, Winter). - Start from the detector with the longest readout deadtime (CDC) next FY. - Repeat replacement year-by-year and move to fully pipelined readout system in 3 years. - Further upgrade (for SuperB) are managed by replacing FINESSE + increasing no. of modules.

* Modularize backend DAQ (event builder+reconstruction farm) and add new units whenever more processing power is required.

* Keep intrinsic dead time ~ 5% even with increased luminosity* Upgrade is consistent with further SuperB detector upgrade* Upgrade scenario is as much as independent of SuperB budget profile* Do not interrupt accelerator running by the upgrade

“Smooth” adiabatic upgrade scenario to SuperB DAQNo distinction between “near-term” and “SuperB” upgrades

Common Readout Platform: COPPER

● O(100) of the COPPER boards are used by each sub-detector system.

- Developed in cooperation with KEK electroncs group

PMCProcessor

Trigger

GenericPMC slot

FINESSE

FINESSE

FINESSE

FINESSE

On-board Ether

FINESSEmodules

Form factor = VME 9U

COPPER module by KEK.- Common Readout Module to handle pipelined readout- Digitizers are implemented as daughter cards (FINESSE)- Works at >30KHz!

Higuchi

Ready for Mass Production

Near-term Upgrade of DigitizersReplace FASTBUS TDC system with COPPER based TDC system COPPER : Common readout module to handle pipeline readout

Controller

TD

MG

ate Generator

FP

I

LeCroy 1877S

Event BuilderFastEther Trigger (SEQ)

VME6U

FASTBUS

Network SW

Readout PC

CO

PP

ERLeC

roy 1877S

LeCroy 1877S

CO

PP

ER

CO

PP

ER

TTRX

TTRX

TTRX

TT-SW

VME9U

X n

Event BuilderGbE

GbE

FastEther

Trigger

Near-term Upgrade of Event Builder/RFARM

Current DAQ : Readout subsystems are connected to single Event Builder through point-to-point network connection

- Modularize (Event Builder + RFARM) as a unit- Have multiple units

CDC

TOFECLKLM

VTXVTX

VTXVTX

Event BuilderRFARM

CDC

TOFECLKLM

VTXVTX

VTXVTX

Event Builder RFARM

Event Builder RFARM

Event Builder RFARM

Tra

nsfe

r N

etw

ork

Mat

rix

L=1.5x1034cm-2sec-1/unit

.....

~1000 Pipeline ROM(COPPER) (pipelined readout)

~50 Readout PCs

.....

>10 Event Building Farms

>10 L3 Farms

Transfer N

etwork

mass storage

Input: ~ 100K channels

all components areLinux-based PC's

Design for SuperKEKBCurrent DAQ

L1 trigger rate 500 Hz 20 kHz

Maximum trigger rate 650 Hz 30 kHz Event size at L1 40 kB/ev 300 kB/ev Data flow rate at L1 25 MB/s 9 GB/sec

Data flow at storage 13 MB/s 250 MB/sec

Subdetector readout 26 >1000 HLT reduction 2 12

Belle(L=1.5x1034) SuperKEKB(L=5x1035)

Estimated DAQ condition at SuperKEKBUpdated!!

Detector Electronics @ SuperKEKBQuick Summary

- SVD : CMS APV25 chip

- Pixel : MAPS(CAP3) or “striplet”(APV25)

- CDC : 2 stage upgrade 1) Pipelined TDC with Q-to-T conversion (shorter shaping time) 2) ADC with waveform sampling (10bit@>100MHz)

- ECL : Wave form sampling needed to manage pileup effect (14bit FADC@2MHz for barrel, >20MHz for pure CsI)

- TOP/RICH : Need to manage pixel photo-detector * Time stretcher, HPTDC, Analog pipeline

- KLM : Readout scheme is not so much different from Belle's regardless of choice of detection device (RPC/Sci. Tile) * "hit" info multiplexing + on-board data compression

TDC FINESSE

• 24 ch LVDS input(96 ch / COPPER)

– ~150 COPPER boards in totalfor the SuperBelle CDC.

● Time resolution 0.78 ns/bit(w/ 40MHz clock) 27 m position resolution

● Dynamic range = 17 bitequivalent to 100 spipeline depth

● Linearity = 0.49%

TDC chip: AMT-3Originally designedfor ATLAShttp://atlas.kek.jp/tdc

Higuchi

* Current prototypes : channel density is too low* Needs to develop high-density 10bit/100MHz ADC (~ 100 ch / module (VME 9U) ) -> can be used for CDC and PID

• 8 ch differential input

● Sampling clock = 65 MHz

● Dynamic range = 12 bit● Linearity = 1.2%● Equips 512 word/ch FIFO

(500 MHz FADC FINESSE of 2 ch / 8 bit has also been developed)The FINESSE with higher channel density is under design.

Flash ADC FINESSEHiguchi

19

128x Wilk ADCs

8 chan. * 256 samples

8x HS Analog out, 1x MUX out

(COPPER2)

2004/5 Hawaii FINESSE Efforts

• CuEval

• HPTDC

• TOF/TOP/F-DIRC electronics

• LAB2

Varner

20

HPTDC FINESSE

HPTDC chip Spartan-3 FPGA

COPPER Interface32-channels (8 channel HP mode)

Varner

Overview of APV25● Based on IBM 0.25µm technology ● Input stage:

– shaping time: 50nsec– ENC(e)=270+38/C (pF) or 2000 electrons at 45 pF input capacitance– Analog deconvolution is possible but will not be used in Belle SVD.

● 160 out of 192 stage analog pipeline operated at 40 MHz clock– ~4 µsec trigger latency– 30 events can be queued for readout

● 40 MHz analog multiplexing (3+) µsec for 128 channel readout)● 2 mW/channel

SVD

Tsuboyama

Readout electronics

• front end: APV25, Vienna repeater (not the final version, a new version with AC coupling and base line restore will be ready in April)

• 15m CAT5 cable (like SVD2)

• back end: Vienna readout system with programmable APV sequencer and fast ADCs (same as on CMS Pixel FED)

Schwanda

4 ADC

4 times 10 bit data 4 clocks with

adjustable phase

Altera Daughter 9 inputs 9 data proc + FIFO’s

P

1

P

2

64 (or 32 bit) bit data bus 40 MHz+4 control lines

9 lines with information for trigger proc.

Fast data transfer to PCI

P

3

VME protocol Altera

Altera daughter

with final FIFO

Data for trigger, serial

Schematics of the ADC-Pixel with the possibility of single channel processing and output with data for trigger processor (largely exists)

2 clocks phase shifted

TTC

TTC input optical connection

12 input opt

Delayed clock and control sig.distributer, VME control

CMS

P 0

Transmit crate clock, control signals and event number

Control bus

* Signal processing on FPGA for occupancy reduction Vienna group has two ideas on the algorithm

Schwanda

Nishida PID

Nishida PID

Schwartz

* Wave form sampling is required

Schwartz

Schwartz

Role of FINESSE

- Two implementations are cosidered

1. On-board digitizers - Original idea - ~ 100 channels / 4 FINESSE cards = 1 COPPER module - AMT based TDC, high-speed ADC cards are being developed

2. Interface to outside digitizer system - For complicated readout electronics unified with digitizers i.e. SVD (APV25), Pixel (CAP), ECL, etc. - Standard interface FINESSE is being considered ex. Optical S-link

Upgrade Timeline

FY2004: - Complete R&D on COPPER Almost done TDC FINESSE design and production in progressFY2005: Summer - Replacement of EFC TDCs with COPPER TDCs - 2nd unit of Event Builder + RFARM Winter (during crab cavity installation) - Replacement of CDC readoutFY2006: - New timing distribution system - Replacement of KLM and TRG readoutFY2007: - Replacement of TOF and ACC readout - 3rd unit of Event Builder+RFARM (if necessary) - SVD readout upgrade + inner SVD upgrade - Full upgrade of readout for ECL (wave form smpl.)

FY20xx: - Full upgrade with new electronics / new FINESSE - Operation with >10 units of Event Builder+RFARM Super B

Conclusions of TRIG/DAQ WS 2005

1. Adiabatic upgrade scenario to pipelined readout + modularized backend meets both requirements by near-term and SuperB upgrades. -> The scenario was approved. - Upgrade project has already started.

2. The design goal for DAQ at SuperKEKB has been updated and it was confirmed that current approach can satisfy the requirements even with the luminosity of L= 5 x 1035cm-1sec-1.

3. COPPER based readout system is ready for mass production.

4. Development of FINESSE cards for each subdetector is now going on.

Backup Slides

6x6 network matrix

Dual Xeon(3.06GHz), RedHat9

PC

PC

PC

PC

PC

PC

PC

PC

PC

PC

PC

PC

PC

PC

PC

PC

PC

PC

1000Base-T

Sender Receiver

Test Bench at KEK

Performance test of Transfer Network Matrix

K.Ito(Tokyo)

Evolution of data transfer rate was studied by adding reciever nodes up to 6.

Measured Performance

# receiver PC = 1

# receiver PC = 6

At typical data size

typical data size

incr

ease

)1(

)6~2(

receiverEventRate

receiverEventRateyScalabilit

K.Ito

- Performance of Transfer Network Matrix is proven to be scalable up to 6 output nodes.

No performance degradation by adding 2nd ,3rd ... Event Builder+RFARM units

COPPER based TDC

LeCroy 1877S Multi-Hit TDC * 96ch/module, 16hits/channel, 16bit(500ps/LSB) * equipped with LeCroy's MTD133B chip

COPPER + TDC FINESSE x 4* Based on AMT3 chip (16bit(780ps/LSB)) - Some difference from MTD133B * depth of multi-hit buffer (256 words for 24ch) * method of multi-hit recording (FIFO vs. Buffer full handling)* TDC FINESSE - 48 ch / 2 FINESSEs - The same input connectors as that of 1877S's (16 inputs * 6 ) - differential ECL input -> needs LVDS conversion

Idea: Build a TDC module compatible with 1877S as possible using COPPER + FINESSE TDC => Pipelining of digitizers becomes possible without modifying frontend electronics

Higuchi's talk

Evolution of intrinsic dead time for higher trigger rate

w/o Event Builder

Upgrade

* Dead time evolution is not linear to trigger rate.

* Prediction is obtained by a fit to 0 ~ 550 Hz region with a 2nd order polynomial.

* Prediction has a large ambiguity.