Upload
lily
View
48
Download
0
Tags:
Embed Size (px)
DESCRIPTION
DAQ Systems and Technologies for Flavor Physics. FPCP Conference Lake Placid 1 June 2009 Beat Jost/ Cern-PH. Acknowledgements. Many thanks to Hans Dijkstra Miriam Gandelman Roger Forty For many useful discussions and data used in this presentation. Introduction. - PowerPoint PPT Presentation
Citation preview
DAQ Systems and Technologies for Flavor Physics
FPCP ConferenceLake Placid1 June 2009
Beat Jost/ Cern-PH
Beat Jost, Cern
Acknowledgements
Many thanks to➢ Hans Dijkstra➢ Miriam Gandelman➢ Roger Forty
For many useful discussions and data used in this presentation...
2FPCP Conference, Lake Placid, 1 June 2009
Beat Jost, Cern
Introduction❏ Challenges in Flavor/B-physics Experiments
➢ Particle Identification (-K, e- separation)➥ RICH detectors, ToF, Transition Radiation Trackers➥ Good electromagnetic Calorimeter➥ Not this talk…
➢ Vertex resolution➥ Secondary vertex identification and determination➥ Not this talk either…
➢ Good momentum resolution➥ Mass reconstruction➥ Good tracking system➥ Also not this talk…
➢ Superior Trigger Capabilities➥ Distinguish interesting (rare) final states from “junk”
– Vertexing (secondary vertices, B-vertex identification)➥ Not only limited region of detector, but ‘entire’ detector has to be
scrutinized to identify interesting events– Need a lot of information…
➥ This talk… based on the example of LHCb3FPCP Conference, Lake Placid, 1 June 2009
Beat Jost, Cern
Introduction❏ The role of the DAQ system is mainly to support the trigger
strategy and (of course) the physics analysis needs➢ Earlier Trigger drove the DAQ system➢ Nowadays the DAQ system can help defining the trigger strategy
❏ Will use LHCb as illustration for the following…➢ Three ‘typical’ channels to illustrate the development…
➥ Bs J/ ➥ Bs
➥ Bd
4FPCP Conference, Lake Placid, 1 June 2009
Beat Jost, Cern
LHCb Detector
5FPCP Conference, Lake Placid, 1 June 2009
Vertexing
Ks Identification Trackingp-Measurement
Particle ID
CalorimetryTrigger Support
Muon IDTrigger Support
Beat Jost, Cern
Choice of Working Point
❏ Contrary to Atlas/CMS which are designed to operate at Luminosities of ~1034cm-2s-1, LHCb will operate at luminosity of ~2x1032cm-2s-1. Why??
❏ Interactions as function of luminosity
6FPCP Conference, Lake Placid, 1 June 2009
❏ Want to avoid having too many multiple interactions per bunch crossing Difficulty to distinguish
multiple primary vertices from displaced vertices…
More complex events
LHCb Trigger/DAQ Evolution
7FPCP Conference, Lake Placid, 1 June 2009
Beat Jost, Cern
LHCb Trigger/DAQ Evolution
❏ 1998: LHCb Technical Proposal❏ 2001: LHCb Online System
Technical Design Report (TDR)
❏ 2001-2003: LHCb too fat➢ Redesign, slimming
❏ 2003: LHCb Trigger TDR❏ 2005: Addendum to LHCb Online
TDR➢ 1 MHz Readout➢ System as it is implemented today
❏ 2008: LHCb Upgrade discussions➢ 40 MHz Readout➢ System to be implemented ~2015 (or
so…)8FPCP Conference, Lake Placid, 1 June 2009
Trig
ger I
neff
icie
ncie
s, Te
chno
logi
cal D
evel
opm
ent
Beat Jost, Cern
2001: Online System TDR❏ Three-level Trigger system
➢ Level-0 (fixed latency 4.0 s)➥ Hardware, unchangeable➥ high-pt Electron, Hadron, Muon➥ 40 MHz 1 MHz
➢ Level-1 (var. latency <2ms) ➥ Secondary Vertex ‘identification’
using Vertex Locator (VeLo)➥ Some ideas of using information of
Level-0 trigger to improve efficiency…– Super Level-1…
➥ 1 MHz 40 kHz➥ Separate functional entity
➢ High-Level Trigger (HLT)latency limited by CPU Power
➥ (partial) reconstruction of full detector data selecting physics channels
➥ Farm of commercial CPUs➥ 40 kHz 200 Hz
9FPCP Conference, Lake Placid, 1 June 2009
Read-out Network (RN)
RU RU RU
4 GB/s
4 GB/s
40 MB/s
Control & Monitoring
(ECS)
LAN
Read-out Units (RU)
Timing & Fast Control (TFC)
Level-0
Front-End Electronics
Level-1
VELO
L0
L1
40 MHz
1 MHz
40 kHzLevel 1Trigger
Variable latency <2 ms
DataRates
40 TB/s
1 TB/s
1 MHz
Front-End Links
SFC SFC
CPU
CPU
CPU
CPU
Sub-Farm Controllers (SFC)
Storage
Thro
ttle
Front-End Multiplexers (FEM)
Level 0Trigger
Fixed latency 4.0 s
Trigger Level 2 & 3Event Filter
Variable latencyL2 ~10 msL3 ~200 ms
TRACK ECAL HCAL MUON RICHLHCb Detector
Baseline Architecture
Beat Jost, Cern
2003: Trigger TDR
❏ Three-Level trigger system➢ Level-0:
➥ unchanged➢ Level-1:
➥ Removed as a separate architectural entity
➥ Still present as separate dataflow through a common network
➥ Add Trigger-Tracker station to Detector and Level-1 trigger to provide some momentum information to the vertex determination
➥ Allows flexibility (within limits) to add data from more detectors
➢ HLT:➥ unchanged
10FPCP Conference, Lake Placid, 1 June 2009
MultiplexingLayer
FE FE FE FE FE FE FE FE FE FE FE FE
Switch Switch
Level-1Traffic
HLTTraffic
126Links
44 kHz5.5 GB/s
323Links4 kHz
1.6 GB/s
29 Switches
32 Links
94 SFCs
Front-end Electronics
Gb Ethernet
Level-1 Traffic
Mixed Traffic
HLT Traffic
94 Links7.1 GB/s
TRM
Sorter
TFCSystem
L1-Decision
StorageSystem
Readout Network
Switch Switch Switch
SFC
Switch
CPU
CPU
CPU
SFC
Switch
CPU
CPU
CPU
SFC
Switch
CPU
CPU
CPU
SFC
Switch
CPU
CPU
CPU
SFC
Switch
CPU
CPU
CPU
CPUFarm
62 Switches
64 Links88 kHz
~1800 Nodes
Baseline Architecture
Beat Jost, Cern
Status after Trigger TDR
❏ Single Network for Level-1 and HLT traffic➢ Separate data flows
❏ Level-1 Trigger Performance➢ Bs J/
➥ 64% efficiency (89.7% L0 71.4% L1)➢ Bs
➥ 25.2% efficiency (41.8% L0 60.3% L1)➢ Bd
➥ 33.6% efficiency (53.6% L0 62.7% L1)Note:
Efficiencies are relative to offline selected events, i.e. selection after full reconstruction
11FPCP Conference, Lake Placid, 1 June 2009
Beat Jost, Cern
Limitation and Development
❏ The fact that the Level-1 trigger only had access some detector data made it susceptible especially to fake secondary vertices from multiple scattering➢ Also the introduction of magnetic field into the VeLo
region and the trigger tracker station did not completely eliminate this
➢ For a fraction of the events the addition of information from other detectors would be advantageous
❏ Delays in the LHC and the breath-taking development in network technology made it possible to think of the complete elimination of the Level-1 Trigger 1 MHz readout
12FPCP Conference, Lake Placid, 1 June 2009
Beat Jost, Cern
2005: 1 MHz Readout, Current system
13FPCP Conference, Lake Placid, 1 June 2009
❏ Three-Level trigger system➢ Level-0:
➥ unchanged➢ Level-1:
➥ Removed➢ HLT:
➥ Access to all data at Level-0 rate
➥ Split into HLT1– ‘verification’ of L0 Trigger
➥ And HLT2– Selection of specific physics
channels➥ Full flexibility of algorithm
design➥ Latency only limited by total
CPU power available➥ Output rate: ~2 kHz
Architecture
~700GbEthernet Links1 MHz, ~35 GB/s
StorageSystem
Front-end Electronics
Switch
CPU
CPU
CPU
Switch
CPU
CPU
CPU
Switch
CPU
CPU
CPU
Switch
CPU
CPU
CPU
Switch
CPU
CPU
CPU
CPUFarm
~1500 Nodes12000 CPUCores
~500GbEthernet Links1 MHz, ~35 GB/s
Readout Network
RB RB RB RB RB RB RB RB RB RB RB
DE DE DE DE DE DE DE DE DE DE DE ~90001.6 Gb Links
1 MHz, ~ 2 TB/s
Detector Electronics
Readout Boards
Beat Jost, Cern
Comparison Atlas/CMS
14FPCP Conference, Lake Placid, 1 June 2009
❏ CMS: High BW school➢ Only one HLT level @ 100 kHz➢ Total BW: 100 GB/s.➢ Can stage network/CPU:➢ 8 x 12.5 GB/s.➢ To Storage: 100 Hz
❏ ATLAS: Low BW school➢ RoI in Level-2.➢ L2 “loads" 10% of data @100 kHz➢ Total BW: 20 GB/s.➢ Can load “full-event“
at low L1-rate, i.e. low-L,B-physics signatures.
➢ To Storage ~200 Hz
Beat Jost, Cern
Where are we now…❏ Great simplification of the DAQ system❏ Latest efficiencies for ‘our’ channels
➢ Bs J/ ➥ 85% efficiency (93% L0 91% HLT1) Compare to➥ 64% efficiency (89.7% L0 71.4% L1)
➢ Bs ➥ 22.0% efficiency (44% L0 50% HLT1) Compare to ➥ 25.2% efficiency (41.8% L0 60.3% L1)
➢ Bd ➥ 50% efficiency (65.0% L0 77.0% HLT1) Compare to ➥ 33.6% efficiency (53.6% L0 62.7% L1)
❏ The increase in Level-0 efficiency stems from the fact that the Global Event Cuts (such as Pile-up Veto) do not seem necessary anymore
❏ The drop in the HLT1 efficiency for Bs is due to a change in trigger strategy (emphasis in Trigger on Signal, ToS)
❏ Bs is still very saddening➢ Some ideas are around to get efficiency up to ~30% (CPU limited)
15FPCP Conference, Lake Placid, 1 June 2009
Beat Jost, Cern
Pictures: LHCb Common Readout Board
16FPCP Conference, Lake Placid, 1 June 2009
Optical Receiver Cards
Controls InterfaceProcessing FPGAs
Output Driver 4-port GbEthernet
Beat Jost, Cern
Pictures: Full Tell1 Crate (Cabling)
Controls Ethernet
17FPCP Conference, Lake Placid, 1 June 2009
Flow Control (Throttle)
Timing and Fast Control
Quad GbEthernet Output
Beat Jost, Cern
Pictures: Readout Network (Switch)
18FPCP Conference, Lake Placid, 1 June 2009
Glossy-PrintReal Life
Beat Jost, Cern
Pictures: CPU Farm (Processors)
19FPCP Conference, Lake Placid, 1 June 2009
Beat Jost, Cern
Pictures: CPU Farm (Cooling)
20FPCP Conference, Lake Placid, 1 June 2009
Beat Jost, Cern
And... It Works!
21FPCP Conference, Lake Placid, 1 June 2009
Injection Spray
Cosmic Event
Beat Jost, Cern
Level-0 Performance❏ Level-0 Hadron
efficiency is pathetically low compared to Muon or Electron channelseliminate Level-0
40 MHz readout
22FPCP Conference, Lake Placid, 1 June 2009
❏ In order to keep the minimum bias contamination low (<<1 MHz) have to cut severely into the signal (Et cut ~3 GeV)
Beat Jost, Cern
40-MHz Readout (LHCb upgrade)
❏ Features➢ No more hardware trigger➢ Full software flexibility for event
selection❏ Requirements
➢ Detector Electronics has to implement Zero-Suppression
➥ Keep cost for links bearable➢ Need ~10-fold increase in
switching capacity➢ Need x-fold increase in CPU power
(x >> 40).➢ New Front-End Electronics
❏ Also increase Luminosity from2x1032cm-2s-110-20x1032cm-2s-1
➢ More pile-up more complexity bigger events
23FPCP Conference, Lake Placid, 1 June 2009
40-MHz Architecture
~280010GbEthernet Links40 MHz, ~1.5 TB/s
StorageSystem
Front-end Electronics
Switch
CPU
CPU
CPU
Switch
CPU
CPU
CPU
Switch
CPU
CPU
CPU
Switch
CPU
CPU
CPU
Switch
CPU
CPU
CPU
CPUFarm
~???? Nodes
~200010 GbEthernet Links
40 MHz, ~1.5 GB/s
Readout Network
RB RB RB RB RB RB RB RB RB RB RB
DE DE DE DE DE DE DE DE DE DE DE ~90003.2 Gb Links
40 MHz, ~1.5 TB/s
Detector Electronics
Readout Boards
Beat Jost, Cern
40-MHz Readout. Expected Performance
❏ Preliminary Studies➢ Bs Efficiencies
➥ ~57% @ 2x1032cm-2s-1
➥ ~50% @ 10x1032cm-2s-1
to be compared with➥ ~22% for the current System
Note: more than double the efficiency and at the same time 5-fold the Luminosity, i.e. 10-fold increase in rate!
➢ Bd Efficiency➥ ~52% @ 10x1032cm-2s-1 (Current system: ~50%)
❏ Critical Issues➢ Rad-(tolerant, hard) front-end electronics performing zero-suppression➢ Large Data rate through network
➥ 1.5 TB/s is sizeable➥ Technological development seems to make it feasible at the time-scale of ~2015
– Expect linecards of ~1Tbit/s capacity (100 10Gb ports)➢ CPU power needed…
➥ First studies show an acceptable increase of factor ~50-70 @10x1032cm-2s-1 (ok-ish)
➥ @20x1032cm-2s-1 studies are ongoing ➥ From Moore’s law we expect ~factor 10-20 (by ~2015)
– Rest by increasing farm size…– And change detector and trigger strategy
24FPCP Conference, Lake Placid, 1 June 2009
Beat Jost, Cern
Outlook on Future Accelerators❏ The future will be in trigger-free DAQ systems
➢ Either dictated by the bunch structure of the machine➥ 0.6 ns bunch spacing at CLIC➥ ‘Digital Oscilloscope’ connected to each channel
➢ Or by choice since technology allows it (LHCb upgrade) because of flexibility and (possibly) improved efficiency and simplicity
❏ Special case might be continuous high-frequency accelerators➢ E.g. SuperB: 2 ns bunch spacing continuous operation
relatively low interaction rate➥ 500 MHz collision rate➥(4S) Cross section ~5 nb➥ Luminosity 1036cm-2s-1
hadronic event-rate ~5 kHz➥ Activity trigger might be indicated
25FPCP Conference, Lake Placid, 1 June 2009
Beat Jost, Cern
Conclusion❏ LHCb DAQ system has developed significantly over time
➢ Partly driven by desire to simplify the system➢ Mainly driven by improving the trigger efficiency
➥ ~50% improvement for non-hadronic channels➥ Some hadronic channels still poor
26FPCP Conference, Lake Placid, 1 June 2009
❏ Shift from ‘hardware’ triggers to software with full access to all detector information
❏ Latest step (LHCb upgrade) is trigger-free readout at full bunch-crossing rate (40 MHz)➢ Expect ~doubling of efficiency in
difficult hadronic channels➢ In-line with possible other future
experiments (e.g CLIC)0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Trigger TDR 1 MHz 40 MHz
Trig
ger E
ffici
ency
Trigger Efficiencies
Bs à J/Y j
Bs à jj
Bd à p p
Bs
Bs J/
Bd