Maximize your calibration event rate with partial events

Ignacio Aracena (SLAC)

SLAC ATLAS forum

March 18th 2009

2

Outline

• Introduction– Motivation

– The ATLAS detector

– Calibration requirements

– The ATLAS TDAQ system

• Partial events for calibration

• Calibration with partial events– Inner detector alignment

– LAr calibration

– Tile calibration

– Test results

• Rate studies with partial events

• Open issues

• Summary

3

Motivation

Physics at the LHC:

New physics with (relatively) high pt at few Hz

vs SM physics at ~kHz

ATLAS trigger strategy for new physics:• Improve physics object selection at three trigger levels• Region of Interest (RoI) minimizes data

requests/shipping through network

Level 1 : Identifies RoI to seed L2 @ 100kHz

Level 2 : Improves RoI eta, phi, pT by using

high detector granularity, accept event rate ~3kHz

Event Filter : full event data available for further

improvement, output rate ~200Hz, i.e. 300MB/s to

permanent storage for offline analyses.

Trigger architecture driven by physics programme

How much trigger rate can we afford for calibration events during physics run?

L1 out 100kHz

L2 out 3kHz5GB/s

EF out 200Hz300MB/s

4

The ATLAS detector

EM calorimeter:

LAr calorimeter

Tracking :

Pixel, SCT, TRT

Muon spectrometer

TGC/RPC (trigger chambers)

CSC/MDT (precision chambers)

Hadronic calorimeter:

Tile calorimeter (barrel)

LAr calorimeter (endcaps)

ATLAS subdetectors

Precise and fast calibration of different subdetectors is crucial for achieving performance goals

5

Event data volume composition

• Event data in terms of Readout buffers (ROBs) requested by

TDAQ:

– LAr 762 ROBs

– Tile 64 ROBs

– Pixel 132 ROBs

– SCT 86 ROBs

– TRT 192 ROBs

– Muons ~400 ROBs

– Total ~1600ROBs

• 1ROB ~ 1kB

• Full event with data of all the ROBs is ~1.5MB (about 50%

from LAr...)How much data is needed for calibration of subdetectors?

6

Calibration requirements

• Inner detector alignment– Need 6-8 M isolated tracks in 6 hours for pixel/SCT and in 24 hours

for TRT

– Requires ~50Hz of isolated tracks

• LAr calibration– Study pulse shape of individual cells

– Requires ~5 Hz to achieve precision of <1%

• Muon spectrometer– Need to collect ~106 muons/day for MDT t0 calibration

– ~1kHz of muon tracks

• Tile calorimeter– Illuminate full calorimeter with dedicated tile calibration system, run

during empty bunch crossings

• New calibration types?

7

Calibration vs Physics

L1 out 100kHz

L2 out 3kHz5GB/s

EF out 200Hz300MB/s

Muon spectromter calibration ~1kHz ≈ L2 accept rate

Inner detector alignment ~50Hz25% of EF output rate

LAr calibration ~5Hz3% of EF output rate

Calibration rate appear to be not matching the

rate limits of TDAQ design scheme at first glance!

Can we take calibration data during physics runs?

8

The ATLAS TDAQ system

L1

L2

EF

ReadOut Systems (ROS)Readout buffers (ROBs)

Event builder (SFI)

Data logger (SFO)

3 kHz

200 Hz

3 kHzRoI

100 kHz

Tier0

200 Hz

hard

war

eso

ftw

are

Level1 : If RoI found, all detector data is kept in the

Readout buffers (ROBs) of the Readout system (ROS).

Level2 : Requests data from RoI, with full detector

granularity. If event is accepted, the event is fully built

Event builder : For L2 accepted events the full event

data from the ROS is pulled to build the full event.

Event Filter : Receives full event, which can be used for

further improvements. Sends accepted events to the data

logger (SFO)

SFO : Writes events into different data streams, sends

data to offline mass storage

Event dataflow

ATLAS TDAQ system is based on:

L1 100kHz → L2 3kHz → EF 200Hz

full event size ≈ 1.5MB, RoI size ≈ 3% of full event

9

The TDAQ system

L1

L2

EF

Event builder (SFI)

Data logger (SFO)

3 kHz

200 Hz

3 kHzRoI

100 kHz

Tier0

200 Hz

hard

war

eso

ftw

are

ATLAS TDAQ system is based on:

L1 100kHz → L2 3kHz → EF 200Hz

full event size ≈ 1.5MB, RoI size ≈ 3% of full event

Network requirements:

bandwidth = event rate * event size

Max. bandwidth is fixed by the system installed at P1

(300MB/s)

(5600MB/s)Calibration events

should use small fraction of total bandwidth

need high trigger rate

don’t need the full event!

Can increase calibration event rates for same bandwidth budget by using partial events!

ReadOut Systems (ROS)Readout buffers (ROBs)

10

Partial events for calibration

L1

L2

EF

Event builder (SFI)

Data logger (SFO)

3 kHz

200 Hz

3 kHzRoI

100 kHz

Tier0

200 Hz

hard

war

eso

ftw

are

(300MB/s)

(5GB/s)

Select calibration event at Level 2: • Fill list of detector (ROB) identifiers needed for a

specific calibration purpose.• In the event builder (SFI) use this list to pull out data

only from identifiers in this list and build partial event• Send partial event to SFO, no need to go through EF.

partial event building

Select calibration event at the Event Filer:• All events that are processed in the EF are fully built• Fill list of detector (ROB) identifiers needed for a

specific calibration purpose.• In the SFO use this list to extract only data fragments

from listed ROB identfiers• Create a partial event from the full event : event

stripping, ship and store only partial event

event stripping

For an allocated bandwidth partial events allow higher trigger rates of calibration events, i.e.partial events save bandwidth at the SFI and SFO

ReadOut Systems (ROS)Readout buffers (ROBs)

11

Calibration + physics events

Logic to recover the partial event for calibration implemented • If only overlap at L2, apply event stripping in the EFD

•Allows further save of bandwidth in the EFD• If overlap at EF, apply event stripping at the SFO

Overlapping events: events that are selected by calibration triggers and physics triggers

PT = EF processing task

EFD = Event Filter data flow program

12

Calibration requirements and partial events

• Inner detector alignment– Need 6-8 M isolated tracks in 6 hours for pixel/SCT and in 24 hours for TRT– Requires ~50Hz of isolated tracks– partial event build

• LAr calibration– Study pulse shape of individual cells– Requires ~5 Hz to achieve precision of <1%– event stripping

• Muon spectrometer– Need to collect ~106 muons/day for MDT t0 calibration– ~1kHz of muon tracks– partial event building? Muon group has developed a standalone “TDAQ”

system to extract muon tracks. Less flexible than using partial event from TDAQ

• Tile calorimeter– Illuminate full calorimeter with dedicated tile calibration system, run during

empty bunch crossings– partial event building

13

Inner Detector alignment

Requirements• Get high rate O(50Hz) over wide pT range

Chains in current Lumi1E31 menu (athena 15.0.0):• trk9i_id, prescale 40• trk16i_id, no prescale • Route to “calibration_IDTracks” stream

Fill list of ROBs at L2 use case for partial event building

(Anna Sfyrla, Carlos Escobar)

14

LAr calibration

Requirements• Sufficient rate (5Hz) given by photon triggers

into EF, no need for additional rate using

partial EB• Minimize bandwidth at the EFD/SFOChains in current Lumi1E31 menu (athena 15.0.0):

• g3_larcalib (PS 10^6)• g10_larcalib (PS 100)• g20_larcalib (PS 1)• g20i_larcalib • j10_larcalib (PS 10^6)• j80_larcalib (PS 10^6)

Fill list of ROBs only at EF selectionuse case for event stripping

(Nicolas Berger, Guillame Unal, Isabelle Wingerter-Seez)

LVL1EM RoI

LVL2photon selection

EFphoton selction

event stripping use only ROBs

inside RoI

15

Tile calibration

Tile calorimeter has dedicated calibration system:• Laser injection (timing studies)• Charge injection (pulse shape)

which illuminates the full tile calorimeter

Used during emtpy bunch crossing

Chains used in cosmic trigger menu• TileCalib_laser• TileCalib_cis

Trivial use case of partial event with a complete subdetector (Andrea Dotti, Oleg Solovyanov)

LVL1CALREQ

LVL2fill list with all

tile ROBs

partial event build

16

Tests with AtlasHLT-14.5.0

• Setup multihost partition on lxplus with AtlasHTL-14.5.0 (tdaq-02-00-00)– 2 L2 segments (L2SV, L2PUs), – 1 EBEF segments (2 SFIs, 1 SFO, 1 EF segment, 4PTs)– 1 ROS segment (8 ROSs)

• Data sample : /pcatr-srv1/data/files/aagaard/14.5.0/enhancedBias10TeV/BS/ (1200 events)– “enhanced bias” = L1 preselection EM3, MU4, TAU5, J18, FJ18,

XE25, TE250 – preload the data on ROS (ROS application running on lxplus node)

• Lumi1E31 menu– testing the ID and LAr calibration streams

• Recorded monitoring histograms after ~1hour run• Monitoring plots from this test on the following slides

17

Partial event building performance – event size

Size of built events in the SFI Event size vs streams in the SFI

Full event size ~2MB (MC data)Partial event ~50kB (~3% of full event)

overlapping events (calib + phys)need to be stripped

partial events for ID alignment

About 30% of ID alignment events overlap with physics eventsOverlap fraction can be tuned with the menu

Impact on resources? See next slide

18

Partial event building performance – rates

Contribution to event rate at the SFI Bandwidth contribution at the SFI

ID alignment event rate comparable to physics event rates

stream X event rate / total event rate stream X event build rate / total event building rate

Pure ID alignment events use very small fraction of total bandwitdh

overlapping events (calib + phys)need to be stripped

Partial event building minimizes the bandwidth fraction used for calibration events

19

Event stripping performance – event size

Event size vs streams in the SFO

(ignore calibration_LArCells, bug in menu)stripped events for LAr calibration

(full events from calibration_LArCells, bug in menu)

Full event size ~2MB (MC data)Partial event ~50kB (~3% of full event)

20

Event stripping performance – event size

Event size vs streams in the SFO

(ignore calibration_LArCells, bug in menu)stripped events for LAr calibration

(full events from calibration_LArCells, bug in menu)

Full event size ~2MB (MC data)Partial event ~50kB (~3% of full event)

ID alignment events not strippedbug, fixed in 15.0.0

21

Event stripping performance – rates

LAr calib event rate comparable to physics event rates

Contribution to event rate at the SFOstream X event rate / total event rate

Bandwidth contribution at the SFOstream X event build rate / total event building rate

LAr calib events use very small fraction of total bandwitdh

Event stripping minimizes the bandwidth fraction used for calibration events

22

Examples from cosmic data taking runs

• Previous slides showed only tests with MC data

• Tile calibration and ID alignment stream exercised during

cosmic data taking runs in following slides

23

Tile calibration – example from cosmic

Exercised during cosmic runs, example here from run 92082• Full event size ~5MB (LAr requesting 10 samples)• Tile calibration event size ~200kB, i.e. ~1% of full event• No rate monitoring plots for this run (Atlas release 14.2.23), but with given event

size uses only ~1% of bandwidth• Event successfully used for calibration

Num

ber

of

even

ts

Event Size (kB)

full event ~5MBpartial event ~200kB

LVL1CALREQ

LVL2fill list with

all tile ROBs

partial event build

24

Inner detector alignment with cosmics

Run 90943

Use chains developed for cosmic data (e.g. CosmicsAllTeTRTxK*, CosmicsAllTeSiTrack*, CosmicsAllTeIDScan*) and apply PEB on those

✘ … CosmicsAllTe chains do not have ‘well defined’ RoIs

✘ … chains that do tracking at the TRT are not good for testing since TRT does not provide η measurement.

Modify the TrigROBListWriter.cxx to loop over the tracks and get η-φ range from the tracks

Apply modified TrigROBListWriter.cxx to two chains: o CosmicsAllTeIDSCAN_AllPhysics_TrkHypo ando CosmicsAllTeSiTrack_AllPhysics_TrkHypo

Generate new cosmic menu that contains these new chains, and run with this menu on cosmic data.

The generated raw data will contain information from the ID only and for the η-φ range that is defined by the modified TrigROBListWriter.cxx

Compared raw data generated with and without

the PEB applied in the chains

(Anna Sfyrla, Carlos Escobar)

Partial events gives same results as full event data

25

HLT rate studies with partial events

• Use partial event to store only trigger decision bits from L1/L2/EF– Allows high statistics rate studies

– https://twiki.cern.ch/twiki/bin/view/Atlas/BeatenbergActions#General_Action_Items , see “T4”

• Could also be combined with monitoring information for TDAQ performance tests

• Basic TDAQ infrastructure ready. Need to configure menu “chain”, study the impact on the resources

26

Open issues

• Only one list of ROBs (subdetectors) per event. List is union

of all partial events. Results in overlapping ROBs (e.g. LAr

ROBs showing up in ID alignment events)

• Tests on MC show no overlap between LAr calibration and ID

alignment. More studies needed

27

Open issues

• Only one list of ROBs (subdetectors) per event. List is union

of all partial events. Results in overlapping ROBs (e.g. LAr

ROBs showing up in ID alignment events)

• Tests on MC show no overlap between LAr calibration and ID

alignment. More studies needed

• Bandwidth budget per sub-detector calibration?

– Need to establish rules/policy

– Needs to be discussed across sub-detector groups and TDAQ group

28

Summary

• Calibration with partial events – Using partial events for calibration of sub-detectors allows to increase

the event rate for a given bandwidth budget

– Flexible system that allows calibration data taking with all subdetectors

• Being used by LAr, ID, Tile– Partial event already tested and used at P1 during technical runs, cosmic

data taking

– Muon MDT t0 calibration? Standalone system does not offer flexibility of partial events. Next customer on the list

• Potential new use case discussed in Beatenberg– Partial event build with only trigger decision bit information for trigger

rate studies

• New calibration types?– Advantage of partial event is flexibility, can accommodate all sub-

detectors

29

Outlook

• Run test at P1– Uses real system

– Larger statistics available

– Can re-play cosmics and/or pre-load MC data

• Implement monitoring– Rate plots shown here not implemented yet in official SFI/SFO tags

– New monitoring plots?