CMS StatusTommaso Boccali (INFN Pisa / CERN)Markus Klute (MIT)

1

Outline

● General status - since last LHCC● Resource request for 2020● News from the project and AoB

2

Since last LHCC: HI endgame

● We were in HI data taking○ Concluded on Dec 3rd○ Final statistics is 4.4 B full format

Minimum Bias events + 250 M “physics triggers”

○ Final data size is 5.3 PB RAW

● Prompt processing went on @ T0 (with HLT when available - slide 5), and was done on Jan 7th

● 2nd full copy of RAW @ FNAL by Dec 28th (1 month earlier than expected due to > 2x speed of transfer);

● AODs on Disk (Vanderbilt T2) and Tape (IN2P3 T1) in sync with the production process 3

Since last LHCC: HI endgame

● All in all pretty much as expected (and tested w/IT!)● In the last days Data Taking pushed at 7.3 GB/s

○ 6 GB/s was expected to be the safety limit

● Processed data is in the hands of the analyzers since early January; better than plans which were pointing to end of Jan

4

Processing from last LHCC - HLT utilization

● We had the HLT partially available for the whole period○ Partially: some complete shutdowns due to P5 work, mostly at reduced capacity due to limited

cooling

● Bottom line: whenever technically available (“on”), we can use HLT

HLT processing HI @ T0 HLT processing offline workflowsNo availability (P5 activities)

HLT needed by DAQ (beam on)

5

MC campaigns

● MC 18 (targeted to Moriond19 analyses):

○ 8.2 B done; M19 high priorities fully processed and available

○ No showstopper for analyzers!

● PU200 production for MTD TDR:

○ Admittedly, we started late (end of Jan); >1 month of delay for sw

○ 10M Events the reduced request○ Processed @ 80% level in 1 week

Restart after the ECAL issue

8.2 B events ready for analysis

Feb 2nd

Feb 8th

6

Parking B

● As already reported in Nov, 12 B events collected, ~ 10 B of B hemispheres via a specialized trigger reaching almost 6 kHz

● Since then:○ Specific electron low Pt reconstruction

ready○ Started staging RAW from tape○ Plan is to process ALL the events + skim

O(100M) events of particular interest

● The low Pt electron reconstruction will enter default workflows in the ultra legacy reprocessing

/ParkingBPH*/Run2018*-v1/RAWTOTALS:Datasets: 23Size on Tape: 7596 TB (single copy)Events: 11878715564 Avg Size 639528 bytes / event

7

The Ultra Legacy reprocessing

● Preparation currently as planned without major delays

● We will probably use the sequence○ 2017 → 2018 → 2016

● <PU> is in line with modelling (35)● HLT should be fully available by April● IT is positive that in the same time

scale the P5 ←→ IT link should be upgraded from 4x40 to 4x100 Gbits

○ Over Xmas we touched for a few hours the limit

○ We expect at least 30% additional HLT available during UL

8

NanoAOD evolution & NERSC

● Commissioning is going on, and a number of Moriond analyses will use NanoAOD

● Eventually, we processed all 2016-2018 DT and MC to have a consistent set for RunII, even before Ultra Legacy

● Processed NANOAOD* in 2018:● MC: 45 B events● DT: 35 B events

… using < 10% of the computing resources …

we can scale much higher

● NERSC is used in production at high levels, until grant limits

● Last 90 days shown below● Xeon (20%) and KNL (80%) seamlessly

in production at the same moment● Slower cores (4x) compensated by

larger MT (4x), possible due to the excellent MT capabilities of CMSSW FW

50kCores

20kCores

NERSC # cores

9

Storage status

● The ParkingB and Minimum Bias HI streams were unexpected / reduced in the 2018 modelling when resources were asked

○ CMS was able to handle these without any resource increase (current/future), via special running modes and aggressive cleanings

● As of now:○ T0 tape nearly full, but no additional expected pressure

during LS2○ T1 Tape ok○ Disk areas in

good shape (70% “unmovable”)

T0 tape

T1 tape

10

2019 Production Plan

Nov Dec Jan2019

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan2020

Feb Mar

Prompt 10_3

10_5_0 10_6_0

MC for PbPb 10_3

HI-park RECO 10_3 B-Parking RECO

Jamboree 17 JEC SF 17

Jamboree 18 JEC, SF 18

Rereco 2017MC for 2017 physics 10_6

MC UL prep 10_5, for the 3 years, for PF Calibration.

MC Alca prep 17

data AlCa prep 2017

10_5 valid

valid

valid

Rereco 2018MC for 2018 physics 10_6

MC Alca prep 18

data AlCa prep 2018 valid

valid

Rereco 2016MC for 2016 physics 10_6

MC Alca prep 16

data AlCa prep 2016 valid

valid

Jamboree 16 JEC, SF’ 16

MTD TDR

2016 re-MiniAOD/NanoAOD

Still not started (waiting for final sw)

All needed pieced of software and condition are integrated together. Start RelVals production for validation

11

2020 Requests

● CMS confirms a request of +0% (wrt 19) in all the areas

● CMS planned activities in 2020:○ Tails of the UL○ Analyses as in 2018○ RunIII preparation, including a dress rehearsal of

the new sw components■ Rucio, DD4HEP, CRIC, …

● For 2021, CMS updated its modelling in view of recent LHC planning; with respect to 2019 the up-to-date figures are

○ CPU: +30%○ Disk: +20%○ Tape: +25%○ (all to be confirmed in Fall’19) - used to be a flat

+30% 12

News from software and AoB

● CMSSW_10_5 contains the first commits of the new CMS Heterogeneous framework

○ To be finalized in 10_6_0○ See next slide

● The development and test of Rucio, DD4Hep and CRIC continue as planned○ CRIC has substituted SiteDB in production○ Rucio generally on schedule with the plan sketched on Jul 18

■ Next one is “Million file test with Rucio & PB scale tests” - March 19; under preparation

● CB122 approved the move to Apache2 license

13

Goals and status for heterogeneous CMSSW● Identify a general pattern for all accelerators

○ Keep CPU busy while running algorithms on the accelerators○ Achieved with ExternalWork extension to EDProducers○ Framework reuses the thread for other modules while waiting

the external work to finish (== no locks)● Use a single CMSSW configuration for all hardware combinations

○ Achieved with SwitchProducer configuration entity○ Discover available hardware at cmsRun start time, and use

the appropriate EDProducers (can be overridden by config, etc…)

○ Files created on different hardware can be used homogeneously

● Model for NVidia GPUs with CUDA is being prototyped as part of the Patatrack project

○ Supports chains of EDProducers passing data on GPU memory

○ Data transferred to CPU only when needed○ Expose as much of the parallelism as possible

● Same or similar components are expected to be useful for other GPUs, FPGA etc.

○ E.g. SONIC project is using the system for connecting to remote FPGA resources for ML inference

14

Conclusions

● CMS has been able to digest smoothly the large HI dataset taken at the end of 2018; the reconstruction of the ParkingB dataset is imminent

● Production activities have already delivered a complete set of processed DT and MC for the full RunII, in due time for Moriond analysis preparation

● Plans for 2019, including the UL, are progressing on schedule

● CMS confirms the “+0%” requests for 2020, and has slightly reduced the 2021 modelling wrt to previous iterations 15