Past and Future of ROOT in the CMS DQM Tools · DQM DQM DQMGUI DQMGUI elements DQMtracksdata qualityandsoftware evolution DQMprocessingruns aspartofCMSSW DQMrunswhere CMSSWruns ButDQMdataisalso

About DQMThe DQMGUI ecosystem

ROOT Wishlist

Past and Future of ROOT in the CMS DQM Tools

Marcel Andre [email protected]

2018-09-12

M. Schneider, ROOT Users’ Workshop 2018 ROOT in CMS DQM 1 / 20


ROOT Wishlist

DQM EverywhereWhy is it hard?

DQM Runs Everywhere

HLTPromptReco

DQM

MC RelVal

HDQM

validatorsDC

expertsP5shift

golden JSON

validated release

detectorexperts

CMSevent data

code

online offline

Run Registry

simulation

monitor

“good/bad”

DQM

DQMGUI

DQM DQM

DQMGUIDQMGUI

elements

▶ DQM tracks dataquality and softwareevolution

▶ DQM processing runsas part of CMSSW

▶ DQM runs whereCMSSW runs

▶ But DQM data is alsohandled in many otherplaces!



ROOT Wishlist


ROOT in CMS DQM

HLTPromptReco

DQM

MC RelVal

HDQM

validatorsDC

expertsP5shift

golden JSON

validated release

detectorexperts

CMSevent data

code

online offline

Run Registry

simulation

monitor

“good/bad”

DQM

DQMGUI

DQM DQM

DQMGUIDQMGUI

elements

▶ ROOT is used for...▶ Filling histograms▶ Rendering histograms▶ Storage of histograms▶ Transport of

histograms



ROOT Wishlist


Why is DQM hard?

▶ The common case is easy!▶ Data is taken at the experiment site▶ The online shifter looks at a few live DQM plots▶ Data is processed offline▶ The subsystem shifters look at a few plots of the runs this week▶ Done.



ROOT Wishlist


Why is DQM hard?

▶ But then data turns bad.▶ Multiple experts look at all the plots they can get hold of.▶ Some experts add new plots, changes are rushed into production.

▶ Anything that might be useful gets monitored, 10000’s of new plots.

▶ Other experts start digging through data from years ago.▶ Yet another expert builds a script to scrape for a similar incident in the past.

▶ The script is not very efficient, but nobody got time for good coding now.

▶ The issue gets fixed.▶ Most of the plots stay in place, they are forgotten, or might be useful in the

future.



ROOT Wishlist


Why is DQM hard?

▶ During all of this, DQM is expected to run reliably.▶ Downtime of DQM in normal operation is not a big problem, does not affect

data taking.▶ But downtime of DQM in this scenario means lost data.



ROOT Wishlist


Maintainability

▶ Even after a decade of operation, DQM is constantly changing.▶ Changes (and bugs!) are introduced every day.▶ The DQM tools must adapt, and tolerate occasional misbehaviour.



ROOT Wishlist

CMS DQMGUI is many things

▶ DQMGUI, the web-based DQM data browser▶ DQMGUI, the histogram visualization tool▶ DQMGUI, the eternal histogram archive▶ DQMGUI, the live monitoring protocol▶ DQMGUI, the API to CMS DQM data



ROOT Wishlist

But also other tools

▶ Historic DQM, time-trends derived from theDQMGUI archive

▶ Tracker Maps, a visualization not possible inDQMGUI

▶ AutoDQM, a comparison tool for shift work▶ FastHadd, fast histogram merging on the

HLT farm▶ ... and more that we don’t know about – we

don’t know all the users of our APIs!



ROOT Wishlist

Session-less ROOTEfficient advanced renderingCompact long-term storage

Long Shutdown 2 is coming

▶ We want to revise some of these tools in the long shutdown.▶ Maintainability is a big problem for some of them.▶ For some, their existence points out omissions in the original design.▶ We could use some help – looking for solutions we can share with others.



ROOT Wishlist


Session-less ROOT

▶ In many places, we need ROOT, but don’t want long-lived sessions.▶ Launching a new session every time is (probably) too slow.▶ Prime target: Histogram rendering.

▶ Time budget: about 100ms per plot.



ROOT Wishlist


Testimonials

Early on it became abundantly evident ROOT was neither robust nor suit-able for long-running servers. Some three quarters of all the effort on theentire DQM GUI has gone into debugging ROOT instabilities and produc-ing countermeasures. We are very pleased with the robustness of the rest ofthe DQM GUI system.

— Lassi Tuura et al.1

1L Tuura, G Eulisse, and A Meyer. “CMS data quality monitoring web service”. Journal ofPhysics: Conference Series. Vol. 219. 7. IOP Publishing. 2010, p. 072055.



ROOT Wishlist


Client side ROOT

▶ We want powerful web interfaces that allow scripting.▶ But we can’t run arbitrary code on the server side.▶ Can we use ROOT7/JSROOT here?▶ Bonus: It has to be fast, 100’s of histograms per page.

▶ We tried D3.js, it was too slow.



ROOT Wishlist


Testimonials

The histograms are rendered in separate fortified processes to isolate theweb server from ROOT’s instability.





ROOT Wishlist


Efficient advanced rendering

▶ We want more natural display of detectorcomponents.

▶ TH2Poly has too much overhead for CMSSW.▶ Complex user-supplied code is too

dangerous.▶ It needs to be accessible for Physicists.



ROOT Wishlist


Compact long-term storage

▶ We need a format that can store billions of histograms for decades andprovide fast, random access.

▶ We have a solution, but others might be interested as well.

▶ The index : Custom-built database with data layout optimized for theapplication.

▶ Running since 2015: 44G objects, 4.1TB, random access time < 1s.▶ But: We can’t maintain a custom database backend and on-disk format

eternally.



ROOT Wishlist


Compact long-term storage

▶ We need a format that can store billions of histograms for decades andprovide fast, random access.

▶ We have a solution, but others might be interested as well.

▶ The index : Custom-built database with data layout optimized for theapplication.

▶ Running since 2015: 44G objects, 4.1TB, random access time < 1s.▶ But: We can’t maintain a custom database backend and on-disk format

eternally.



ROOT Wishlist


Testimonials

As reading ROOT files in the web server itself would be too slow, use toomuch memory, prone to crash the server, and would seriously limit concur-rency, we index the ROOT data files on upload.





ROOT Wishlist


Robust Schema Migration

▶ We use TBuffer in a number of places.▶ (Lack of) schema migration broke things three times this year4.▶ Are we doing it wrong?

▶ Example fastHadd: Transfer and merge histograms out of HLT viafile-based DAQ.

▶ Merge ∼10M, transfer > 100M histogram bins every 23s.

4After years of reliable operation.M. Schneider, ROOT Users’ Workshop 2018 ROOT in CMS DQM 18 / 20


ROOT Wishlist


Robust Schema Migration

▶ We use TBuffer in a number of places.▶ (Lack of) schema migration broke things three times this year4.▶ Are we doing it wrong?

▶ Example fastHadd: Transfer and merge histograms out of HLT viafile-based DAQ.

▶ Merge ∼10M, transfer > 100M histogram bins every 23s.

4After years of reliable operation.M. Schneider, ROOT Users’ Workshop 2018 ROOT in CMS DQM 18 / 20


ROOT Wishlist


Testimonials

A new, dedicated utility, fastHadd, has been developed [...] in order to beable to keep the pace of HLT event processing. This work required the in-troduction of a new file format which is based on Protocol Buffer, whichinternally still uses the native ROOT objects, serialization and summingroutines. The careful handcrafting of the new file-layout allowed to signif-icantly boost the performances.

— Marco Rovere5

5M Rovere. “The Data Quality Monitoring Software for the CMS experiment at the LHC”.Journal of Physics: Conference Series. Vol. 664. 7. IOP Publishing. 2015, p. 072039.



ROOT Wishlist


Summary

▶ Our main concern is long-term reliablility.▶ We are happy with what our tools do, but not with how they do it.▶ We want to redo some of the tools and reuse/share more technology.▶ We need advanced visualization without server-side state, and scalable

storage.


BackupThe recent ROOT issuesThe CMSSW sideDQMGUI

Digression: the recent ROOT issues in CMS online DQM

▶ We had issues with ROOT schema migrations three times this year:▶ When the patch for ROOT-9173 was introduced (TH1 7 -> 8)▶ When the patch for ROOT-9173 was forgotten on ROOT 6.12.▶ When TF1 changed version.

▶ Each time, it broke (parts) of the live monitoring network protocol, that usesbare TBuffers.

▶ Each time, it was unexpected:▶ ROOT-9173 was a bugfix applied by CMS with no version number change.▶ ROOT 6.12 then caused a decrease (!) in ClassDef version.▶ TF1 was never supposed to end up in DQM but somehow did.



The CMSSW side

▶ To be able to do DQM in multiple threads at acceptable memory use, weneed concurrent Fill calls.

▶ The main problem is making everything except the fill calls thread safe.▶ Then, any solution for Fill is fine:

▶ A lock in our wrapper (MonitorElement) around TH1 (used today for a fewmodules)

▶ Atomic fill as a separate API in TH1, to be used via the wrapper▶ Atomic fill in our own (fictional) histogram type▶ RHist with concurrent fill

▶ We will (most likely) need a conversion to TH1 for a significant amount of“harvesting” code.



Why does TH2Poly not work for us?

▶ Sparse backing array: not acceptable for CMSSW.▶ Geometry should be shared between many instances.▶ TH2Poly with fixed geometry as a rendering backend in a render plugin is

feasible, but complicated.▶ Even “normal” TH1 overhead (>1kB) is problematic!



How does DQMGUI rendering work today?

▶ All rendering happens server side in about 5 long-running render processes.▶ Rendering is6 stateless, all parameters in the URL.▶ Subsystems can supply render plugin code (C++) to run in the renderer.

▶ No versioning: Render plugin needs to handle any data ever produced.▶ Tight review: Render plugins must not leak memory or modify state.

▶ CPU budget is about 10ms (!) per plot.▶ JSROOT/d3 based rendering was tried, but too slow for thumbnails.

6pretends to beM. Schneider, ROOT Users’ Workshop 2018 ROOT in CMS DQM 24 / 20


How could DQMGUI rendering work in the future?

▶ Users can write aggregation, projection, and rendering code in the browser.▶ Code can be shared, and eventually be enabled by default.▶ User supplied code will never break more than the user’s own session.▶ It is possible to, and we encourage people to, do advanced things, like

detector-specific visualizations and statistical comparisons.▶ See also: Machine Learning



How does the histogram archive work today?

▶ Custom built database (the index) with application-optimized structure.▶ TH1 are decomposed into axis, labels, etc, which are stored only once.▶ Streamers are stored in the database, managed manually.▶ 44G histograms in 4.1TB on a single machine, performing well.



How could the histogram archive work in the future?

▶ Same structure as the current index mapped onto a SQL schema?▶ Split the database by e.g. year, to get archivable blocks.▶ Move to a parallel/distributed storage backend.

▶ Off-the-shelf key-value store was considered for many years.

▶ Move to a non-ROOT, e.g. compressed JSON, histogram representation?



How does the online live monitoring protocol work today?

▶ Simple network protocol based on TCP and TBuffer.▶ No streamers: CMSSW producing histograms and DQMGUI receiving need

compatible versions.▶ That is not easy, as they are completely independent projects.▶ CMSSW uses non-standard patched ROOT!

▶ Multiple sources (CMSSW) and consumers (DQMGUIs) connect to onecentral collector.

▶ Worked fine for many years, broke multiple times this year.



How could a future live monitoring protocol work?

▶ Based on files on network shares, like all of CMS DAQ▶ Causes additional latency (about 1 CMS Lumisection = 23s), but DAQ has a

multiple of that anyways.

▶ Unify online and offline, feed everything through the archive.▶ Functionally possible today but too slow.

▶ File formats to choose from: basic ROOT (used in offline today),CMSSW-flavoured ROOT (DQMIO), Protobuf with TBuffers (used infastHadd)



What is fastHadd and why does it exist?

▶ Utility that merges collections of histograms and sums matching ones.▶ Uses a ProtoBuf based format with TBuffers (without streamers) and keeps

histograms sorted.▶ Used to transport DQM histograms computed at HLT out of the HLT farm

via the file-based DAQ.▶ Exists for performance reasons: 1000’s of histograms per HLT machine need

to be handled every 23s.


Documents

Past and Future of ROOT in the CMS DQM Tools · DQM DQM DQMGUI DQMGUI elements DQMtracksdata qualityandsoftware evolution DQMprocessingruns aspartofCMSSW DQMrunswhere CMSSWruns ButDQMdataisalso