Rob Lambert, NIKHEFCore Soft, 26th September 20121 FSR – frustrating summary records R. Lambert file ^

Rob Lambert, NIKHEF Core Soft, 26th September 2012 1

FSR – frustrating summary records

R. Lambert

file

^

Log file

Idea


Event data

File Metadata

File MetadataData file

Event data

Data file

Log file

File Metadata

Log file

File Metadata

Log file

File Metadata

Book-keeping

Provenance information

Provenance information

Gaaah!


… and these are justthe savannah bugs/tasks

Current Structure


/FileRecords /Event

…FSR …/GUID /GUID

FSR …/GUID /GUID

FSR …/GUID /GUID

Event …Event …Event …

FSR

FSR

FSR


Current Structure


/FileRecords /Event


FSR …/GUID /GUID

FSR …/GUID /GUID

File MetadataCustom algorithms

Provenance InformationAutomatically Created

Event DataCustom Algorithms

Open File

Current Procedure


/Event

FSRs

Provenance

Output FileTES

/Event

FSRs

Execute Finalize

/Event

FSRs

Provenance

Input File(s)

+

+

Pack

Provenance

FSRsFSRs

Current Streaming


Output File(s)

TES

/Event/…

/Event

FSRs

Provenance

Input File(s)

+

+

/Event/…

Open File Execute Finalize

+

+ /Event/…

/Event

FSRs

Provenance

/Event

FSRs

Provenance

/Event

FSRs

Provenance

Pack

Pack

Pack

Provenance

FSRsFSRs

FSRs

Observations FSRs are sparse, Events are chunky

FSRs have many levels in the tree, Events have few levels

FSRs encode information in the structure, only one “event”


/FileRecords /Event


FSR …/GUID /GUID

FSR …/GUID /GUID


Observations FSRs are sparse, Events are chunky

FSRs have many levels in the tree, Events have few levels

FSRs encode information in the structure, only one “event”


/FileRecords /Event


FSR …/GUID /GUID

FSR …/GUID /GUID


Physical Problems Many levels = many baskets



Basket size is waaaaaaay too large


FSR , ~128 bits

Basket , ~262144 bits

FSR , ~128 bits

Basket , ~262144 bits



Only one “event” written, basket size never optimized


I’m sure this is fine. Don’t worry about it. Ask me again in 9 events time.




Slow to navigate





Slow to navigate

Huge memory footprint (Gigabytes)

Massive file size increase (Hundreds of Megabytes)

Crazy computing time (30 minutes in finalize)


Open File

Merging in Production


/Event

FSRs

Output FileTES

/Event

Execute Finalize

/Event

FSRs

Provenance

Input File(s)

+

+

Pack

Provenance

FSRsFSRs

FSRsMerge

Best Case Scenario Normal Production


RAW SDST DST

uDST

Brunel DaVinci DaVinci

DaVinci

Reconstruction Stripping+Streaming Merging

DST

uDST

~5 x1 FSRs0 Daughter1 Level

~5 x1 +1 FSRs1 Daughter2 Levels

~5x1+1 FSRs0 Daughter1 Level

Worst Case Scenario MC Filtering

Nominally A=1, B~10, C~100… nominally 5,000 FSRs!


SIMGauss DIGIBoole

DST LHCb DST

DST

DaVinci

DSTMoore

“A” FSRs0 Daughter1 level

A+2 FSRs1 Daughters2 levels



(A+4)xB+1 FSRs3*B Daughters5 levels

((A+4)xB+1)xC+1 FSRs3*B*C Daughters6 levels

B C

Brunel

Immediate prospects Current work around:

Delete and clean up FSRs (might as well not write them) Parse and merge the XMLSummaries instead

Fixes required Resurrect provenance information Write different FSRs out depending on stream

New smarter EventCountFSR IOFSR (prototype written) Similar IO information to the XMLSummary Stores input GUIDs with number of events read Stores vectors of information for daughter files

Creating IOFSR done by new FSR writer (prototype t.b.d.)


With new RootCnv Heard a nice talk last week on basket optimization

Repeat my older tests

Well done Markus and Ivan! But it’s still not great for FSRs


SetupProject DaVinci v32r2p1Time gaudirun.py ~rlambert/public/forMarkus/fsrMemLeak/options.py#20 files with one event each, but 4681 FSRs in total

Existing RootCnv Time (usr+sys) / s VMEM / MiB File Size / MiBWith FSR Cleaning 39.7 995 2.4

No FSR Cleaning 619.9 2704 15.2

RootCnv Head Time (usr+sys) / s VMEM / MiB File Size / MiBWith FSR Cleaning 22.5 936 2.1

No FSR Cleaning 116.7 1073 5.0

Conceptual problemsX Provenance information requires complicated navigation

… and then we throw it away anyway. Great.

X Requires a second writer and separate service

X Sequencing!1. Keep output file open2. Create FSR object and register on TES during finalize3. Write FSR object to file Very complicated once Streaming is involved

X All FSRs in output streams are identical Output files don’t necessarily have the same metadata

X Different structure suggests different optimal working point


Fixing any of these will fix FSRs for good

The Future?


10,000 FSRs Sparse Trees in TES

ROOT can’t handle sparse trees

Sparse Trees Persisted

ROOT Tree used

Deconstruction





ROOT Tree used


1. Merging (what we do right now)• Currently throws away provenance information • Requires very very careful sequencing for finalization order• Requires each FSR type to have some associated merger

• OK for small number of FSR types and a lot of manpower• … there are several other options, though…

Deconstruction





ROOT Tree used


2. Merging (automatic, and IOFSR)• Provenance information will be kept• Have merging done by a service or tool, on FSR baseclass• Do this somehow automatically and always cleanup correctly

• OK, but needs a lot of thought about how best to implement

Deconstruction





ROOT Tree used


3. Event-like FSRs• Treat each FSR as an event, file GUID instead of event number• Write to the same location several times, a mini event container• Completely changes FSR mechanics of Gaudi

• Good for any purpose.



4. Flattening (LHCb “Packed” FSR)• Write a master class which holds information at one top level• Let there be only one “FSR” which holds everything• Frequent and invasive schema migration for new FSRs

• Good solution for up to thousands of FSRs

Deconstruction




ROOT Tree used



5. Flattening (LHCb Packed FSR)• Each writer creates a /FileRecords/Packed location• Vector of data objects, only write out this location• Complicated to work around existing Gaudi FSR system

• Good solution for up to thousands of FSRs

Deconstruction




ROOT Tree used



6. Flattening (Gaudi Packed FSRs)• Let the TES and Persistent classes be different• Have the persistent class be a __flattened__ tree (auto-unpack)• Requires invasive re-coding of parts of Gaudi

• A good solution for up to thousands of FSRs

Deconstruction




ROOT Tree used


7. Abandon Root trees completely• Don’t use data objects at all, just append an ntuple to the file• Avoids all baskets and other problems• Completely changes FSR mechanics of Gaudi

• The only solution if 10,000 FSRs are needed


Deconstruction




ROOT Tree used



8. Get sparse trees in Root to work properly• Have basket size and optimization tuneable tree-by-tree• Needs work from the ROOT team• Only possible on new stack ~6-month time scale

• Perfect solution, no new LHCb/Gaudi code required


Deconstruction



ROOT Tree used


There is a possible solution for each problem …

What do we persue?

Fixing the whole idea of FSRs is very desirable

Not biting the bullet now, means more manpower later …

The Future?





ROOT Tree used

Summary FSRs are a new addition to Gaudi, not yet robust

MC Filtering is being addressed IOFSR to keep provenance information

Currently to add a new FSR type is very complicated

We can work around the problems, but shouldn’t we fix this part of our software which is underperforming? Automatic merging Event-like FSRs Flattening (Three possible directions) Abandon trees altogether Fix sparse trees


End Backups are often required


Discussion


Documents

Rob Lambert, NIKHEFCore Soft, 26th September 20121 FSR – frustrating summary records R. Lambert file ^