David Malon, Peter van Gemmeren (Argonne National Laboratory)Richard Hawkings, (CERN)R.D. Schaffer (LAL Orsay)

1. Abstract

In the ATLAS event store, files are sometimes "an inconvenient truth." From the point of view of the

ATLAS distributed data management system, files are too small--datasets are the units of interest.

From the point of view of the ATLAS event store architecture, files are simply a physical clustering

optimization: the units of interest are event collections--sets of events that satisfy common conditions or

selection predicates--and such collections may or may not have been accumulated into files that contain

those events and no others.

2. Architecture

3. In-File Conditions Data Example

4. Provenance Metadata

An Inconvenient Truth: file-level metadata and in-file metadata caching in the (file-agnostic) ATLAS distributed event store

It is nonetheless important to maintain file-level metadata, and to cache metadata in event data files.

When such metadata may or may not be present in files, or when values may have been updated after

files are written and replicated, a clear and transparent model for metadata retrieval from the file itself or

from remote databases is required. In this paper we describe how ATLAS reconciles its file and non-file

paradigms, the machinery for associating metadata with files and event collections, and the

infrastructure for metadata propagation from input to output for provenance record management and

related purposes.

DataFile1

MetaDataStore

CondOutStream

1 1 1 1

DataFile2

2 2

InputMetaDataStore

1 1

InputMetaDataStore

2 2

FileIncident:•EventSelectorsends endFileincident•open next file•sends beginFileincident.

FlushStore

MetaDataTool:•Create MetaData (no Input)•Read and Combine MetaData•Write MetaData.

Write

Read

EventData

EventData

MetaDataSvc:•Retrieves all registered IMetaDataTools•Listens to beginFileand endFileincidents.•Reads next DataFileand provides transient addresses.

EventSelector: Fire FileIncidents

DataFile1

MetaDataStore

CondOutStream

1 11 1 1 11 1

DataFile2

2 22 2

InputMetaDataStore

1 11 1

InputMetaDataStore

2 22 2

FileIncident:•EventSelectorsends endFileincident•open next file•sends beginFileincident.

FlushStore

MetaDataTool:•Create MetaData (no Input)•Read and Combine MetaData•Write MetaData.

Write

Read

EventData

EventData

MetaDataSvc:•Retrieves all registered IMetaDataTools•Listens to beginFileand endFileincidents.•Reads next DataFileand provides transient addresses.

EventSelector: Fire FileIncidents

Use of In-File Metadata

Data Caching: It is convenient and more effective not to have to use (rely upon) a remote Database.E.G.: Trigger information could be stored in the event file

Stream-level Metadata: A summary of the characteristics of the Events within a Stream (File). Data which is common for many or all Events can be made easily accessible by writing them on a Stream (File) level.

E.G.: Number of Events, Run Numbers and Luminosity Blocks, Type of Events and Process Stage Information.

Writing In-File Metadata

In-File Metadata is written using a Conditions Stream “shadowing” the Event Data Output Stream writing Data Objects during finalize() at the end of the job into the Event Data file. Separate POOL container are used for the Metadata DataHeader and the Data Objects.

Reading In-File Metadata

The EventSelectorAthenaPool uses the Gaudi Incident Service to fire beginInputFile / endInputFile FileIncidents before a file is opened and after the end of file was found. The MetaDataSvc will listen to these FileIncidents and act as an AddressProvider for In-File Metadata. In order to access Metadata, the MetaDataSvc will create a POOL Implicit collection over the Metadata DataHeader. Than the MetaDataSvc creates a new InputMetaDataStore into which it reads File-Level Metadata when handling a beginInputFile incident and clears that store on an endInputFile incident. In addition, the MetaDataSvc will retrieve all registered (via jobOptions) IMetaDataTools to allow the Metadata Maker to be initialized to listen to the FileIncidents.

Event collections (sets of events that share common characteristics) are fundamental, and their metadata requirements do not depend upon whether the events of interest have been gathered into their own file set.

For event data files (“implicit” collections), provenance metadata sufficient to support cross-section calculation can be stored within files and propagated correctly through the downstream workflow, even if that workflow further culls the event selection.

Event Collection

List of references to the Events satisfying the selection criteria.

MetaData for collectionprovenance

Event File

= “Implicit” Collection

In-File MetaData for collection provenance

Persistent DataObjectsfor the Events

Tag Database

Query

Event Collection

List of references to the Events satisfying the selection criteria.

MetaData for collectionprovenance

Event File

= “Implicit” Collection

In-File MetaData for collection provenance

Persistent DataObjectsfor the Events

Event File

= “Implicit” Collection

In-File MetaData for collection provenance

Persistent DataObjectsfor the Events

Persistent DataObjectsfor the Events

Tag Database

Query

Event File

EventData

IOV Db Svc•Resolve TimeStamp to IOV.•Provide Proxies for Conditions Data.•Retrieve Conditions DataObjects

Check forIn-File IOV

IOVIn-File

Provide Proxy Information for Conditions Data

Cond DBIOV DB

Reconstruction /Analysis

Check forIn-File Cond

Retrieve Conditions DataObjects

Cond.In-File

Read Conditions Data

Read Event Data

Event File

EventData

IOV Db Svc•Resolve TimeStamp to IOV.•Provide Proxies for Conditions Data.•Retrieve Conditions DataObjects

Check forIn-File IOV

IOVIn-File

Provide Proxy Information for Conditions Data

Cond DBIOV DB

Reconstruction /Analysis

Check forIn-File Cond

Retrieve Conditions DataObjects

Cond.In-File

Read Conditions Data

Read Event Data

CachingMany kinds of interval-of-validity data (e.g., trigger configurations), while authoritatively managed externally, may be cached within event data files for efficiency

Architecture is robust: if such data are not found within the file, fallback to conditions database occurs.

1

DataFile

POOLContainer_DataHeader

CollectionTree

DataHeaderDataHeader

DataHeader

abcabc

abc

StreamColl_DataHeader

Stream

DataHeader 1referencesreferences

1

DataFile

POOLContainer_DataHeader

CollectionTree

DataHeaderDataHeader

DataHeader

DataHeaderDataHeader

DataHeader

abcabc

abc

abcabc

abc

StreamColl_DataHeader

Stream

DataHeader 1referencesreferencesreferences

In the ATLAS event store, files are sometimes "an inconvenient truth." From the point of view of the

ATLAS distributed data management system, files are too small--datasets are the units of interest.

From the point of view of the ATLAS event store architecture, files are simply a physical clustering

optimization: the units of interest are event collections--sets of events that satisfy common conditions or

selection predicates--and such collections may or may not have been accumulated into files that contain

those events and no others.