Horst Fischer University Freiburg /

CERN

COMPASS Computing

Outline

•Central Data Recording and Event Database

•Event Reconstruction

•Some words on statistics

•Analysis Model

A word of warning:2001 COMPASS Detector commissioning2002 COMPASS Analysis commissioningHence, some numbers are preliminary and not yet on rock solid ground

Long-distanceG igabit E thernet

C P U C P U C P UC P UC P U C P UC P U C P U C P UC P UC P U C P U

C P U C P U C P UC P UC P U C P UC P U C P U C P UC P UC P U C P U

C P U C P U C P UC P UC P U C P UC P U C P U C P UC P UC P U C P U

C P U C P U C P UC P UC P U C P UC P U C P U C P UC P UC P U C P U

C P U C P U C P UC P UC P U C P UC P U C P U C P UC P UC P U C P U

C P U C P U C P UC P UC P U C P UC P U C P U C P UC P UC P U C P U

C P U C P U C P UC P UC P U C P UC P U C P U C P UC P UC P U C P U

C P U C P U C P UC P UC P U C P UC P U C P U C P UC P UC P U C P U

C ASTO RD ata Server

D ata Server

D ata Server

D ata Server

D ata Server

D ata Server

35 MB/s input rate

Central Data RecordingCentral Data Recording

Data servers:Up to 20 x 500GB During shutdown: Use disk of online farmas additional stage pool(additional 3.5 TB)

CASTOR:CERN development of a“infinite file system”COMPASS is the first experiment which uses CASTOR heavily

DST production

tape

Central Data Recording

Design value (35MB/s)

260 TByte in ~100 days

Compare to BaBar:

1TByte/day

662 TByte in 1999-2002

Or CDF: 500 TByte/year

(U.S. year >> 100 days)

May 27 Sep 18

GB

GB

Data in Database

2001:Detector commissioning: 65 days, 13 TByteTarget polarized longitudinally: 14 days, 15 TByte total: 28 TByte

2002:Target polarized longitudinally: 57 days, 173k spillsTarget polarized transversely: 19 days, 52k spills total: 260 TByte

2002 on average 22k triggers/spill 5 Gev (3.8Gev, 1.2Gev) online: ~260,000 files (~80 files /

run)

Objectivity faced out at CERN and replaced by Oracle9i

After transfer to CCF ~2000ev/s are formatted into Objectivity/DB Different data bases for raw events and special events Planed: event reconstruction before storing DB federations to tape 2002: offline reconstruction (calibrations were/are not available)

Database Migration

Raw events• Original DATE format, flat files kept in Castor

Metadata• Mainly event headers and navigational information for raw and

reconstructed events in relational database• 0.25% of the raw data volume• at CERN stored in Oracle (but without Oracle-specific features)• Possibility for another database in the outside institutes

(MySQL)

Conditions data• Migrated to the new CDB implementation based on Oracle• No interface change (abstract interface)

DST• Object streaming to files

More on DB Migration

CERN IT/DB is involved in the migration project to a large extent

Planning activities started early this yearSeveral people (IT/DB/PDM section) working on the projectCoordinating experiments and with other IT groups

– Servers: ADC group

– Castor: DS groupMigrate database to “borrowed” tapes

COMPASSIT/DB is working closely with COMPASS expertsCertify interfaces to Oracle DB

Licenses being negotiated between CERN and Oracle for >2003

Event Reconstruction

Compass Computing Farm: 100 dual Processor PIII Linux PCs Mainly for event reconstruction and (Mini)-DST

production maintained and operated by CERN staff Part of CERN LSF BATCH-system

Average time to reconstruct one event: 400ms/ev (300…500 ms, during the first 400ms of a spill up to 1300ms/ev)

5Gev * 400ms/ev = 2Gs / 200CPUs = 10 Ms/CPU at 100% efficiency: = 115 days on 200

CPUs

Today: about 30% of 2002 data are being pre-processedfor preliminary physics analysis (since September)

Process as much in parallel as possible: 1 run : ~ 80 data files

: ~ 80 batch jobs running at the same time

Batch Statistics

Jobs per weekIt’s a full time job to control data flow and quality of output

Must set-up

production shifts

to share work load

like one does for

data taking

Alternative: Outsourcing

= hire technicians?

CPU time per week

COMPASS Analysis Model

CalibrationsMySQL

DSTOracle 9i

mini DSTROOT tree

RAW dataOracle 9i

CORAL

PHAST

micro DSTROOT tree

micro DSTROOT treemicro DST

ROOT tree

ConditionsPVSS/Oracle 9i

e-LogbookMySQL

CORAL - Event Reconstruction

= COMPASS Reconstruction

and AnaLysis Program Modular architecture

Following OO techniques

Fully written in C++

Written from scratch

Defined interfaces for easy exchange of external packages (e.g. OB/DB Oracle9i

Access to event, conditions and calibration data bases

Not as user friendly, however, as it may look

Calibrations etc.

Presently (2001 & 2002):Presently (2001 & 2002):

Calibration DB: 42 MByte

Map DB: 4.5 MByte

Geometry DB: 4.4 MByte

Calibrations stored in Run dependent file structured data base which allows for different versions Bookkeeping: MySQL (project launched recently)Calibrations: Alignment (Torino)

RICH refraction index, PID likelihood (Trieste)

Time-zero calibrations (Bonn, Erlangen & Mainz)

r-t relations (Saclay & Munich) Amplitude and time for GEM & Silicon

(Munich) Calorimeter (Dubna & Protvino)

Detector calibrations can not be automated yet. Still everyday new surprises …

Is a drain for our most valuable resource “manpower”Tasks must stay in responsibility of the listed groups, otherwise very inefficient because of learning curve

Experience: Reconstruction & Analysis

(my personal, subjective, view)

“Distributed” data analysis is very inefficient (presently!)Size of individual groups participating in analysis is not yet large enough to make fast & significant progress by their own

•Frequently struggling with infrastructural problems•Communication between satellites is poor or not existing•Missing discussions & stimulations This unsatisfactory situation may improve with time (?)

COMPASS needs a strong & centralized analysis group

Exchange with satellite groups due to travel of people coming/leaving from/to home institutes

Short term SOLUTION:

Analysis: Where & Resources

significant core of experts @

CERN

“Analysis Center”

Resources: CCF+…

Trieste(55 GHz, 4 TB)

TU Munich(26 GHz, 2.4TB)2003: double

GridKa (German LHC Tier 1)

530 GHz, 50TBCOMPASS: 20 GHz

2003: double

Freiburg(40 GHz, 3TB)

2003: *1.5

Dubna(14.5 GHz,1.7TB)

2003: double

Saclay(Lyon)

Mainz/Bonn

Torino(66 GHz, 1 TB)

Conclusions & Suggestions

COMPASS faces its first year of “real” data analysisChallenges are enormous: calibrations, data handling, etc.… but very interesting physics ahead and thus its worth all the efforts (and sleepless nights)

Unfortunately we are forced to change event data base in a moment when we should work with full pace on our first data analysis (disadvantage when you are dependent on third party decisions!)

Tools for event reconstruction and (Mini- & Micro-) DST analysis exist but need continuous improvements and adaptations to reality

Still weak on tools to submit & monitor and manipulate & handle output

of 200+ batch jobs per hour (effort was underestimated by many of us)

Presently under evaluation: ALIEN (ALICE development)

Organizational structure of analysis group: 1. Establish a strong core group at CERN with regular

WORK(!)shops2. Setup shifts for DST production