David Adams

ATLAS

DIAL status

David AdamsBNL

July 16, 2003

ATLAS GRID meeting

CERN

July 16, 2003 DIAL status ATLAS GRID meeting 2

David Adams

ATLAS

ContentsIntroduction

Design

Release 0.3

Release 0.4

Later this year

Batch production

Other projects

July 16, 2003 DIAL status ATLAS GRID meeting 3

David Adams

ATLAS

IntroductionDIAL described many times before

• This talk is status report

More information:• Talk for RTAG11 yesterday• Talk at the grid session of the last SW workshop• CHEP paper• DIAL home page (above and more)

– http://www.usatlas.bnl.gov/~dladams/dial

DIAL provides interface to connect• user analysis (or production) environment• to distributed job processing

July 16, 2003 DIAL status ATLAS GRID meeting 4

David Adams

ATLAS

In tera ctive a na lysise .g . R O O T , JA S , ...

D IA L

D istribu ted p rocessing ru nn ing da ta -specific a pp lica tion

D a ta s e t S c he d u le r A A AJ o b

July 16, 2003 DIAL status ATLAS GRID meeting 5

David Adams

ATLAS

DesignDIAL has the following major components

• Dataset describing the data of interest

• Application defined by experiment/site

• Task is user extension to the application

• Job uses application and task to process a dataset

• Result is the output of a job

• Scheduler creates and manages jobs

Together these define a high-level JDL • (job definition language)

Figure shows how these components interact →

July 16, 2003 DIAL status ATLAS GRID meeting 6

David Adams

ATLAS

UserAnalysis

Job 1

Job 2

Application Task

Dataset 1

Scheduler

1. Create or locate

2. select 3. Create or select

4. select

8. run(app,tsk,ds1)

5. submit(app,tsk,ds)8. run(app,tsk,ds2)

6. splitDataset

Dataset 2

7. create

e.g. ROOT

e.g. athena

Result9. fill

10. gather

Result 9. fill

Result Code

July 16, 2003 DIAL status ATLAS GRID meeting 7

David Adams

ATLAS

Release 0.3Release 0.3 was made last monthIt includes:

• Implementations of identified DIAL components• Simple local scheduler with no job splitting• Dataset describing combined ntuple file• Application dial_cbnt to run PAW using PAW macro

from task• Demo to illustrate look and feel

– Runs at BNL (ACF and RCF) and CERN (lxplus)– Script runs job defined by

> Application (dial_cbnt)> Task with PAW macro and empty hbook file> Combined ntuple dataset

July 16, 2003 DIAL status ATLAS GRID meeting 8

David Adams

ATLAS

Release 0.4Release 0.4 expected in mid-August

It will add the following• Access to files in Magda

• Local distributed processing– Most likely using LSF

– Available at least at BNL

• Interface and simple implementation of dataset/job splitting

• Demo(s) to illustrate distributed processing with combined ntuple dataset with multiple files

July 16, 2003 DIAL status ATLAS GRID meeting 9

David Adams

ATLAS

Later this yearCondor-based processing

• local, COD, master-worker, Condor-G?

Remote access to scheduler• Job submission from anywhere• Web service

Add response time to scheduler interfaceATLAS-POOL dataset

• After ATLAS incorporates POOL

Atlsim and Athena as applications• DIAL provides interface for data production• Need transformation and dataset catalogs

– AMI, next week’s metadata meeting

July 16, 2003 DIAL status ATLAS GRID meeting 10

David Adams

ATLAS

Batch productionOriginal goal of DIAL was to provide means for interactive data analysis.

However interactive analysis and batch production are not so easily separated

• DIAL JDL is appropriate for either

• Schedulers will need to operate over a continuum of– Produced data size

– Acceptable response time

• Some users will want to interactively browse a small sample and then submit larger sample to batch

July 16, 2003 DIAL status ATLAS GRID meeting 11

David Adams

ATLAS

Other projectsJDL (job definition language)

• DIAL may be thought of as a proposal for a user-level (high-level) JDL

• Like to come to agreement on this JDL with other projects so components can be shared

• Language issues: C++, Java, Python, XML, …

Scheduler• Look to other projects to deliver robust and efficient

implementations of the DIAL scheduler• In the mean time DIAL will

– Try to wrap analogous components from other projects– Make relatively simple implementations to meet our project

goals

July 16, 2003 DIAL status ATLAS GRID meeting 12

David Adams

ATLAS

Other projects (cont)Candidates to provide DIAL schedulers:

• Look to Condor to deliver a grid scheduler

• Scheduler can be implemented using Chimera

• Scheduler restricted to ROOT as an application could be implemented using PROOF

• Scheduler for athena jobs can come from GANGA

• Tech-X has SBIR to develop JDAP scheduler for use with JAS

• A java-based scheduler is being developed by STAR

• Last (but not least?), DIAL will provide schedulers

July 16, 2003 DIAL status ATLAS GRID meeting 13

David Adams

ATLAS

Other projects (cont)Build web services to create and access DIAL schedulers

• Using Clarens and/or OGSA

Environments that might use DIAL schedulers:• A Python implementation of the JDL would enable

GANGA and PI/SEAL to use DIAL schedulers

• DIAL is already imported into ROOT

• A Java version of the JDL would enable JAS to use DIAL schedulers

• Web or script based user-level production for any experiment could benefit from the scheduler interface

July 16, 2003 DIAL status ATLAS GRID meeting 14

David Adams

ATLAS

P I/S E A L

G A N G A

R O O T

J A S

E xp e rim e ntP ro d u c tio n

D IA LJ D L

P R O O F

G A N G A

C o nd o r

C him e ra

S T A R

J D A P

D IA L

C la re ns O G S A

U se r in te r fa ce s S ch e d u le r s

W e b s e rv ice in f ra s tru ctu re