View
47
Download
0
Category
Preview:
DESCRIPTION
DIAL Status and Plans. ATLAS Software Workshop Grid session CERN. David Adams BNL September 25, 2003. Goals of DIAL What is DIAL? Design Status Development plans Integration with GANGA. Contents. Goals of DIAL. 1. Demonstrate the feasibility of interactive analysis of large datasets - PowerPoint PPT Presentation
Citation preview
David Adams
ATLAS
DIAL Status and Plans
David AdamsBNL
September 25, 2003
ATLAS Software WorkshopGrid session
CERN
September 25, 2003 DIAL ATLAS SW - Grid 2
David Adams
ATLAS
ContentsGoals of DIAL
What is DIAL?
Design
Status
Development plans
Integration with GANGA
September 25, 2003 DIAL ATLAS SW - Grid 3
David Adams
ATLAS
Goals of DIAL1. Demonstrate the feasibility of interactive analysis of large datasets
• How much data can we analyze interactively?
2. Set requirements for GRID services• Datasets, schedulers, jobs, results, resource discovery,
authentication, allocation, ...
3. Provide ATLAS with a useful analysis tool• For current and upcoming data challenges• Real world deliverable• Like to add another experiment would show generality
September 25, 2003 DIAL ATLAS SW - Grid 4
David Adams
ATLAS
What is DIAL?DIAL provides a connection between
• Interactive analysis framework– Fitting, presentation graphics, …
– E.g. ROOT, JAS, …
• and Data processing application– Natural to the data of interest
– E.g. athena for ATLAS
DIAL distributes processing• Among sites, farms, nodes
• To provide user with desired response time
Look to other projects to provide most infrastructure
September 25, 2003 DIAL ATLAS SW - Grid 5
David Adams
ATLAS
In tera ctive a na lysise .g . R O O T , JA S , ...
D IA L
D istribu ted p rocessing ru nn ing da ta -specific a pp lica tion
D a ta s e t S c he d u le r A A AJ o b
September 25, 2003 DIAL ATLAS SW - Grid 6
David Adams
ATLAS
DesignDIAL has the following major components
• Dataset describing the data of interest
• Application defined by experiment/site
• Task is user extension to the application
• Job uses application and task to process a dataset
• Result is the output of a job
• Scheduler creates and manages jobs
Together these define a high-level JDL • (job definition language)
Figure shows how these components interact →
September 25, 2003 DIAL ATLAS SW - Grid 7
David Adams
ATLAS
UserAnalysis
Job 1
Job 2
Application Task
Dataset 1
Scheduler
1. Create or locate
2. select 3. Create or select
4. select
8. run(app,tsk,ds1)
5. submit(app,tsk,ds)8. run(app,tsk,ds2)
6. splitDataset
Dataset 2
7. create
e.g. ROOT
e.g. athena
Result9. fill
10. gather
Result 9. fill
Result Code
September 25, 2003 DIAL ATLAS SW - Grid 8
David Adams
ATLAS
StatusDIAL release 0.5 last week
User interfaces• ROOT
– Uses ACLiC to make all dataset/DIAL classes accessible
• Command line– Command dial_submit processes one job
– Application, task and dataset taken from XML files
Schedulers• LocalScheduler: single local job
• MasterScheduler: distributed processing at a site– Fork, LSF, lsrun or Condor
– Run at both BNL and CERN
September 25, 2003 DIAL ATLAS SW - Grid 9
David Adams
ATLAS
Status (cont)Datasets
• CbntDataset describes a single CBNT hbook file– Inherits from EventDataset
• EventMergeDataset holds multiple EventDataset
• Together these can be used build dataset which corresponds to a collection of CBNT hbook files
• A few files with XML descriptions of such datasets are available for demonstration
Results• HbookResult holds an hbook file assumed to hold a
collection of histograms
• Merge method uses PAW to add histograms
September 25, 2003 DIAL ATLAS SW - Grid 10
David Adams
ATLAS
Status (cont)Application
• dial_cbnt uses PAW to fill histograms using a user-supplied fortran function
Task• For use with the above, task holds hbook file with
empty histograms and fortran code to fill them
Possible to do distributed processing of a collection of CBNT hbook files
See dial root demo 2
September 25, 2003 DIAL ATLAS SW - Grid 11
David Adams
ATLAS
Development plansAdd dataset catalog
• Enable users to select dataset
• Fill with DC1 datasets
• See talk at DB meeting
Grid enable• Condor-G
– Scaling, reliability, response time issues
• Define DIAL web service– Client scheduler to connect from user interface
– User could submit jobs to BNL or CERN from anywhere
• DIAL grid scheduler– Use above DIAL service and/or “conventional” grid WMS
September 25, 2003 DIAL ATLAS SW - Grid 12
David Adams
ATLAS
Development plansPython user interface
• Python busses can be used as user interface– GANGA, ATLAS interactive, PI, …
ROOT CBNT application• Similar to HBOOK version
Athena application• Enable DIAL users run distributed athena• Initial application to fill histograms• Later add support to create new dataset (production)• Need athena-compatible dataset
– ATLAS POOL event collection– Zebra, Athena-ROOT obsolete?
September 25, 2003 DIAL ATLAS SW - Grid 13
David Adams
ATLAS
Integration with GANGAWe identified the following goals and tasks for DIAL/GANGA integration:
• Provide dataset access from GANGA– Add DS catalog to DIAL
– Import dataset classes to Python
• Enable DIAL job submission (CBNT datasets) from GANGA
– Modify GANGA job interface to use datasets
– Import DIAL classes into Python
– Provide GANGA GUI interface to DIAL scheduler
September 25, 2003 DIAL ATLAS SW - Grid 14
David Adams
ATLAS
Integration with GANGA (cont)• Enable GANGA job monitor to track DIAL jobs
– Define common XML job description
– Incorporate in GANGA and DIAL
• Enable GANGA to used DIAL for distributed production with athena
– Add athena application support to DIAL
– Add GANGA GUI for this job submission
September 25, 2003 DIAL ATLAS SW - Grid 15
David Adams
ATLAS
What about ARDA?
Information Service
Authentication
Authorisation
Audi ting
Grid Monitoring
Workload Management
Metadata Catalogue
File Catalogue
Data Management
Computing Element
Storage Element
Job Monitor
Job Provenance
Package Manager
DB Proxy
User Interface
API
7:
12:
5: 13:
8:
15: 11:
9: 10:
1:
4:
2:
3:
6:
14:
DIALdataset
GSI
Chimera
Pacman
MonaLisaAMI
Magda
RLS
GANGA
AMI?MDS
EDG RB
Condor
ROOT
September 25, 2003 DIAL ATLAS SW - Grid 16
David Adams
ATLAS
What about ARDA? (cont)Architectural Roadmap towards Distributed Analysis
• LCG RTAG
DIAL dataset• More general than HEPCAL/ARDA dataset
• Datasets and their catalogs are part of the metadata and data management services identified by ARDA
DIAL components• Define language by which user interacts with workload
management service
• May also intrude into this service– At what point in the grid do we turn DIAL datasets into files?
– How do we get interactive response?
Recommended