LCG 4-Feb-2004GridPP-9 : EdinburghSlide 1 ARDA Architectural Roadmap towards Distributed Analysis

4-Feb-2004 GridPP-9 : Edinburgh Slide 1

LCG

ARDA

Architectural Roadmap towards Distributed Analysis


LCGWhat is ARDA ?

• Is it middleware (e.g. “second generation” ) ?• Is it a common application layer above middleware ?• Is it a co-ordination project ?• Is it about distributed analysis or wider HEP grid use ?• Is it “All things to all men” ?• Is it the solution to all our (Grid) problems ?

Possibly ALL the above…and more !!

• Does any one understand what this all means ?

A call for volunteers !!


LCGOutline

• History– HEPCAL I– GAG and HEPCAL II

• ARDA Research & Technology Assessment Group– Set up Spring 2003; Reported late Autumn 2003– ToR– Survey– Conclusion– Recommendations

• Workshop (21st/22nd Jan 2004)• Status


LCG

Some History


LCGWhat we want from a

GRID(EDG meeting, NIKHEF, March 2001, © fca 2004)

OS & Net services

Bag of Services (GLOBUS)

GRID middleware

HEPVO common application layer

Earth Obs. Biology

ALICE ATLAS CMS LHCb

Specific application layer

WP9 WP 10

Mar

ch 9

2001


LCGWhat we have(HEPCAL presentation, CHEP 2003, © fca 2004 )

OS & Net services


Middleware



WP1 WP2 WP3 WP4 WP5Semantic gap


LCGA proposal(HEPCAL presentation, CHEP 2003, © fca 2004 )

OS & Net services



DataGRID middleware

WP1 WP2 WP3 WP4 WP5

Common use casesVO common application layer

If we manage to define

Middleware

WP1 WP2 WP3 WP4 WP5

It will be easier for them to arrive at



LCGHEPCAL II

• (If HEPCAL I was Batch, then HEPCAL II is Analysis)• Analysis Execution Models

– Support for queries by common layers and at dataset level– Support for job pipelines

• Users, Groups, Quotas and Permissions• Interactive .vs. Batch Grid Activity• System Requirements

– Provenance & Job Traceability– Log Book and Reports– Persistent Interactive Environment– Analysis Software deployment

• Use cases (as sequence of user operations)– Production Analysis– (Sub-)Group level Analysis– End-user level Anlaysis


LCG

Enter ARDA


LCG


LCGRTAG ToR

Motivation:• To agree on requirements as laid out in a first step by recent

work within the GAG and identify commonalities within the current projects that might allow the LCG (both in the AA and GTA areas) to provide a focus of effort.

• To provide guidance to the LCG on future Middleware development directions and interfacing work to match the experiment requirements

• To build on the richness of the current technical solutions to avoid duplication of efforts

• To clearly identify the roles and responsibilities of the components/layers/ services in the experiment DA planning

• To give guidance to the community on the expected division of work between the experiments, the LCG and the external projects.


LCGRTAG ToR

Mandate:• To review the current Distributed Analysis (DA) activities and to

capture their architectures in a consistent way• To confront these existing projects to the HEPCAL II use cases and

the user's potential work environments in order to explore potential shortcomings

• To consider the interfaces between Grid, LCG and experiment-specific services– Review the functionality of experiment-specific packages, state of

advancement and role in the experiment– Identify similar functionalities in the different packages– Identify functionalities and components that could be integrated in the

generic Grid middleware• To confront the current projects with critical Grid areas• To develop a roadmap specifying wherever possible the architecture,

the components and potential sources of deliverables to guide the medium term (2 year) work of the LCG and the DA planning in the experiments


LCG


LCGThe Report

• Reviewed existing projects– PROOF and the Grid– AliEn – web services (ALICE) **– Clarens – web services (CMS)– DIAL (ATLAS) – workflow– GANGA (LHCb/ATLAS) – high level job submission– DIRAC (LHCb) – distributed MC production

** selected for further analysis (best meets requirements of experiment, used in anger by an experiment…)

• The ARDA Blueprint– Service Descriptions + APIs (access to services)Information, Authentication, Authorisation, Audit, Accounting,

Workload Management, Job Provenance, File Catalogue, Metadata Catalogue, Data management, Site Gatekeeper, Storage Element, Computing Element, Job Monitoring, Package Manager, Grid Monitoring


LCGExample : AliEn


LCGExample : AliEn


LCGExample : Clarens

Web Services-Secure file access-SRB-POOL-VO mgmt


LCGExpanded AliEn Services


LCGSet of Services


LCGThe ARDA Prototype

• “The main goal of an ARDA prototype is to provide a more complete blueprint and to develop the specifications and interfaces for the ARDA services and API”

• “We recommend that the LCG setup a project to develop a prototype…”

• 6 months for first prototype (driven by need for TDRs in 2005 ?)

• ARDA Project– Careful definition of work areas– Define project constituency– Define project lead -> project team– Work plan, schedule, milestones

• Particularly plan for interfacing to and engaging with LHC experiments and the LHC related Grid community


LCGRTAG Recommendations

RTAG report proposed a four-prong approach :

• Re-factoring of AliEn and other services into ARDA, with an initial release; consolidation of the API working with the experiments and the LCG-AA; release of a fully functional prototype. Subsequently implementation of agreed interfaces, testing and release of the prototype implementation.

• Modelling of an OGSI-based services infrastructure, performance tests and quality assurance of the prototype implementation

• Interfacing to LCG-AA software like POOL and ROOT• Interfacing to experiment's frameworks, with specific meta-

data handlers and experiment specific services

• n.b. OGSI WSRF (or perhaps WS in the first instance)


LCGMessage of ARDA Report

(© Miron Livny)

Deliver end-to-end capabilities (from user to fabric) and stability (deployable) at the price of services offered (functionality)

Services provide a natural abstraction and powerful software engineering constructs

AliEn provides a useful and stable suite of services as it meets the expectations of the Alice experiment


LCG

The ARDA Workshop


LCGARDA Workshop - Goals

CERN : 21st/22nd January 2004 : (Les Robertson)

Define the ARDA project• The scope of the distributed analysis requirements that should be

addressed• The scope of a generic middleware component,

– the approach to implementation– target timescales

• Which HEP-specific components should or could be done in common in the LCG Applications Area– e.g. POOL, collections, meta-data, SEAL, ..

• A process for agreeing on the specification of ARDA services – middleware projects, experiments

• The framework for an ARDA implementation project, coordinating – Middleware ↔ LCG AA ↔ experiment analysis s/w ↔ end-users


LCGWorkshop Agenda

• Requirements (HEPCAL II) - Federico Carminati

• ARDA Architecture - Lothar Bauerdick• Experiment expectations from ARDA – 4 talks• Generic Middleware - Frederic Hemmer

- Miron Livny• Applications Area involvement in ARDA - Torre Wenaus• POOL-ARDA collaboration - Dirk Duellmann• Discussion…


LCGExperiments Sumamry

(© Andreas Peters)


LCGExperiments Summary

(© David Adams)

• ATLAS strategy– Use grid service model– Quickly define high-level service interfaces and implement

services and clients– Deliver end-to-end system to users– Frequently re-design and re-implement based on user feedback

• ARDA collaboration– Ideally we would come to consensus within ARDA on a high-level

interface along the lines of AJDL and share end-to-end effort– In any case, we will work closely with ARDA to define middleware

services



(© Julian Bunn (et al))

CMS• Distributed Production

– CMS is already using successfully LCG and VDT grid middleware to build its distributed production system

– A distributed batch-analysis system is being developed on top of the same LCG and VDT software

– CMS suggests that in the short term (6 months) ARDA extends the functionality of the LCG middleware to meet the architecture described in the ARDA document

• Grid Analysis Environment– CMS asks that the GAE be adopted as the basis for the ARDA

work on a system that supports interactive and batch analysis. Support for Interactive Analysis is the crucial goal in this context.



(© Andrei Tsaregorodtsev )

LHCb• Expected from ARDA

– More efficient development process:• Rapid development cycles;• Keeping the functional core and adding functionality incrementally;• Emphasis on intensive testing while the development.

– Concurrent development of components to try out different ideas and to enhance the quality by competition.

• Participation– Participate to the definition of the services interfaces, testing and

feedback;– Prototyping ARDA components using OGSI compliant implementations;– Developing the DIRAC WMS into an ARDA compliant service

• The first tests of the distributed analysis will be done during the DC2004 (May) using LHCb developed and LCG tools;

• We see the further evolution of the LHCb distributed analysis tools within the context of the ARDA architecture and the proposed development process.


LCGExperimentsSummary Summary !

• A cynical view…!!

• We think ARDA’s the way forward

• We will be ARDA compliant

• …but would like ARDA to be based on us as a starting point !!


LCGMiddleware(from report to prototype)

(© Miron Livny)

• Understand AliEn (including services) • Identify potential contributions from existing middleware• Understand requirements (how does analysis differ from

“production”?)• Develop a plan – what, how, when, who

– Semantic of exposed services– Authentication/protection model – Integration/testing/deployment procedures– Documentation– …

• Execute the plan!


LCGMiddleware(working document)

(© Miron Livny)

Abstract: This working document is used to break down the high level services defined by ARDA to actual components and tries to map these components to existing implementations coming from AliEn, EDG, and VDT. The structure and initial AliEn input is taken from Chapter 5 of Draft v0.2 of the ARDA document (unpublished)

• Started after the December meeting as a vehicle to exchange and record information and ideas among the middleware providers.– Identification of services

• Service interplay and semantics– Understand how existing MW could implement these services

• Input from AliEn, EDG, VDT, commercial, …. (others?) – Specify interfaces to applications


LCG

General ARDA-AA Objectives (© Torre Wenaus)

• Common software above the middleware layer– Adapting, extending, interfacing AA software for ARDA– Participating in ARDA interface definition; ensuring AA

requirements met– Applying lower level middleware services in specialized higher

level services directed at HEP and analysis• Early PEB agreement on ARDA: Middleware covers as much as

possible; remaining higher levels covered by AA (if common) or experiments (if not)

• Integration and validation– Integrating ARDA middleware services and analysis application

level services into end-to-end distributed analysis prototype– Assisting integration of distributed analysis prototype or

components thereof into experiment environments– Validation of the prototype [and feedback to middleware

providers]• Proposal to use the GAG as the principal feedback channel

seems a very good one


LCGSummarizing TW’s

Current Thoughts on WPs (© Torre Wenaus)

1) Integration and Validation– Primarily providing coordination, communication, coherence for

efforts residing in the experiments and projects• Some similarity to Physics Validation in the simulation

project• Though the (majority of the) work will go on in the

experiments and projects, a common focal point is needed if it is to be a common effort

2) Event data management– Physics-driven event collections– Joint WP with POOL Collections

3) Framework integration– ‘Thin’ adaptation of middleware services to whatever is required

for integration in experiment analysis frameworks– Joint WP with SEAL


LCG

Status

Al At C L

POOL

SEALGAG

EGEE/VDT

ARDAcoordination

integration, validationdevelopment specific

services

ARDA ServicesGrid middlewarePOOL’, SEAL’,HEP-specific

SEAL’ Services

POOL’ Services

Grid Middleware Services

Experiment PrototypeApplications

Al

At

C

L

Experiment Requirementsand Use Cases

Consolidated Requirements

Operational experience

physicists with real

requirements

ADA GAE DIRACALIEN

grid technologyand experience

ExperimentLCG

Grid MWARDA

AliEn

EDG

VDT.. ..

OtherCommonproject

HEP-common Service

Nordu

(© Les Robertson)


LCG

GAG

LCG

PEB

Alice ATLAS CMS LHCb

Generic Middleware

Resource providers

RequirementsGuidelines

Service specs

Integration: workers’ forum LCG AAprojects

ARDA – “A Realisation of Distributed Analysis”


LCGPostscript

• HEPCAL II provides reasonable set of use cases as input– But must be complemented by use cases from non-HEP (within

EGEE)

• Proposed set of services for DA (under HEPCAL II) is reasonable

• Prototypes should leverage existing technologies and experience within a web services framework

• Prototype(s) needed (urgently) within 6 months• Applications middleware must interface (POOL, LCG-AA,

ROOT, GAE, …) as should experiments software

• The PEB are still working on the details


LCG

• If you think ARDA is a threat …– It’s here to stay, so get used to it !!

• If you think ARDA is an opportunity– Then grasp it (but don’t hang around as timescales are

ridiculously short (what’s new ?)) !!

It’s for you to choose !!

“We should not miss this opportunity to put it all

together!” (© fca 2004)


LCG

The End

Documents

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 1 ARDA Architectural Roadmap towards Distributed Analysis