ATLAS and the Grid ACAT02 Moscow June 2002 RWL Jones Lancaster University

ATLAS and the GridATLAS and the Grid

ACAT02 Moscow June 2002ACAT02 Moscow June 2002RWL Jones Lancaster UniversityRWL Jones Lancaster University

RWL Jones, Lancaster University

The ATLAS Computing ChallengeThe ATLAS Computing Challenge

Running conditions at startup:Running conditions at startup:

0.8x100.8x1099 event sample event sample 1.3 PB/year1.3 PB/year, before data processing, before data processing

““Reconstructed” events, Monte Carlo data Reconstructed” events, Monte Carlo data ~10 PB/year (~3 PB on disk) ~10 PB/year (~3 PB on disk)

CPU: ~1.6M SpecInt95 including analysisCPU: ~1.6M SpecInt95 including analysis

CERN alone can handle only a fraction of these resources

2007Average Luminosity (10^33) 1Trigger Rate (Hz) 160Physics Rate (Hz) 140Running (Equiv. Days) 50Physics Events (10^9) 0.8


The Solution: The GridThe Solution: The Grid

Note: Truly

HPC, but requires

more

Not designed for tight-coupled

problems, but spin-offs many


ATLAS Needs Grid ApplicationsATLAS Needs Grid Applications

The ATLAS OO software framework is Athena, which co-The ATLAS OO software framework is Athena, which co-evolves with the LHCb Gaudi frameworkevolves with the LHCb Gaudi framework

ATLAS is truly intercontinentalATLAS is truly intercontinental In particular, it is present on both sides of the AtlanticIn particular, it is present on both sides of the Atlantic

Opportunity: the practical convergence between US and European Grid projects will come through the transatlantic applications

Threat: There is an inevitable tendency towards fragmentation/divergence of effort to be resisted

Other relevant talks:Other relevant talks: Nick Brook: co-development with LHCb, especially through

UK GridPP collaboration (or rather, I’ll present this later) Alexandre Vaniachine, describing work for the ATLAS Data

Challenges


Data ChallengesData ChallengesTest Bench –Data ChallengesTest Bench –Data Challenges

Prototype IPrototype I May 2002May 2002 Performance and scalability testing of components of the

computing fabric (clusters, disk storage, mass storage system, system installation, system monitoring) using straightforward physics applications. Test job scheduling and data replication software (DataGrid release 1.2)

Prototype IIPrototype II Mar 2003Mar 2003 Prototyping of the integrated local computing fabric, with

emphasis on scaling, reliability and resilience to errors. Performance testing of LHC applications. Distributed application models (DataGrid release 2).

Prototype IIIPrototype III Mar 2004Mar 2004 Full scale testing of the LHC computing model with fabric

management and Grid management software for Tier-0 and Tier-1 centres, with some Tier-2 components (DataGrid release 3).


The Hierarchical ViewThe Hierarchical View

Tier2 Centre ~1 TIPS

Online System

Offline Farm~20 TIPS

CERN Computer Centre >20 TIPS

UK Regional Centre (RAL)

US Regional Centre

French Regional Centre

Italian Regional Centre

SheffieldManchesterLiverpoolLancaster ~0.25TIPS

Workstations

~100 MBytes/sec

~100 MBytes/sec

100 - 1000 Mbits/sec

•One bunch crossing per 25 ns

•100 triggers per second

•Each event is ~1 Mbyte

Physicists work on analysis “channels”

Each institute has ~10 physicists working on one or more channels

Data for these channels should be cached by the institute server

Physics data cache

~PBytes/sec

~ Gbits/sec or Air Freight



~Gbits/sec

Tier Tier 00

Tier Tier 11

Tier Tier 33

Tier Tier 44

1 TIPS = 25,000 SpecInt95

PC (1999) = ~15 SpecInt95

Northern Tier ~1 TIPS

Tier Tier 22


A More Grid-like ModelA More Grid-like Model

CERNTier2

Lab a

Lancs

Lab c

Uni n

Lab m

Lab b

Uni bUni y

Uni x

PhysicsDepartment

Desktop

Germany

Tier 1

USAFermiLab

UK

France

Italy

NL

USABrookhaven

……….

The LHC Computing

Facility


Features of the Cloud ModelFeatures of the Cloud Model

All regional facilities have 1/3 of the full reconstructed dataAll regional facilities have 1/3 of the full reconstructed data Allows more on disk/fast access space, saves tapeAllows more on disk/fast access space, saves tape Multiple copies mean no need for tape backupMultiple copies mean no need for tape backup All regional facilities have all of the analysis data (AOD)All regional facilities have all of the analysis data (AOD) Resource broker can still keep jobs fairly localResource broker can still keep jobs fairly local Centres are Regional and NOT National Centres are Regional and NOT National

Physicists from other Regions should have also Access to the Computing Resources

Cost sharing is an issue Implications for the Grid middleware on accountingImplications for the Grid middleware on accounting

Between experiments Between regions Between analysis groups

Also, different activities will require different prioritiesAlso, different activities will require different priorities


Resource EstimatesResource Estimates

2007Average Luminosity (10^33) 1Trigger Rate (Hz) 160Physics Rate (Hz) 140Running (Equiv. Days) 50Physics Events (10^9) 0.8

1.6 0.5 10 0.1 2 0.5


Resource EstimatesResource Estimates

Analysis resources?:Analysis resources?: 20 analysis groups 20 jobs/group/day = 400 jobs/day sample size : 108 events 2.5 SI95s/ev => 1011 SI95 (s/day) = 1.2*106 SI95 Additional 20% for activities on smaller samples

CPU (MSI95) Tape (PB) Disk (PB)

CERN (T0+T1) 0.4 6.7 0.5Each RF 0.2 0.2 0.46 External RFs 1.2 1.2 2.4Total 1.6 7.9 2.9


Rough ArchitectureRough Architecture

Installation ofSoftware and Env

Compute + Store Sites

User Interface to Grid+ experiment framework

User

MiddlewareRB, GIS

Data Catalogue

Job Configuration/VDC/metadata


Test BedsTest Beds

EDG Test Bed 1EDG Test Bed 1 Common to all LHC experiments Using/testing EDG test bed 1 release code Already running boxed fast simulation and installed full simulation

US ATLAS Test BedUS ATLAS Test Bed Demonstrate success of grid computing model for HEP

in data production in data access in data analysis

Develop & deploy grid middleware and applications wrap layers around apps simplify deployment

Evolve into fully functioning scalable distributed tiered grid NorduGridNorduGrid

Developing a regional test bed Light-weight Grid user interface, working prototypes etc see talk by Aleksandr Konstantinov


EDG Release 1.2EDG Release 1.2

EDG has strong emphasis on middleware development; EDG has strong emphasis on middleware development; applications come secondapplications come second

ATLAS has been testing the `stable’ releases of the EDG ATLAS has been testing the `stable’ releases of the EDG software as they become available as part of WP8 (ATLAS software as they become available as part of WP8 (ATLAS key contact Silvia Resconi)key contact Silvia Resconi)

EDG Release EDG Release ((1.21.2)) is under test by Integration Team people is under test by Integration Team people plus Loose Cannons (experiment independent people) on plus Loose Cannons (experiment independent people) on the development testbed at CERN.the development testbed at CERN.

Standard rStandard requirementsequirements must be must be metmet before the before the ATLAS ATLAS Applications Applications peoplepeople test test a a releaserelease::

1. The development testbed “must” consist of at least 3 sites in 3 different countries ( e.g. CERN, CNAF, RAL )

2. There “must” be a long ( > 24 hours) unattended period with a low error rate ( < 1% of jobs failed )

http://pcatl0a.mi.infn.it/~resconi/validation/valid.html


EDG TestBed 1 StatusEDG TestBed 1 Status28 May 2002 17:0328 May 2002 17:03

Web interface Web interface showing status of showing status of (~400) servers at (~400) servers at testbed 1 sitestestbed 1 sites

5 Main Production Centres5 Main Production Centres


GridPP Sites in Testbed(s)GridPP Sites in Testbed(s)


NorduGrid OverviewNorduGrid Overview

Launched in spring 2001, with Launched in spring 2001, with the aim of creating a Grid the aim of creating a Grid infrastructure in the Nordic infrastructure in the Nordic countriescountries

Partners from Denmark, Partners from Denmark, Norway, Sweden, and FinlandNorway, Sweden, and Finland

Initially the Nordic branch of Initially the Nordic branch of the the EU DataGridEU DataGrid (EDG) (EDG) project testbedproject testbed

Independent developmentsIndependent developments Relies on funding from Relies on funding from

NorduNet2NorduNet2

http://www.nordugrid.org


Lawrence BerkeleyNational Laboratory

BrookhavenNationalLaboratoryIndiana

University

Boston University

ArgonneNationalLaboratory

U Michigan

University ofTexas atArlington

OklahomaUniversity

US -ATLAS testbed launched February 2001

US Grid Test Bed SitesUS Grid Test Bed Sites


US Hardware and DeploymentUS Hardware and Deployment

8 gatekeepers - ANL, BNL, LBNL, BU, IU, UM, OU, UTA8 gatekeepers - ANL, BNL, LBNL, BU, IU, UM, OU, UTA Farms - BNL, LBNL, IU, UTA + Multiple R&D gatekeepersFarms - BNL, LBNL, IU, UTA + Multiple R&D gatekeepers Uniform OS through kickstartUniform OS through kickstart

Running RH 7.2 First stage deploymentFirst stage deployment

Pacman, Globus 2.0b, cernlib (installations) Simple application package

Second stage deploymentSecond stage deployment Magda, Chimera, GDMP… (Grid data management)

Third stageThird stage MC production software + VDC

Many US names mentioned later, thanks also to Craig Tull, Dan Engh, Mark Many US names mentioned later, thanks also to Craig Tull, Dan Engh, Mark SosebeeSosebee


Important ComponentsImportant Components

GridView - simple script tool to monitor status of test bed GridView - simple script tool to monitor status of test bed (Java version being developed)(Java version being developed)

Gripe - unified user accountsGripe - unified user accounts Magda - MAnager for Grid DataMagda - MAnager for Grid Data Pacman - package management and distribution toolPacman - package management and distribution tool Grappa - web portal based on active notebook technologyGrappa - web portal based on active notebook technology


Grid User InterfaceGrid User Interface

Several prototype interfacesSeveral prototype interfaces GRAPPA EDG Nordugrid

Lightweight

Nothing experiment specific Nothing experiment specific GRAT

Line mode (and we will always need to retain line mode!)

Now defining an ATLAS/LHCb joint user interface, GANGANow defining an ATLAS/LHCb joint user interface, GANGA Co-evolution with Grappa Knowledge of experiment OO architecture needed

(Athena/Gaudi)


Interfacing Interfacing Athena/Gaudi Athena/Gaudi to the to the GRIDGRID

Athena/GAUDI Application

GANGA/GrappaGU

IjobOptions/Virtual DataAlgorithms

GRIDServices

HistogramsMonitoringResults

?

?


GRAPPAGRAPPA

Based on XCAT Science Portal, framework for building personal Based on XCAT Science Portal, framework for building personal science portals science portals

A A science portalscience portal is an application-specific Grid portal is an application-specific Grid portal Active notebookActive notebook

HTML pages to describe the features of the notebook and how to use it HTML forms which can be used to launch parameterizable scripts

(transformation) Parameters stored in a sub-notebook (derivation)

Very flexibleVery flexible Jython - access to Java classesJython - access to Java classes

Globus Java CoG kit XCAT XMESSAGES

Not every user has to write scripts Not every user has to write scripts Notebooks can be shared among usersNotebooks can be shared among users

Import/export capabilityShava Smallen, Rob GardnerShava Smallen, Rob Gardner


GRAPPA/XCAT Science Portal ArchitectureGRAPPA/XCAT Science Portal Architecture

Portal Web Server(tomcat server + java servlets)

JythonIntepreter

NotebookDatabase

GSI Authentication

User’s Web Browser

Grid

•Submit Athena jobs to Grid computing elements

•Manage JobOptions, record sessions

•Staging and output collection supported

•Tested on US ATLAS Grid Testbed

The prototype can:


GANGA/Grappa Development GANGA/Grappa Development StrategyStrategy

Completed existing technology + requirement surveyCompleted existing technology + requirement survey Must be Grid aware but not Grid-dependentMust be Grid aware but not Grid-dependent

Still want to be able to `pack and go’ to a standalone laptop

Must be component-basedMust be component-based Interface Technologies (Standards needed Interface Technologies (Standards needed GGF) GGF)

Programmatic API (eg. C, C++, etc) Scripting as Glue ala Stallman (eg. Python) Others eg. SOAP, CORBA, RMI, DCOM, .NET, etc. ………

Defining the experiment software services to capture and Defining the experiment software services to capture and present the functionality of the Grid servicepresent the functionality of the Grid service


Possible DesignsPossible Designs

Two ways of implementation: Two ways of implementation: Based on one of the general-purpose grid portals (not tied to a

single application/framework): Alice Environment (AliEn) Grid Enabled Web eNvironment for Site-Independent User Job

Submission (GENIUS) Grid access portal for physics applications (Grappa)

Based on the concept of Python bus (P. Mato): use different modules whichever are required to provide full

functionality of the interface use Python to glue this modules, i.e., allow interaction and

communication between them


Workspaces

DB

InternetInternet

GAUDI

Python BusPython Bus

Localuser

GaudiPython

Remote userHTML page

GRIDGRID

JobConfiguration

DB

Bookkeeping

DB

ProductionDB

GU

IJava Module OS Module EDG API PythonROOT

PYTHON SW BUS

GAUDI client


Installation ToolsInstallation Tools

To use the Grid, deployable software must be deployed on the Grid fabrics, and the To use the Grid, deployable software must be deployed on the Grid fabrics, and the deployable run-time environment established (Unix and Windows)deployable run-time environment established (Unix and Windows)

Installable code and run-time environment/configurationInstallable code and run-time environment/configuration Both ATLAS and LHCb use CMT for the software management and environment Both ATLAS and LHCb use CMT for the software management and environment

configurationconfiguration CMT knows the package interdependencies and external dependencies CMT knows the package interdependencies and external dependencies this is the this is the

obvious tool to prepare the deployable code and to èxpose’ the dependencies to the obvious tool to prepare the deployable code and to èxpose’ the dependencies to the deployment tool (Christian Arnault, Chas Loomis)deployment tool (Christian Arnault, Chas Loomis)

Grid aware tool to deploy the aboveGrid aware tool to deploy the above

PACMAN (Saul Youssef) is a candidate which seems fairly easy to interface with CMT PACMAN (Saul Youssef) is a candidate which seems fairly easy to interface with CMT


Installation IssuesInstallation Issues

Most Grid projects seem to assume either code is pre-installed or else Most Grid projects seem to assume either code is pre-installed or else can be dumped each time into the input sandboxcan be dumped each time into the input sandbox

The only route for installation of software through the Grid seems to be The only route for installation of software through the Grid seems to be as data in Storage Elementsas data in Storage Elements

In general these are non-local Hard to introduce directory trees etc this way (file based)

How do we advertise installed code?How do we advertise installed code? Check it is installed by a preparation task sent to the remote fabric

before/with the job Advertise the software is installed in your information service for use by the

resource broker Probably need both!Probably need both! The local environment and external packages will always be a problem The local environment and external packages will always be a problem

Points to a virtual machine idea eventually; Java? Options?Options?

DAR – mixed reports, but CMS are interested PACKMAN from AliEn LGCG, OSCAR – not really suitable, more for site management?


CMT and deployable code CMT and deployable code

Christian Arnault and Charles Loomis have a beta-release Christian Arnault and Charles Loomis have a beta-release of CMT that will produce package rpms, which is a large of CMT that will produce package rpms, which is a large step along the waystep along the way

Still need to have minimal dependencies/clean code! Need to make the package dependencies explicit Rpm requires root to install in the system database (but not

for a private installation) Developer and binary installations being produced, Developer and binary installations being produced,

probably needs further refinementprobably needs further refinement Work to expose dependencies as PACMAN cache files Work to expose dependencies as PACMAN cache files

ongoingongoing

Note: much work elsewhere in producing rpms of ATLAS Note: much work elsewhere in producing rpms of ATLAS code, notably in Copenhagen; this effort has the advantage code, notably in Copenhagen; this effort has the advantage of the full dependency knowledge in CMT being exposableof the full dependency knowledge in CMT being exposable


pacmanpacman

Package manager for the grid in development by Saul Package manager for the grid in development by Saul Youssef (Boston U, GriPhyN/iVDGL)Youssef (Boston U, GriPhyN/iVDGL)

Single tool to easily manage installation and environment Single tool to easily manage installation and environment setup for the setup for the longlong list of ATLAS, grid and other software list of ATLAS, grid and other software components needed to ‘Grid-enable’ a sitecomponents needed to ‘Grid-enable’ a site fetch, install, configure, add to login environment, update

Sits over top of (and is compatible with) the many software Sits over top of (and is compatible with) the many software packaging approaches (rpm, tar.gz, etc.)packaging approaches (rpm, tar.gz, etc.)

Uses dependency hierarchy, so one command can drive the Uses dependency hierarchy, so one command can drive the installation of a complete environment of many packagesinstallation of a complete environment of many packages

Packages organized into Packages organized into cachescaches hosted at various sites hosted at various sites How to fetch can be cached rather than the desired object

Includes a web interface (for each cache) as well as Includes a web interface (for each cache) as well as command line toolscommand line tools


# An encryption package needed by Globus# An encryption package needed by Globus##name = ‘SSLeay’name = ‘SSLeay’description = ‘Encryption’description = ‘Encryption’url = ‘http://www.psy.uq.oz.au/~ftp/Crypto/ssleay’url = ‘http://www.psy.uq.oz.au/~ftp/Crypto/ssleay’source = ‘http://www.psy.uq.oz.au/~ftp/Crypto/ssleay’source = ‘http://www.psy.uq.oz.au/~ftp/Crypto/ssleay’systems = { ‘linux-i386’: [‘SSLeay-0.9.0b.tar.gz’,’SSLeay-0.9.0b’],\systems = { ‘linux-i386’: [‘SSLeay-0.9.0b.tar.gz’,’SSLeay-0.9.0b’],\ ‘ ‘linux2’ : [‘SSLeay-0.9.0b.tar.gz’,’SSLeay-0.9.0b’],\linux2’ : [‘SSLeay-0.9.0b.tar.gz’,’SSLeay-0.9.0b’],\ ‘ ‘sunos5’ : [‘SSLeay-0.9.0b.tar.gz’,’SSLeay-0.9.0b’] }sunos5’ : [‘SSLeay-0.9.0b.tar.gz’,’SSLeay-0.9.0b’] }depends = []depends = []exists = [‘/usr/local/bin/perl’]exists = [‘/usr/local/bin/perl’]inpath = [‘gcc’]inpath = [‘gcc’]bins = []bins = []paths = []paths = []enviros = []enviros = []localdoc = ‘README’localdoc = ‘README’daemons = []daemons = []install = { \ install = { \ ‘ ‘root’: [‘./Configure linux-elf’,’make clean’, \root’: [‘./Configure linux-elf’,’make clean’, \ ’ ’make depend’,’make’,’make rehash’,’make test’,’make install’], \make depend’,’make’,’make rehash’,’make test’,’make install’], \ ‘ ‘*’ : [‘./Configure linux-elf’,’make clean’,’make depend’,’make’, \*’ : [‘./Configure linux-elf’,’make clean’,’make depend’,’make’, \ ’ ’make rehash’,’make test’] }make rehash’,’make test’] }


Grid Applications ToolkitGrid Applications Toolkit

Horst Severini, Kaushik De, Ed May, Wensheng Deng, Jerry GHorst Severini, Kaushik De, Ed May, Wensheng Deng, Jerry Gieraltowskiieraltowski..… (US Test Bed)… (US Test Bed)

Repackaged Athena-Atlfast (OO fast detector simulation) for grid testbed Repackaged Athena-Atlfast (OO fast detector simulation) for grid testbed (building on Julian Phillips and UK effort)(building on Julian Phillips and UK effort)

Script 1: can run on any globus enabled node (requires transfer of ~17MB source)

Script 2: runs on machine with packaged software preinstalled on grid site

Script 3: runs on afs enabled sites (latest version of software is used)

Other user toolkit contentsOther user toolkit contents to check status of grid nodes submit jobs (without worrying about underlying middleware or ATLAS

software) uses only basic RSL & globus-url-copy


Monitoring ToolMonitoring Tool

GridView - a simple visualization tool using Globus ToolkitGridView - a simple visualization tool using Globus Toolkit First native Globus application for ATLAS grid (March 2001) Collects information using Globus tools. Archival information is stored in

MySQL server on a different machine. Data published through web server on a third machine.

Plans:Plans: Java versionJava version Better visualizationBetter visualization

Historical plots Hierarchical MDS information Graphical view of system health

New MDS schemasNew MDS schemas Optimize archived variablesOptimize archived variables Publishing historical information through GIIS servers??Publishing historical information through GIIS servers?? Explore discovery toolsExplore discovery tools Explore scalability to large systemsExplore scalability to large systemsPatrick McGuiganPatrick McGuigan



MDS InformationMDS Information

Listing of available object classes


More DetailsMore Details


Data Management Data Management ArchitectureArchitecture

AMI

ATLAS Metatdata Interface

Query LFN

Associated attributes and

values

MAGDA

MAnager MAnager for Grid-for Grid-

based Databased Data

Manage Manage replication, replication,

physical locationphysical location

VDC

Virtual Data

Catalog

Derive and

transform LFNs


MAnager for Grid-based Data (essentially the `replica catalogue’ tool)MAnager for Grid-based Data (essentially the `replica catalogue’ tool) Designed for ‘managed production’ Designed for ‘managed production’ andand ‘chaotic end-user’ usage ‘chaotic end-user’ usage Designed for rapid development of components to support users quickly, with components later Designed for rapid development of components to support users quickly, with components later

replaced by Grid Toolkit elementsreplaced by Grid Toolkit elements Deploy as an evolving production tool and as a testing ground for Grid Toolkit components GDMP will be incorporated

Application in DCsApplication in DCs Logical files can optionally be organized into collections File management in production; replication to BNL; CERN, BNL data access GDMP integration, replication and end-user data access in DC1

DevelopmentsDevelopments Interface with AMI (ATLAS Metadata Interface, allows queries on Logical File Name collections by Interface with AMI (ATLAS Metadata Interface, allows queries on Logical File Name collections by

users, Grenoble project)users, Grenoble project) Interfaces to Virtual Data Catalogue, see AV’s talk)Interfaces to Virtual Data Catalogue, see AV’s talk) Interfacing with hybrid ROOT/RDBMS event storeInterfacing with hybrid ROOT/RDBMS event store Athena (ATLAS offline framework) integration; further grid integrationAthena (ATLAS offline framework) integration; further grid integration

Managing Data -MagdaManaging Data -Magda

Info: http://www.usatlas.bnl.gov/magda/info

Engine: http://www.usatlas.bnl.gov/magda/dyShowMain.pl

T Wenaus, W Deng


Magda ArchitectureMagda Architecture

DB access via perl, C++, java, cgi (perl) scripts; C++ and Java APIs auto-generated off the MySQL DB schema

User interaction via web interface and command lineUser interaction via web interface and command line


ConclusionConclusion

The Grid is the only viable solution to the ATLAS Computing The Grid is the only viable solution to the ATLAS Computing problemproblem

The problems of coherence across the Atlantic are large ATLAS (and CMS etc) are àt the sharp end’, so we will force the

divide to be bridged Many applications have been developed, but need to be Many applications have been developed, but need to be

refined/mergedrefined/merged These revise our requirements – we must use LCG/GGF and any

other forum to ensure the middleware projects satisfy the real needs; this is not a test bed!

The progress so far is impressive and encouragingThe progress so far is impressive and encouraging Good collaborations (especially ATLAS/LHCb)

The real worry is scaling up to the full systemThe real worry is scaling up to the full system Money! Manpower! Diplomacy?!

Documents

ATLAS and the Grid ACAT02 Moscow June 2002 RWL Jones Lancaster University