Automated software packaging and installation for the ATLAS experiment Simon George Royal Holloway, University of London Christian Arnault, LAL Orsay;

Automated software Automated software packaging and installationpackaging and installationfor the ATLAS experimentfor the ATLAS experiment

Simon GeorgeSimon GeorgeRoyal Holloway, University of LondonRoyal Holloway, University of London

Christian Arnault, LAL Orsay; Michael Gardner, RHUL; Roger Jones, Christian Arnault, LAL Orsay; Michael Gardner, RHUL; Roger Jones,

University of Lancaster; Saul Youssef, Boston UniversityUniversity of Lancaster; Saul Youssef, Boston University

[email protected]

e-Science All Hands Meetinge-Science All Hands Meeting

NottinghamNottingham

2-4 September 20032-4 September 2003ATLASexperiment.org

Wed 03Sep03Wed 03Sep03 Simon George RHUL Simon George RHUL 22

IntroductionIntroduction This talk is about packaging, distribution and This talk is about packaging, distribution and

installation for a large software projectinstallation for a large software project It is essential becauseIt is essential because

The project computing resources are widely distributed The project computing resources are widely distributed around 140 institutes, who all want to use the softwarearound 140 institutes, who all want to use the software

We want to be able to use Grid resources that do not We want to be able to use Grid resources that do not have locally managed installations of the softwarehave locally managed installations of the software

Our working model also requires the ability to deploy Our working model also requires the ability to deploy user code that is not part of an official distributionuser code that is not part of an official distribution

I’ll describe the process developed and the tools I’ll describe the process developed and the tools used.used.


ContentsContents ATLAS and its softwareATLAS and its software RequirementsRequirements Tools and formatsTools and formats Meta dataMeta data Naming conventionsNaming conventions Creating and installing the kitsCreating and installing the kits Conclusions and outlookConclusions and outlook


The ATLAS ExperimentThe ATLAS Experiment A Particle Physics experiment at A Particle Physics experiment at

the Large Hadron Collider, CERNthe Large Hadron Collider, CERN 1600 physicists, 140 institutes,1600 physicists, 140 institutes,

6 continents6 continents Studies include Studies include

• search for the search for the origin of massorigin of mass• excess of matter over antimatter in excess of matter over antimatter in

the universethe universe• evidence for Supersymmetry evidence for Supersymmetry • other new physicsother new physics


ATLAS software suiteATLAS software suite Simulation, data processing and analysisSimulation, data processing and analysis 500 “packages”, 50 external, inter-dependent.500 “packages”, 50 external, inter-dependent. 100s of developers and 1000s of users in 140 institutes100s of developers and 1000s of users in 140 institutes One release build is 2.5 GB of filesOne release build is 2.5 GB of files It takes 10 hours to buildIt takes 10 hours to build Build types and frequenciesBuild types and frequencies

Production release 3-4 times per yearProduction release 3-4 times per year Developer release every 2-3 weeksDeveloper release every 2-3 weeks Nightly build of snapshotNightly build of snapshot

Build configuration permutationsBuild configuration permutations Optimised, debug and sometimes also profile builds.Optimised, debug and sometimes also profile builds. Two platforms (RedHat 7.3 on Intel x86, Solaris 8 on SPARC)Two platforms (RedHat 7.3 on Intel x86, Solaris 8 on SPARC) One or more compilers (gcc 3.2)One or more compilers (gcc 3.2)

Config. management, build and install handled by CMTConfig. management, build and install handled by CMT So not a trivial task to package, distribute and installSo not a trivial task to package, distribute and install


CMTCMT Configuration management toolConfiguration management tool

Concerned with setting up user’s environment to build Concerned with setting up user’s environment to build and run softwareand run software

Needs help of tools for a large projectNeeds help of tools for a large project CMT helps to define and impose conventionsCMT helps to define and impose conventions

For naming packages, files, directoriesFor naming packages, files, directories For describing their relationshipsFor describing their relationships In other words, package metadataIn other words, package metadata This is the key feature exploited for this project.This is the key feature exploited for this project.

Useful features to manage sub projects, Useful features to manage sub projects, dependenciesdependencies

A broad user base, especially in Particle Physics A broad user base, especially in Particle Physics and Astronomy experiments.and Astronomy experiments.

CMT

www.cmtsite.org


Packaging RequirementsPackaging Requirements Three types of kit requiredThree types of kit required

Binary kitBinary kit• Pre-built executables, libraries and configuration files needed to Pre-built executables, libraries and configuration files needed to

run the softwarerun the software• Used for data challenges, production, basic usersUsed for data challenges, production, basic users

Developer’s kitDeveloper’s kit• Binary kit plusBinary kit plus• Headers, libraries and configuration needed to build against itHeaders, libraries and configuration needed to build against it• For developers and most usersFor developers and most users

Full source kitFull source kit• To rebuild from scratch on binary-incompatible platformsTo rebuild from scratch on binary-incompatible platforms• When local source code browsing is requiredWhen local source code browsing is required

For each permutation of platform, config, compilerFor each permutation of platform, config, compiler


Installation requirementsInstallation requirements For large facilities: unattended, push button deploymentFor large facilities: unattended, push button deployment For normal user: relocateable, no root accessFor normal user: relocateable, no root access Automatic configurationAutomatic configuration Updates, multiple versionsUpdates, multiple versions Avoid duplication and unnecessary downloadsAvoid duplication and unnecessary downloads Possibility to take subset of softwarePossibility to take subset of software Self contained, apart from …Self contained, apart from … Prerequisite software: modest list and automatic checkPrerequisite software: modest list and automatic check Set up user’s environment (e.g. LD_LIBRARY_PATH)Set up user’s environment (e.g. LD_LIBRARY_PATH) Reversible: uninstallReversible: uninstall Install and work disconnected from network, Install and work disconnected from network,

e.g. install onto a laptop from CDse.g. install onto a laptop from CDs


ConstraintsConstraints ATLAS software is divided into sub-projectsATLAS software is divided into sub-projects

Currently ATLAS and GaudiCurrently ATLAS and Gaudi Could be more in the future, e.g. split ATLAS into Could be more in the future, e.g. split ATLAS into

simulation and reconstructionsimulation and reconstruction Each sub-project consists off many packagesEach sub-project consists off many packages

External/Internal package distinctionExternal/Internal package distinction Internal packages are developed and managed within Internal packages are developed and managed within

the ATLAS software projectthe ATLAS software project External packages are the opposite, e.g. software from External packages are the opposite, e.g. software from

the Particle Physics community, public domain software the Particle Physics community, public domain software or commercial products.or commercial products.

Interface packages for externalsInterface packages for externals• Pure metadata packagePure metadata package• Actual external sw can be installed anywhere, any way. Actual external sw can be installed anywhere, any way. • Gives it the outside appearance of an internal packageGives it the outside appearance of an internal package


Constraints, continuedConstraints, continued Existing use of CMTExisting use of CMT

Package structure already in placePackage structure already in place Meta data provided by packages or implied by default Meta data provided by packages or implied by default

policies is already enough for automated packaging.policies is already enough for automated packaging. ProblemsProblems

ATLAS software is written by large communities with a ATLAS software is written by large communities with a mixed level of experiencemixed level of experience

All such software projects will have small flaws All such software projects will have small flaws introduced in each releaseintroduced in each release

These must be worked around when they impact on the These must be worked around when they impact on the packaging.packaging.

For example, one problem of particular relevance to For example, one problem of particular relevance to packaging & installation is cyclic dependenciespackaging & installation is cyclic dependencies


Packaging: starting pointPackaging: starting point One kit per packageOne kit per package

Follow existing granularityFollow existing granularity Separate metadata and payloadSeparate metadata and payload

Two parts to each kitTwo parts to each kit Performed by librarian as integral part of Performed by librarian as integral part of

release procedurerelease procedure Distribution by web or distributed filesystem Distribution by web or distributed filesystem

(e.g. AFS)(e.g. AFS)


Tools usedTools used CMTCMT

Define and impose conventions on packagesDefine and impose conventions on packages Query the metadata needed for packagingQuery the metadata needed for packaging

PacmanPacman Metadata formatMetadata format Tool used to manage kit installationTool used to manage kit installation

Tar and RPMTar and RPM Payload format – the package itselfPayload format – the package itself

““Deployment tools” shell scriptsDeployment tools” shell scripts Construct the kits using CMTConstruct the kits using CMT Control location of Pacman cache and distributionControl location of Pacman cache and distribution Post-installation configurationPost-installation configuration


Overview of process and toolsOverview of process and tools

Create kitsWeb server or AFS

Local computers

CMT

CMTPacman

DeveloperLocal s/w manager

Librarian

Deployment Tools

Fetch packages

Install and configureConfigure

environment

Compile and

link own library

Run software


PacmanPacman A package manager A package manager Packager defines how the software should be Packager defines how the software should be

fetched, installed, configured, updated, in a fetched, installed, configured, updated, in a “Pacman” file. The package itself can be in any “Pacman” file. The package itself can be in any format as that file is separate.format as that file is separate.

A directory of these files is known as a cache, A directory of these files is known as a cache, usually available on the web.usually available on the web.

Pacman tool is used to install the softwarePacman tool is used to install the software Pacman’s feature list is a good match to the Pacman’s feature list is a good match to the

requirements for installation.requirements for installation. Already used by several Particle Physics and Already used by several Particle Physics and

GRID projects.GRID projects.

http

://p

hysi

cs.b

u.ed

u/~

yous

sef/

pacm

an


Package distribution formatPackage distribution format Tar vs. RPMTar vs. RPM Both can be made relocateableBoth can be made relocateable Feature setFeature set

Tar has a simple feature set but is complementary to CMT and Tar has a simple feature set but is complementary to CMT and PacmanPacman

RPM overlaps with CMT and PacmanRPM overlaps with CMT and Pacman• e.g. RPM also handles dependencies and prerequisitese.g. RPM also handles dependencies and prerequisites

PlatformsPlatforms RPM is only widely used on Linux, while tar is standard on pretty RPM is only widely used on Linux, while tar is standard on pretty

much any Unixmuch any Unix AnnoyancesAnnoyances

Default RPM database needs root access to write to itDefault RPM database needs root access to write to it• There are workarounds for this but not prettyThere are workarounds for this but not pretty

ConclusionConclusion Decided to use tarDecided to use tar but retained RPM as an optionbut retained RPM as an option


Meta dataMeta data For each packageFor each package

Other packages it uses (dependencies)Other packages it uses (dependencies) Location of constituentsLocation of constituents

• Applications and librariesApplications and libraries• Header filesHeader files• Run time/config filesRun time/config files• CMT requirements fileCMT requirements file

External packagesExternal packages Pure meta data “glue” packagesPure meta data “glue” packages Just define paths to exportJust define paths to export

All defined in CMT requirements filesAll defined in CMT requirements files or implied by default conventions of ATLASor implied by default conventions of ATLAS

Can be queried through cmtCan be queried through cmt cmt show usescmt show uses cmt show macro <package>_export_pathscmt show macro <package>_export_paths


Naming and structure Naming and structure Package naming conventionPackage naming convention

Packages in a sub-projectPackages in a sub-project• <package name>-<sub-project release id><package name>-<sub-project release id>

External packagesExternal packages• <package name>-<version id><package name>-<version id>

These names are used when expressing the inter-package These names are used when expressing the inter-package dependenciesdependencies

Directory structure within each kitDirectory structure within each kit <sub-project>/<release-id>/InstallArea/<sub-project>/<release-id>/InstallArea/

• contains the sub-directories bin, lib, include, share.contains the sub-directories bin, lib, include, share. <sub-project>/<release-id>/<package>/<version>/cmt/<sub-project>/<release-id>/<package>/<version>/cmt/

• Contains the configuration management filesContains the configuration management files <external-package>/<external-package>/

• Assumed to have their own internal structure for versions & buildsAssumed to have their own internal structure for versions & builds This is designed to support coexistence of:This is designed to support coexistence of:

Different versions of every piece of softwareDifferent versions of every piece of software Different binary versions (platform and build config) Different binary versions (platform and build config)


ExamplesExamplesCMT requirements file:package ExamplePkgAauthor A. Person <[email protected]>use ExamplePkgBuse ExampleExtPkglibrary ExamplePkgA *.cxxapply pattern component_libraryapply pattern declare_runtime

Package name and author

Inter-package dependencies

Instruction to build a library from source files

Type of library to build, implies library file namesDefault location implied

Pacman file:description=‘Package ExamplePkgA-01-07-02 in release 6.5.0’url=‘http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO’source=‘../dist’download = { ‘*’:’ExamplePkgA-6.5.0.tar.gz’ }depends = [ ‘ExamplePkgB-6.5.0’, ‘ExampleExtPkg-v1’ ]


Creating the kitsCreating the kits First, build a releaseFirst, build a release Discover cycles in the dependenciesDiscover cycles in the dependencies

Use a feature of CMT to discover cycles in the dependencies, as Use a feature of CMT to discover cycles in the dependencies, as these must not be propagated to the kits. Record the output in a these must not be propagated to the kits. Record the output in a file.file.

Then, use a feature of CMT to visit every package in a Then, use a feature of CMT to visit every package in a dependency tree and apply a command theredependency tree and apply a command there

cmt broadcast <command>cmt broadcast <command> Usage of the script to create a kit:Usage of the script to create a kit:

create_kit.sh –release <release-id> -cycles <file> [-rpm] create_kit.sh –release <release-id> -cycles <file> [-rpm] <target distribution directory><target distribution directory>

Creates a pacman file and tar file, optional RPM fileCreates a pacman file and tar file, optional RPM file Finally, there are often a few things to fix by hand specific Finally, there are often a few things to fix by hand specific

to each release.to each release. Note that CMT itself is included as a kitNote that CMT itself is included as a kit


InstallationInstallation Performed by site software manager or end user on Performed by site software manager or end user on

desktop or laptopdesktop or laptop Straightforward procedure:Straightforward procedure:

Install Pacman, if not already doneInstall Pacman, if not already done Install prerequisite softwareInstall prerequisite software

• Currently just RedHat 7.3 o/s, gcc-3.2 and Java SDK 1.4.1Currently just RedHat 7.3 o/s, gcc-3.2 and Java SDK 1.4.1 Choose directory for the installationChoose directory for the installation

• Probably the same as beforeProbably the same as before Choose which release to installChoose which release to install

• Available releases are listed on a web pageAvailable releases are listed on a web page Use Pacman to download, install and configure it, e.g.Use Pacman to download, install and configure it, e.g.pacman –get ATLAS:AtlasRelease-6.5.0pacman –get ATLAS:AtlasRelease-6.5.0

• Dependencies followed automatically to get everything you needDependencies followed automatically to get everything you need Optionally, run script to set up a user environment and run a testOptionally, run script to set up a user environment and run a test

User configures software in the usual wayUser configures software in the usual way Just choose release and private working area as normalJust choose release and private working area as normal Run a setup script provided by CMTRun a setup script provided by CMT


ConclusionsConclusions Procedures and tools have been developed for the Procedures and tools have been developed for the

packaging, distribution and installation of ATLAS softwarepackaging, distribution and installation of ATLAS software Based on Pacman, CMT, tar/rpm and some shell scriptsBased on Pacman, CMT, tar/rpm and some shell scripts The basic principles could be applied more generallyThe basic principles could be applied more generally

Using some or all of the same toolsUsing some or all of the same tools It satisfies most of the requirements for run-time and It satisfies most of the requirements for run-time and

developers’ kits and for installation.developers’ kits and for installation. Full source kit still to be done.Full source kit still to be done.

Early adopters have given useful feedback and it is now Early adopters have given useful feedback and it is now being imported into Grid production systemsbeing imported into Grid production systems

Must now move to its use as part of the standard release Must now move to its use as part of the standard release procedure in ATLASprocedure in ATLAS

by December 2003, for our global `Data Challenge 2'by December 2003, for our global `Data Challenge 2'


Future developmentsFuture developments Better handling of prerequisite software and Better handling of prerequisite software and

platform compatibility checksplatform compatibility checks EDG WP4 configuration management taskEDG WP4 configuration management task

Potential to work with an installation on Potential to work with an installation on demand mechanism for GRID farmsdemand mechanism for GRID farms

LCG/EDG/iVDGL GLUELCG/EDG/iVDGL GLUE Meta packaging proposal for Grid middleware Meta packaging proposal for Grid middleware

and applications, and applications, O. Barring et al.O. Barring et al. Pacman version 3Pacman version 3

Documents

Automated software packaging and installation for the ATLAS experiment Simon George Royal Holloway, University of London Christian Arnault, LAL Orsay;