Upload
cecil-thomas
View
217
Download
2
Embed Size (px)
Citation preview
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
CVS setup at CC-IN2P3 and Datagrid edg-build tools
CVS management, centralized code checking and automatic building of rpm packages from CVS
Yannick Patois
E-mail [email protected]
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
Table Of Content
CVS service at CC-IN2P3 (Loïc Tortay)
How it started
Authorized user access in read/write setup
Anonymous access setup
CVSweb
Datagrid CVS tools (edg-build)
General idea
Code checking
RPM building
Package publishing
RPM distribution management
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
CVS service at CC-IN2P3 - starting point
To share code development among local users:
CVS repository on AFS
But a project (Datagrid) needed a CVS repository open to more people
Lab policy wouldn't allow local login for everyone
Needed another solution
Once created and tested for Datagrid, this CVS service was extender to other experiments: cvs.in2p3.fr was created.
The presented setup has been set in place by Loïc Tortay at CC-IN2P3.
CVS.IN2P3.FR
Dedicated authentication mechanism (not part of the generic users account policy)
Offer open CVS services to experiements.
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
CVS at CC-IN2P3 - Read/Write access setup
Authentication by ssh keys
Interactive login is disabled
CVS operations are performed through a specific pseudo-shell that only allow CVS operations to be executed.
rcvssh : remote CVS shell
Build from 'anoncvssh' (OpenBSD project)
Authentication managed above (by sshd)
The shell ensure that the command and arguments are in a predefined set of allowed commands (cvs commands in our case)
Can chroot the user to a subtree (anonymous access)
Verbose login facility
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
CVS setup at CC-IN2P3 - Anonymous access
Anonymous access is done on a chrooted cvs server
Use a different port than the authenticated access mechanism
The dedicated ssh daemon uses specifics configurations parameters (allow password authentication, allow empty password, only one account allowed)
Picture taken from http://www.winternet.com/~mikelr/ with kind courtesy of Mike Reed.
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
CVS setup at CC-IN2P3 - CVSweb and other tools
CVSweb setup
Module based configuration
Web access aliased cvs.in2p3.fr/experiment
Apache access rights can be managed by experiment
CVS commit info on mailing list
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
CVS Services at CCIN2P3
cvs@in2p3 Service just started a few weeks ago
A few experiments (EROS, SUPERNOVAE, ..) uses it
Datagrid CVS repository
Around 120 accounts
360Mb of sources
A few hundreds connexions per day (more than one thousand if accounting for automatic connexion -like autobuild)
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
Autobuild tools: The Datagrid collaboration
European grid project
A Grid for HEP Biomedical applications (genomic
and imaging) Earth study (ESA)
Constraint
Distributed development
Several dependent modules (around 30)
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
The Datagrid Software Repository The Datagrid Software
Repository
Hosted at CC-IN2P3 (Lyon)CNRS - France
Main services: Source code management for
developers (CVS) Packages distribution (RPMS) The autobuild an check system
(automatic building of packages) Packages distribution
Management
http://datagrid.in2p3.fr/autobuild/rh6.2/http://datagrid.in2p3.fr/autobuild/rh6.2/
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
Autobuild system - generalities
Autobuild
Around 3000 SLOC of python
One external Java program called
A lot of os.system() calls to various unix utilities (from mkdir to cvs)
Context
Run on RH6.2 and RH7.2, have been tested on Solaris
CVS and package repository machine is remote (btw it's an AIX machine).
Publication Machine IBM-AIX
CVS repositoryPackages (rpm) repository
Publication Machine IBM-AIX
CVS repositoryPackages (rpm) repository
Build Machine RedHat Linux
Building, package managing
Build Machine RedHat Linux
Building, package managing CVS checkoutScp
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
Autobuild system : what it does
Statistical informations about the project
RPMS number/size
SLOCs for every edg code
Auto-documentation pages
Processed user doc (LaTeX files, ...)
Autogenerated code doc (Doxygen...)
Build platform informations
Installed rpms, environments...
CVS module processing for each module
Conformance check README and INSTALL files CVS tags
Build autotools make make doc make check make install make dist make rpm
Publish build RPMs
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
Autobuild : result sample
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
Edg-release rpm lists
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
Autobuild present and futur state
Autobuild program (called edg-build) can be downloaded from http://datagrid.in2p3.fr
The code is most likely still very specific to Datagrid own needs, but is improving.
Current evolution plan
Build on demand
Email to developers on build failure
Package managing capability (partially implemented)
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
Creating RPM lists from dependencies I (idea)
Problem : Managing huge lists of RPM (edg-release) can be difficult.
Idea : Creating them from the RPM inter-dependenciesPractical implementation
Datagrid RPMS Striped RPMS (only headers)
Custom “everything” RPM database
Build Machine
Local copyLocal copy
RPM task(virtual rpm pkgs)
RPM task(virtual rpm pkgs)
Full RPM listFull RPM list
--whatprovide--whatrequire
Customlist RPM database
Rpm -Va
ValidationValidation
Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1
Creating RPM List from Dependencies II (example)
Redhat-base-edg-0.0.1-1.i386.rpmredhat-kernel-edg-0.0.1-1.i386.rpm
task-build-wp6-0.0.1-1.i386.rpm
Redhat-base-edg-0.0.1-1.i386.rpmredhat-kernel-edg-0.0.1-1.i386.rpm
task-build-wp6-0.0.1-1.i386.rpm
Name: task-build-wp6[...]
# This is for edg-exampleRequires: libtool
Requires: automakeRequires: autoconf
Requires: makeRequires: c++
Requires: doxygenRequires: sgml-toolsRequires: tetex-dvipsRequires: ghostscript
[...]Full RPM ListFull RPM List