Upload
penha
View
35
Download
0
Embed Size (px)
DESCRIPTION
Overview of GridSAM and its Deployment. Steve Crouch, OMII-UK [email protected] 29/07/09. Contents. What is GridSAM and what does it offer? For resource owners and end users Example scenario Packaging options The job submission pipeline Open development Installation and configuration - PowerPoint PPT Presentation
Citation preview
Web: www.omii.ac.uk Email: [email protected]
Overview of GridSAM and its Deployment
Steve Crouch, [email protected]
29/07/09
Web: www.omii.ac.uk Email: [email protected]
Contents
• What is GridSAM and what does it offer?o For resource owners and end users
• Example scenario• Packaging options• The job submission pipeline• Open development• Installation and configuration• Future developments
Web: www.omii.ac.uk Email: [email protected]
GridSAM Overview
• What is GridSAM to the resource owners?o A web service to uniformly expose a computational
resource• Condor• Portable Batch Scheduler (PBS)• Globus• Sun GridEngine• Platform Load Sharing Facility (LSF)• Single machine through Fork or SSH
o Acts as a client to these resources• What is GridSAM to end-users?
o A means to access computational resources in an open standards-based uniform way
o A set of end-user command-line tools and client-side APIs to interact with GridSAM Web Services
• Submit and monitor compute jobs• Cross-protocol file transfer (gsiftp, ftp, sftp, WebDav, http, https,
soon SRB, iRODS) via Commons-VFS (http://sourceforge.net/projects/commonsvfsgrid)
Web: www.omii.ac.uk Email: [email protected]
For Resource Owners…
ComputationalResourceManager
ComputationalResourceManager
DRMDRM
…
One of:PBS (Torque/OpenPBS/PBSPro)LSF, Condor, Sun GridEngine, Globus, Fork
GridSAMService
GridSAMService
X509certificate
Linux Many flavours:RHEL 3,4,5, Fedora 7,8, Scientific Linux 4Java: JDK 1.5.0+
Linux + JavaLinux + Java
Tomcat/AxisTomcat/Axis
Tomcat: 5.0.23, 5.0.28, 5.5.23Axis: v1.2.1
Persistence provided by one of:Hypersonic, PostgreSQL, or existing MySQL
Web: www.omii.ac.uk Email: [email protected]
For End-Users…
GridSAMService
GridSAMService
JSDL
HTTPS/HTTP
WS-Security:X509User/Password
Service InterfaceService Interface
Any/all of:GridSAM native interface, OGSA-BES v1.0, HPC Basic Profile v1.0
…
MyProxy (for Globus/GridFTP)
MyProxy (for Globus/GridFTP)
Globus-
styl
e
Proxy
Certif
icat
e
+ MyProxycredentials
Windows/Linux + Java
Windows/Linux + Java
GridSAM ClientGridSAM Client
Many flavours:RHEL 3,4,5, Fedora 7,8, Debian, Ubuntu, Scientific Linux 4, Windows XP, Windows VistaJava: JDK 1.5.0+
AxisAxisX509certificate
Generic BES/HPC Basic ProfileClient
Generic BES/HPC Basic ProfileClient
OSGA-BESOSGA-BESHPC Basic Profile
HPC Basic Profile
Web: www.omii.ac.uk Email: [email protected]
Example Scenario - CPOSS ENGAGE Project
CondorCondor …
UCL Legion (PBS Torque + Moab)
UCL Legion (PBS Torque + Moab)
…
UK NGSUK NGS
…
Job Managemen
tSystem
Job Managemen
tSystem
Workflow
WorkflowPortalPortal
GridSAM
GridSAM
GridSAMGridSAM
ActiveBPEL
ActiveBPEL
Web: www.omii.ac.uk Email: [email protected]
Package Options: CGT
• Campus Grid Toolkit (CGT) – automated client or server install
• Client/server solution that containso Apache Tomcat/Axis/WSS4j (WS-Security)o GridSAMo Optional - Application Hosting Environment (AHE)o PostgreSQL or HyperSonic database
• Easy, quick installation via further autoconfiguration:o Service:
• Initial security setup for testing• Hooks into existing computational resources
o Client:• Downloadable from existing CGT server and pre-configured for that
server• End-to-end, ‘out of the box’ working examples
• Currently version 1.1.3 (with GridSAM 2.1.6)• Installation videos – see http://www.omii.ac.uk/wiki/CGT
Web: www.omii.ac.uk Email: [email protected]
Package Options: Development Kit
• Development Kit – heavily assisted client or server installation
• Service comprised of:o Web Services Container: Tomcat/Axis with optional WS-
Securityo Data persistence: readily-installable PostgreSQL or
HyperSonic databases, or pre-installed MySQLo Configurable Distributed Resource Manager (DRM) system
• Client essentially ‘container’ of libraries and security configuration
• Optionally downloadable and installable components for client and server:
o GridSAM: Job submission and monitoringo AHE: Application Hosting Environmento Grimoires: UDDI + metadata registryo …
• Currently version 3.4.4
Web: www.omii.ac.uk Email: [email protected]
GridSAM e.g. with Condor
• A staged event-driven architectureo Submission pipeline is constructed as a network of
stages connected by event queueso Each stage performs a specific action upon incoming
events
Example Pipeline: Condor
Web: www.omii.ac.uk Email: [email protected]
Open Community Development
• GridSAM is Open Source, Open Community Development
• GridSAM SourceForge project:
o 99.03% activity, 1 release/month
o SVN source code repositoryo Developer & discuss
mailing lists
http://sourceforge.net/projects/gridsam/
Web: www.omii.ac.uk Email: [email protected]
Installation Process – e.g. Development Kit• Pre-requisites:
o Download omii-server-3.4.4.tar from OMII-UK website (Software -> Downloads) & unpack
o Ensure JDK 1.5.0+ is installed & JAVA_HOME set & Java on PATH
o e.g. for PBS• Ensure submit and monitor PBS commands available on PATH• Installable on PBS server node, or elsewhere with ssh access to PBS
server node
• cd to omii-server-3.4.4 unpacked directory and run ./OMIIstackInstall.pl once each for:
1. Database install: provide PostgreSQL version, install location & port
2. Tomcat/Axis container: Tomcat version, install location, database port, test certificate details
3. Database setup for optional components4. Optional components: select GridSAM only
• Install complete
Web: www.omii.ac.uk Email: [email protected]
Config. Process – e.g. Development Kit• For example, for PBS (any flavour):
o GridSAM is a PBS client for submission and monitoring
o Edit usermap_config.xml and supply DN -> local user account mappings
o Edit batch_config.xml for local PBS configuration• PBS batch queue name to use• User with sudo privileges to enable job submission as
submitting usero Ensure PBS qstat, pbsnodes, qnodes, qsub on PATH
when starting containero Restart container
• For CGT, you specify install requirements in an easily changed specification file prior to installation
Web: www.omii.ac.uk Email: [email protected]
Planned Future Developments
• For end-users:o Refactored documentation and improved OGF
standards coverageo Full support for JSDL DataStaging credential
specificationo Full support for JSDL Resource selection across PBS,
Condor & Fork DRMso JSDL Parameter Sweep Extensiono Support for SRB and iRODS
• For resource owners:o Packaging option as a standalone, manually
configurable web archive (WAR) fileo Job submission through multiple remote ssh accounts
• Direct PBS deployment throughout NGS sites