View
26
Download
3
Category
Tags:
Preview:
DESCRIPTION
Using ICENI to run parameter sweep applications across multiple Grid resources. Case Studies on Grid Applications – GGF10. Murtaza Gulamali Stephen McGough, Steven Newhouse, John Darlington London e-Science Centre Department of Computing, Imperial College London. Contents. The GENIE project - PowerPoint PPT Presentation
Citation preview
Using ICENI to run parameter sweep applications across
multiple Grid resourcesMurtaza Gulamali
Stephen McGough, Steven Newhouse, John Darlington
London e-Science Centre
Department of Computing, Imperial College London
Case Studies on Grid Applications – GGF10
2
Contents
1. The GENIE project
2. The ICENI middleware
3. GENIE as an ICENI application
4. Summary and conclusions
5. Acknowledgements
3
The GENIE projectBackground
Grid ENabled Integrated Earth system model. Investigate long term changes to the Earth’s climate (i.e.
global warming) by integrating numerical models of various components of the Earth system.
3D atmosphere
3D ocean
2D sea ice
AtmosphericCO2
2D land surface
Land biogeochemistry
Ocean biogeochemisty
Ocean sediments
3D ice sheets
Schematic diagram of model framework for GENIE.Courtesy of T. Lenton, CEH Edinburgh, UK.
4
The GENIE projectBackground
Grid ENabled Integrated Earth system model. Investigate long term changes to the Earth’s climate (i.e.
global warming) by integrating numerical models of various components of the Earth system.
Require a Grid infrastructure to:– flexibly couple together components to form a unified Earth
System Model (ESM).– execute the resultant ESM efficiently and accurately.– archive and share the resultant data produced by the model.– provide a high-level open access system to allow a virtual
organisation of Earth System modellers to collaborate.
5
The GENIE projectPrevious scientific work
Investigate the vulnerability of the thermohaline circulation of the world ocean using a prototype model consisting of just 3 coupled components.
Run simulation across two different parameter ranges.
perform 31 31 = 961 individual simulations.
parameter sweep application!
6
The GENIE projectPrevious e-scientific work
Provided Grid infrastructure to support this activity…– flocked Condor pool between three institutions. – web-portal to allow experiment management.– database management system (based on Geodise) to allow
data archiving and retrieval.
Disadvantages of this infrastructure…– firewalls!… between institutions hosting Condor pools.– web-portal not very flexible… model and parameter choices
hard-coded.– true resource brokering not taking place… all compute and
database resources belonging to virtual organisation not utilised.
Solution: use ICENI middleware
7
The ICENI middlewareBackground
IC e-Science Networked Infrastructure. Developed by LeSC Grid Middleware Group. Service oriented Grid middleware. Represents compute, storage and software resources
as services. Services can communicate using standard protocols
(eg. Jini, SOAP, JXTA). ICENI provides an end-to-end middleware consisting of:
– Grid service infrastructure– dynamic service management framework– application toolkit
8
The ICENI middlewareApplication development in ICENI
ICENI uses a component programming model to describe Grid applications.
application development application composition Example: linear equation solver
matrixsource
vectorsource
linear equation
solver
vectorsink
linear equation
solver
Cholsky decomposition
LU decomposition
linear equation
solver
9
The ICENI middlewareApplication development in ICENI
service list composition pane parameters
10
GENIE as an ICENI applicationParameter sweep as component app.
setup component
GENIEbinary
component
archive component
GENIEbinary
component
GENIEbinary
component
splitter component
collator component
11
GENIE as an ICENI applicationExecuting over multiple resources
Beowulf cluster
setup component
GENIEbinary
component
archive component
GENIEbinary
component
resource launcher
resource launcher
Condor pool
splitter component
collator component
12
GENIE as an ICENI applicationResults
Using ICENI, ran 4 GENIE parameter sweep experiments on Beowulf Cluster (using Sun Grid Engine) and Linux PC based Condor pool.– Sun Grid Engine: 481 jobs– Condor pool: 480 jobs– Total: (31 31 =) 961 jobs
Find that ICENI takes ~2 minutes to schedule and submit jobs to both high throughput job managers.
Each experiment took ~5 days to run.
13
Summary and conclusions
Are able to execute GENIE parameter sweep experiments across multiple resources administered by members of virtual organisation.
Execution time same as before but:– Can leverage all the flexibility of a service oriented
Grid middleware.– Can create ICENI Grid based on resources owned
and federated by collaborators in the virtual organisation.
– Don’t have to contend with firewalls… (sort of)
14
Acknowledgements
My co-authors:
– Dr. Stephen McGough, Dr. Steven Newhouse, Prof. John Darlington.
The ICENI development team:
– http://www.lesc.ac.uk/iceni/ The GENIE team:
– http://www.genie.ac.uk/
Recommended