Embed Size (px)
DESCRIPTION
3 What is GENIE? Grid ENabled Integrated Earth system model. Investigate long term changes to the Earth’s climate (i.e. global warming) by integrating numerical models of the Earth system. e-Science aims: Flexibly couple together state-of-the-art components to form unified Earth System Model (ESM). Execute resultant ESM on a Grid infrastructure. Share resultant data produced by simulation runs. Provide high-level open access to the system, creating and supporting virtual organisation of Earth System modellers.
Citation preview
Performance guided scheduling in GENIE through ICENIhttp://www.genie.ac.uk/
2
Contents
1. What is GENIE?2. Previous work – Grid infrastructure3. Limitations of present infrastructure4. Introduction to ICENI5. Performance experiments6. Summary and conclusions7. Future work
3
What is GENIE?
Grid ENabled Integrated Earth system model. Investigate long term changes to the Earth’s climate (i.e.
global warming) by integrating numerical models of the Earth system.
e-Science aims:• Flexibly couple together state-of-the-art components
to form unified Earth System Model (ESM).• Execute resultant ESM on a Grid infrastructure.• Share resultant data produced by simulation runs.• Provide high-level open access to the system,
creating and supporting virtual organisation of Earth System modellers.
4
GENIE model components
G EN IE
www.genie.ac.uk
G rid EN abled Integra tedEarth system m odel
open ocean
marine biota
sea ice
icesheets
land surface and hydro logy
deepocean
terrestrialbiota
atmosphere
soilscoasta l seas
5
Previously in GENIE…Investigating the thermohaline circulation
Investigated influence of freshwater transport upon global ocean circulation
Performed several parameter sweep experiments, each consisting of ~1000 simulations of a GENIE prototype.
Used a Grid infrastructure:i. Portal – create, submit and manage
experiments.ii. Condor pool – execute simulations in
parallel.iii. Database management system – archive
and process resultant data.
6
Previously in GENIE…Scientific achievements
Intensity of the thermohaline circulation as a function of freshwater flux between Atlantic and Pacific oceans and mid-Atlantic and North Atlantic.
Surface air temperature difference between extreme states (off - on) of the thermohaline circulation.
North Atlantic 2C colder when the circulation is off.
New scientific findings papers published!
7
Limitations of Grid infrastructure
GENIE model hard-coded – cannot use alternative models without recoding.
Format of input/output data not flexible. Parameter space being investigated is fixed. True resource brokering not taking place.
Solution:
8
Advantages of ICENI
ICENI Netbeans client allows experiment to be built in a systematic and repeatable way.
Component based programming model provides flexibility and extensibility.
Service oriented architecture allows for true resource brokering and Grid enablement of application.
9
ICENI Binary Component
Allows you to wrap a binary executable as an ICENI component.
binary executable
JDML
stdin
files
stdout
stderr
10
A GENIE experiment as an ICENI application
setup component
GENIEbinary
component
archive component
GENIEbinary
component
GENIEbinary
component
splitter component
collator component
11
A GENIE experiment as an ICENI application
Introduce high-throughput resource launcher…
setup component
archive component
GENIEbinary
component
resource launcher
Sun Grid Engine
Condor pool
Bash
LAU
NC
HE
STO
12
Performance experiments
Ran 8 different types of experiments to evaluate performance of ICENI:
Each experiment run several times in order to obtain an average sojourn time.
# Resource
Job Type
Middleware
1
SolarisBash Non-ICENI
2 ICENI3 Condor Non-ICENI4 ICENI5
LinuxBash Non-ICENI
6 ICENI7 Condor Non-ICENI8 ICENI
Solaris• Shared memory server• 8 900MHz UltraSparc II CPUs• 16Gb memory
Linux:• Beowulf cluster• 16 2GHz Intel Dual Xeon CPUs• 2Gb memory
13
Performance results
2.332.52.44
11.46
2.512.522.49
11.65
0
2
4
6
8
10
12
Bash Condor Bash Condor
Solaris Linux
Resource
Tim
e (h
ours
)
Non-ICENI
ICENI
14
Performance results
2.332.52.44
11.46
2.512.522.49
11.65
0
2
4
6
8
10
12
Bash Condor Bash Condor
Solaris Linux
Resource
Tim
e (h
ours
)
Non-ICENI
ICENI
Under Solaris Condor is ~5 times faster than Bash sequential vs
parallel execution.
15
Performance results
2.332.52.44
11.46
2.512.522.49
11.65
0
2
4
6
8
10
12
Bash Condor Bash Condor
Solaris Linux
Resource
Tim
e (h
ours
)
Non-ICENI
ICENI
Under Linux Condor and Bash
perform similarly only 2 nodes in
Linux Condor pool.
16
Performance results
2.332.52.44
11.46
2.512.522.49
11.65
0
2
4
6
8
10
12
Bash Condor Bash Condor
Solaris Linux
Resource
Tim
e (h
ours
)
Non-ICENI
ICENI
ICENI introduces scheduling and
deployment overhead to the
sojourn time.
17
Summary
Shown how ICENI middleware can be used to launch GENIE experiments.
Performance overhead is insignificant when compared to advantages of using ICENI to deploy jobs.
Can create and schedule ensemble experiments across multiple computational resources using true resource brokering.
18
Future work
Need to repeat experiments with Sun Grid Engine launcher.
Need to incorporate component based GENIE model in experiments.
ICENI is evolving…• Adopt new web services.• Decouple into separate functionalities.• Use GridSAM to launch experiments.
(please visit the LeSC booth for a demo)
19
Acknowledgments
GENIE investigators:Prof. Paul Valdes (Bristol), Prof. John Shepherd (SOC, Southampton), Prof. Andrew Watson (UEA), Prof. Melvyn Cannell (CEH Edinburgh), Dr. Anthony Payne (Bristol), Prof. Richard Harding (CEH Wallingford), Prof. Simon Cox (SReSC), Dr. Steven Newhouse (OMII) and Prof. John Darlington (LeSC).
Recognised researchers:Dr. Stephen McGough (LeSC), Andrew Yool (SOC), Dr. Robert Marsh (SOC), Dr. Timothy Lenton (UEA) and Dr. Neil Edwards (Bern).
