Upload
alden-michaelson
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
The Ibis Project:Simplifying Grid Programming &
Deployment
Henri [email protected]
Vrije Universiteit Amsterdam
The ‘Promise of the Grid’
Efficient and transparent (i.e. easy-to-use) wall-socket computing over a distributed set of resources [Sunderam ICCS’2004, based on Foster/Kesselman]
● Performance & scalability
● Heterogeneous● Low-level & changing
programming interfaces
● writing & deploying grid applications is hard
Reality: ‘Problems of the Grid’
● Connectivity issues
● Fault tolerance
● Malleability
Wide-Area Grid Systems
UseUserr
!
The Ibis Project
● Goal:● drastically simplify grid
programming/deployment
● write and go!
Approach
● Minimal assumptions about execution environment● Virtual Machines (Java) deal with heterogeneity
● Use middleware-independent APIs
● Different programming abstractions
● Designed for dynamic/hostile grid environments
● Modular and flexible: can replace Ibis components by external ones
Global picture
Grid programming
● Programming models:● Low-level message passing (RMI, MPJ)
● High-level divide-and-conquer (Satin)
● IPL (Ibis Portability Layer)● Java-centric “run-anywhere” library, sent with
application● Can handle fault-tolerance, malleability
● SmartSockets● Solve connectivity problems automatically
(firewalls, NAT, addressing problems)
Grid deployment
● Zorilla P2P system● Jobs management, gossiping,
clustering, flood scheduling
● JavaGAT: Java Grid Application Toolkit● Make applications independent of underlying grid
● Used for file copying, resource discovery, job submission & monitoring, user authentication
● API is currently standardized (SAGA)
Java GAT
GAT Engine
Remote
FilesMonitoring
Info
service
Resource
Management
GridLab Globus Unicore SSH P2P LocalGAT
Grid ApplicationFile.copy(...)
submitJob(...)
gridftp globus
Intelligentdispatching
[van Nieuwpoort et al., SC’07 ]
Ibis applications● e-Science (VL-e)
● Brain MEG-imaging● Mass spectroscopy
● Multimedia content analysis, video processing
● Various parallel applications● SAT-solver● N-body● Grammar learning● …
European users
● D-Grid: Workflow engine for astronomy● U. Erlangen: grid file system● INRIA: ProActive on Ibis RMI● U. Patras: Jylab scientific computing
system● HITACHI: Peta-scale data management
DAS-3272 nodes
(AMD Opterons)
792 cores1TB memoryLAN: Myrinet 10G Gigabit
EthernetWAN
(StarPlane): 20-40 Gb/s
OPN
See CCGrid’08 session 11 (Verstoep)
Gene sequence comparison in Satin (on
DAS-3)
Speedup on 1 cluster Run times on 5 clusters
• Divide&conquer scales much better than master-worker
• 78% efficiency on 5 clusters (with 1462 WAN-msgs/sec)
Multimedia content analysis
● Analyzes video streams to recognize objects
● Extract feature vectors from images● Describe properties (color, shape)● Data-parallel task implemented with C+
+/MPI
● Compute on consecutive images● Task-parallelism on a grid
See CCGrid’08 SCALE challenge & sessions 12+16
High Resolution Video Processing
● Realtime processing of CineGrid movie data● 3840x2160 (4xHD) @ 30 fps = 1424 MB/sec
● Multi-cluster processing pipeline ● Using DAS-3, StarPlane and Ibis
Summary● Goal: Simplify grid
programming/deployment
● Key ideas in Ibis● Virtual machines (JVM) deal with
heterogeneity● High-level programming abstractions
(Satin)● Handle fault-tolerance, malleability,
connectivity problems automatically (Satin, SmartSockets)
● Middleware-independent APIs (JavaGAT)● Modular
Acknowledgements Current members Rob van Nieuwpoort Jason Maassen Thilo Kielmann Frank Seinstra Niels Drost Ceriel Jacobs Kees Verstoep
Roelof Kemp Kees van Reeuwijk
Past members John Romein Gosia Wrzesinska Rutger Hofman Maik Nijhuis Olivier Aumage Fabrice Huet Alexandre Denis
More information● Ibis can be downloaded from
● http://www.cs.vu.nl/ibis
● Papers:● Satin [PPoPP’07], SmartSockets [HPDC’07],
Gossiping [HPDC’07], JavaGAT [SC’07],MMCA [IEEE Multimedia’07]
• Ibis tutorials• CCGrid’08