Upload
rolf
View
39
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Grid-Powered Scientific & Engineering Applications. Ho Quoc Thuan. INSTITUTE OF HIGH PERFORMANCE COMPUTING. Agenda. Introduction Grid@IHPC Real-life Applications Future Work. Introduction. Grid computing: - PowerPoint PPT Presentation
Citation preview
Grid-Powered Scientific & Engineering Applications
Ho Quoc Thuan
INSTITUTE OF HIGH PERFORMANCE COMPUTINGINSTITUTE OF HIGH PERFORMANCE COMPUTING
Agenda
• Introduction• Grid@IHPC• Real-life Applications• Future Work
Introduction
• Grid computing:– Flexible, secure, coordinated resource sharing among
dynamic collections of individuals, institutions, and resource.
(From “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”)
• Benefits: resource sharing• Issues:
– Connection– Dynamic environment– Heterogeneity– Security– Management
Grid@IHPC
• To Grid-enable scientific computing focusing on end-to-end solutions– Developing Grid computing middleware:
• Information service• Execution management• Data management• User interfaces …
– Grid-enabling applications:• Parallelize applications• Integrate applications with Grid middleware• Deploy applications• Supply resources and execute applications …
Grid@IHPC
GUI
Portal
Command
Components Integration Interface
Data Grid
Multimedia Grid
Access Grid
Infrastructure – IT Services
Compute Grid
Info. Grid
Information Service
Execution Management
Data Management
Multimedia Management
Deployment Service
APIs
Workflow
VO Information
Hub
Information Service
Allow users to discover & select resources that meet their needs.
– Capturing/publishing/supporting queries
– Large-scale information collection
– Able to publish any types of information:
• Hardware• Software• Workload...
– Hot plugable sensors
…
End users
…
SiteInformation
Service
Info Agents & Info Sensors
Site XResources
…
Grid VO
Info Agents & Info Sensors
Site YResources
SiteInformation
Service
Execution Management Framework
Execution Service
LRM
Execution Service
LRM
Metascheduler
Execution Service
Client Client Client
Grid VOInformation Hub
Site IS
Site IS
Sensor
Sensor Sensor Sensor
Sensor Sensor
dynamic discoveryand matching
reservation
LRM integration
• Distribute job across VO resources• Integrated with IS for dynamic resource
discovery & matching• Integrated with DS for file movement• Assist clients to reserve resources using
access token• Allow workload optimization plug-in• Job Execution Service component at
resource layer for queue management & multiple job submission
• Integration with LRM (SGE, Torque, LSF)
• GT4 and DRMAA compliant• Sequential, simultaneous and MPI jobs• Allow hybrid execution modes
Handle-driven
Forwarding
Hybrid
Data/Software Management• Automatic indexing, file change
notification• Supporting search and application
mapping (search-found-process)
Data Component
Data Manager
MetadataStore
Distributed Storage (1)
Client
ReplicaCatalog (RLI)
File Sensors
Monitored Directory
Replica Catalog (LRC)
Content Store
File To Application
Mapping
OGSA-DAI
OGSA-DAI
Files Files
XQueryUpdate Metadata Store
LFN of files with matching keywords
Con
tent
Inde
Query by Keyword
Metadata Information (XML)
XQueryMetadata & Appln mapping info
Query with LFNList of PFN
Insert LFN
Distributed Storage (N)
...
Multi-modal GUI
XMLForm
AutomatiAutomaticc
FormFormGeneratoGenerato
rr
ManPage
Usereditable
User User CustomizatioCustomizatio
nsns
Command Line Application
Rendered inRuntime
DynamicDynamicFormForm
RendererRenderer
Grid Portal• Remote access to Grid resource
via browsers• Submit batch job• Monitor job status• Download data• Secure access
Workflow Editor
• Grid-based PSE for Engineering of Material• Target complex scientific workflows• Integrate with Grid middleware
Virtualization & Resource Management
• Apply virtualization technologies (Xen, VMWare)
• Package scientific tools in virtual machine images for different applications (template-based)
• Quick deployment• Easy backup/recovery/migrate• OSs as resources
Grid Middleware
SUN GRID ENGINE
MPI
VM1 VM2 VM3 VMn…
Hardware Resources
vm
vm
vm
vm
vm
Software/VMRepository Configurations
UserWorkspace
provisioning customization use and accounting
Xen/VMWare
discovery selection deployment adaptation
End-to-End Solution
Interfaces
Upload, download & search data
Submit, monitor jobs
View/download results
Middleware
Capture, monitor, publish
information
Extract information, transfer data
Schedule jobs
Resources
Capture information
Store data
Execute jobs
Construct & run applications
Real-life Applications
• Fire Dynamic Simulation (FDS)• Multiphysics (Femlab and Comsol)• Computational Electromagnetics (GECEM)• Motif Finder (GIS)• Digital Media Grid (Adaptive Enterprise@Singapore)
Grid-enabled Fire Dynamic Simulation
• Fire & smoke simulation• Compute intensive applications• Data intensive (output)
Portal
Execution Service
LRM
Execution Service
Sensor
Sensor Sensor Sensor
Metascheduler
high bandwidth and low latency
network
Grid-enabled Multiphysics• Grid-enabled commercial software
(Femlab, Comsol)• Resource discovery & execution• Different models of sharing:
1. Sharing license2. Sharing licence & computational power3. Sharing desktop
License ServerUser Computing
Servers
jobs
(2)
License ServerUser
license
(1)
licenses
License ServerUser Desktop
Servers
desktop
(3)
licenses
(2)
Computational ElectromagneticsGrid-enabled Computational Electromagnetics (GECEM)
• Grid infrastructure• Grid middleware• Portal• Remote visualization• Security
Other locations
Geometry dataUWS
IHPC
BAE SYSTEMS
Create geometry
Generate mesh
CEM simulation
MeshOutput Output
Output
Partners:• BAE SYSTEMS• HP UK• University of Swansea• Cardiff University
Grid-enabled Motif Finder
• Multi-modal GUI• Session management
Digital Media Grid Project(IHPC, IDA, HP, NTU, SMU)
Torque
SGE
Portal
IHPC Meta-schedulerJESSubmit job
SOAPJES
JES
LSF Meta-scheduler LSF GridSubmit jobweb browser
Maya
submitiMac
sshhttp
ssh
animatorWeb service
IHPC IS
Future Work
• Build a powerful platform to support science & engineering applications – A*STAR Digital Nervous System (ADNS)– Share resources for inter-research institutes
across A*STAR– Provide software as utility services
Status Updates2005
• Information Service:– Sensors– Data Collection Server– Grid/Web Service
• Execution Management:– Execution Service Container
can execute:• Sequential jobs• Concurrent jobs
– Metascheduler:• Resource selection
based on workload and speed.
• Grid/Web service• DRMAA standard
• Data Service:• Sending data over
Grid/Web service protocol
• Grid Portal• Applications:
– In house codes– Commercial software
2006• Information Service:
– Multi-domain– P2P– Hot-plugable sensors
• Execution Management:– Resource reservation for
parallel/interactive applications– Interface to Local Resource
Managements:• SGE, LSF, Torque
• Data Service:– Reliable data transfer– Transfer/Stream data in parallel
• Data management• Multi-modal GUI• Workflow editor• Virtualization technologies• Applications:
– 18 GB of data– More than 30 hours of
processing time– Windows applications
• Start design and implement ADNS
2007-2008
ADNS in action