Embed Size (px)
DESCRIPTION
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002 Biomedical Application –Input: 3-D model of arteries –Simulation: LB of blood flow –Results: in a virtual reality –User: analyses results in near real-time, interacts, changes the structure of arteries
Citation preview
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
CrossGrid Architecture
Marian Bubak
and TATInstitute of Computer Science & ACC CYFRONET
AGH, Cracow, Poland
www.eu-crossgrid.org
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Overview
– Applications and their requirements– Tools for X# applications development– New grid services– X# architecture
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Biomedical Application
– Input: 3-D model of arteries– Simulation: LB of blood flow– Results: in a virtual reality– User: analyses results in near real-time, interacts, changes
the structure of arteries
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Steering in the Biomedical Application
CT / MRI scan
MedicalDB
Segmentation
MedicalDB
LB flowsimulation
VEWDPC
PDA
Visualization
Interaction
HDB
10 simulations/day60 GB20 MB/s
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
VR-Interaction
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Asynchronous Execution of Biomedical Application
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Interaction in Biomedical Application
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Cascade of Flood Simulations Data sources
Meteorological simulations
Hydraulic simulations
Hydrological simulations
Users
Output visualization
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Basic Characteristics of Flood Simulation
– Meteorological• intensive simulation (1.5 h/simulation) – HPC• large input/output data sets (50MB~150MB /event)• high availability of resources (24/365)
– Hydrological• Parametric simulations - HTC• Each sub-catchment may require different models
– Hydraulic• Many 1-D simulations - HTC• 2-D hydraulic simulations need HPC
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Distributed Data Analysis in HEP
– Objectives• Distributed data access• Distributed data mining techniques with neural networks
– Issues• Typical interactive requests will run on o(TB) distributed data• Transfer/replication times of order of 1h • Data transfers once and in advance of the interactive session.• Allocation, installation and set up the corresponding database
servers before the interactive session starts
Weather Forecast and Air Pollution Modeling
– Distributed/parallel codes on Grid• Coupled Ocean/Atmosphere Mesoscale Prediction System• STEM-II Air Pollution Code
– Integration of distributed databases – Data mining applied to downscaling weather forecast
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Key Features of X# Applications
– Data • Data generators and data bases geographically distributed• Selected on demand
– Processing• Needs large processing capacity; both HPC & HTC• Interactive
– Presentation• Complex data require versatile 3D visualisation• Support interaction and feedback to other components
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Modules of Tool Environment
GridMonitoring(Task 3.3)
PerformancePrediction
Component
High LevelAnalysis
Component
User Interface and Visualization Component
PerformanceMeasurementComponent
Benchmarks(Task 2.3)
Applications (WP1)executing
on Grid testbed
Applicationsourcecode
G-PM
RMD PMD
LegendRMD – raw monitoring data
PMD – performance measurement data
data flow
manual information transfer
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Tools Environment and Grid Services
ApplicationsPortals(3.1)
G-PMPerformance
Measurement Tools (2.4)
MPI Debugging and Verification
(2.2)
Metrics and Benchmarks
(2.4)
Grid Monitoring (3.3)(OCM-G, RGMA)
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
User Interaction Service (1/3)
– Synchronization between the simulation and visualization – Control of data flow between various Grid components– A plug-in mechanism for the required components– Software interfaces to:
• Resource broker (both DataGrid and Globus)• Condor-G system for High Throughput Computing• Nimrod/G tool for advanced parameter study• Grid monitoring service and Globus GIS/MDS
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
User Interaction Service (2/3)
User Interaction Services
User Interaction Services
User Interaction Services
User Interaction Service
Resource Broker
Resource Broker
Resource Broker
Resource Broker
Scheduler
User Interaction Services
User Interaction Services
User Interaction ServicesServiceFactory
Visualization/Interaction
In VE
Running Simulation
SimulationAgent
VisualizationAgent
InteractionAgent
Event Notification Mechanism
Other connections
Data transfer
NetworkBandwidth
Reservation Job
SubmissionService
Portal
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
User Interaction Service (3/3)
– Factory - creates UIS instances– UIS - event channel based service– Running Simulation – simulation sw– Simulation Agent – orchestrates simulation sw– Visualization/Interaction – virtual environment sw– Visualization Agent – orchestrates visualization sw– Interaction Agent – orchestrates interaction sw
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Portals and Roaming AccessApplications
Portals(3.1)
Roaming Access Server (3.1)
Scheduler(3.2)
GIS / MDS(Globus)
Grid Monitoring (3.3)
– Allow access user environment from remote computers– Independent of the system version and hardware– Run applications, manage data files, store personal settings
•Remote Access Server•user profiles •authentication, authorization•job submission
•Migrating Desktop•Application portal
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Monitoring System
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
OMIS Approach to Grid Monitoring
– Application oriented– on-line– data collected immediately delivered to tools– normally no storing for later processing
– Data collection based on run-time instrumentation– enables dynamic choosing of data to be collected– reduced monitoring overhead
– Standardized interface between tools and the monitoring system – OMIS
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Service Managers and Monitors
– Service Managers– one or more in the system– request distribution– reply collection
– Local Monitors– one per node– handle local objects– actual execution of requests
– Application Monitors– buffering data– filtering of instrumentation– monitoring requests
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Optimization of Grid Data Access
ApplicationsPortals(3.1)
Optimization of Grid Data Access (3.4)
Scheduling Agents(3.2)
Replica Manager(DataGrid / Globus)
Grid Monitoring (3.3)
GridFTP
Service consists of•Component-expert system•Data-access estimator•GridFTP plugin
–Different storage systems and applications’ requirements–Optimization by selection of data handlers
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Building Blocks of the CrossGrid
CrossGrid
DataGrid
GLOBUS
EXTERNAL
To be developed in X#
From DataGrid
Globus Toolkit
Other
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
Overview of the CrossGrid Architecture
Supporting Tools
1.4Meteo
Pollution
3.1 Portal & Migrating Desktop
ApplicationsDevelopment
Support
2.4Performance
Analysis
2.2 MPI Verification
2.3 Metrics and Benchmarks
App. Spec Services
1.1 Grid Visualisation
Kernel
1.3 DataMining on Grid (NN)
1.3 Interactive Distributed
Data Access
3.1Roaming Access
3.2Scheduling
Agents
3.3Grid
Monitoring
MPICH-G
Fabric
1.1, 1.2 HLA and others
3.4Optimization of
Grid Data Access
1.2Flooding
1.1BioMedApplications
Generic Services
GRAM GSIReplica CatalogGIS / MDSGridFTP Globus-IO
DataGridReplica
Manager
DataGrid Job Submission
Service
Resource Manager
(CE)CPU
ResourceManager
Resource Manager
(SE)Secondary
Storage
ResourceManager
Instruments ( Satelites,
Radars)
3.4Optimization of
Local Data Access
Tertiary Storage
Replica Catalog
GlobusReplica
Manager
1.1User Interaction
Services
BOF at GGF5, Edinburgh, Scotland, July 21-24, 2002
www.eu-crossgrid.org
