Cracow Grid Workshop CGW'05 (Cracow, Poland)

ADAPTING THE DEVELOPMENT MODEL OF THE GRID

ANATOMY TO MEET THE NEEDS OF VARIOUS

APPLICATION DOMAINS

Soha Maad, Brian Coghlan, Gabriel Pierantoni,

Eamonn Kenny, John Ryan, Watson Ronan

Trinity College Dublin, Dublin, Ireland

Cracow Grid Workshop CGW'05 (Cracow, Poland)

Outline

● The grid in various application domains● Application domain and grid middleware development● Software Engineering perspective on grid middleware

development● Software Engineering Practice Proposed● A concrete implementation example


The grid in various application domains


Che

mis

try

Gov

ernm

ent

Phy

sics

Med

icin

e

Ast

rono

my

Env

iron

men

t

Eng

inee

ring

Med

ia

Fin

ance

Challenge

Challenge

Limitation

Limitation

Grid Solution

Grid Solution



Che

mis

try

Gov

ernm

ent

Phy

sics

Med

icin

e

Ast

rono

my

Env

iron

men

t

Eng

inee

ring

Med

ia

Fin

ance

meeting the deadline of the start date for conducting the physics experiments

Limitation

Limitation

conducting experiments that need to process tera-bytes scale of data (e.g. LHC, ALICE, ATLAS).

Challenge

Challenge

the EGEE middleware is developpedGrid Solution

Grid Solution



Che

mis

try

Gov

ernm

ent

Phy

sics

Med

icin

e

Ast

rono

my

Env

iron

men

t

Eng

inee

ring

Med

ia

Fin

ance

1) producing interactive medical simulations (e.g. heart simulation).2) analysing and managing medical images.3) supporting virtual collaboration in e-hospital.

Challenge

Challenge

Grids provide computing power but lack capabilities needed for communication and knowledge sharing. Limitation

Limitation - Special purpose middleware (e.g. innerGrid Nitya developed by Grid Systems, also CrossGrid)- Resource brokers for the discovery of suitable clusters for the execution of parallel image reconstruction algorithms. - Metagrid Engine that provides a superset of functionalities across different Grid Engines with privacy and QoS.

Grid Solution

Grid Solution



Che

mis

try

Gov

ernm

ent

Phy

sics

Med

icin

e

Ast

rono

my

Env

iron

men

t

Eng

inee

ring

Med

ia

Fin

ance

Challenge

Challenge

Analysis of tera-bytes of astronomical image data from telescopes. This necessitates: data intensive computation; scalable file I/O in GB/s; replica management; and parallel/distributed processing of files.

Grid Solution

Grid Solution

GFARM Grid file system offered a special purpose Grid middleware for data intensive computation

Limitation

Limitation

No direct connection between the GFARM system and the telescope devices.



Che

mis

try

Gov

ernm

ent

Phy

sics

Med

icin

e

Ast

rono

my

Env

iron

men

t

Eng

inee

ring

Med

ia

Fin

ance

Challenge

Challenge parallel running of hundreds of programs corresponding to large scale air pollution, nuclear waste storage, pollution and weathering models

Grid Solution

Grid Solution

For large scale air pollution models:National (UK): 7 institutions using GT3International: 21 institutions using CrossGrid

Limitation

Limitation

Future research work aims at deploying a WSRF compliant middleware.



Che

mis

try

Gov

ernm

ent

Phy

sics

Med

icin

e

Ast

rono

my

Env

iron

men

t

Eng

inee

ring

Med

ia

Fin

ance

Challenge

ChallengeVirtual collaboration on design, production and maintenance of products that are described in complex structured product model databases.

Limitation

Limitation

- the need for the grid to be aware of the business concepts - to make typical server side applications grid-computing compatible, - to allow CAD applications to interface with the grid.

Grid Solution

Grid Solution

The VO concept should be enhanced to a robust framework with flexibility, adaptability, and security.Proposed generic business-object-aware middleware extensions: 1) grid-enabling existing applications using toolkits; 2) extending the grid architecture with semantics and ontologies.



Che

mis

try

Gov

ernm

ent

Phy

sics

Med

icin

e

Ast

rono

my

Env

iron

men

t

Eng

inee

ring

Med

ia

Fin

ance

Challenge

ChallengeThe production, broadcasting, delivery and playout, of interactive media content (audio, video, image) in real time.

Limitation

Limitation

Difficulties:- To split media production pipeline distribute it to grid resources;- To guarantee reliable and secure interaction with the pipeline. - To provide reliable and secure interactivity with the grid

(CrossGrid is only ‘near-interactive)

Grid Solution

Grid SolutionCrossGrid:- The development of the Grid Visualisation Kernel (GVK) which allows the visualisation pipeline to be ported on grid resources. - The development of G-Vid which is based on GVK for production of real time interactive video.



Che

mis

try

Gov

ernm

ent

Phy

sics

Med

icin

e

Ast

rono

my

Env

iron

men

t

Eng

inee

ring

Med

ia

Fin

ance

Challenge

Challenge1) Solving real world large scale investment problems 2) Realistic financial modelling 3) Regulatory constraints (Basle II) force big finance houses to create their own grid apps.

Grid Solution

Grid Solution

Implementation of the Open Grid Service Environment (model large scale computational financial problems as abstract workflows).

Limitation

Limitation

Need to converge to web service implementation



Che

mis

try

Gov

ernm

ent

Phy

sics

Med

icin

e

Ast

rono

my

Env

iron

men

t

Eng

inee

ring

Med

ia

Fin

ance

Challenge

ChallengeMolecular design and engineering (e.g. QSAR/QSPR methodology involves 3D structure generation, semi-empirical calculations, descriptor calculation, and model building.

Grid Solution

Grid Solution

OpenMolGRID system, based on UNICORE grid middleware.



Che

mis

try

Gov

ernm

ent

Phy

sics

Med

icin

e

Ast

rono

my

Env

iron

men

t

Eng

inee

ring

Med

ia

Fin

ance

Challenge

Challenge

Limitation

Limitation

Grid Solution

Grid Solution

DiscussionImpact of the Application Domain on Grid Middleware


Che

mis

try

Gov

ernm

ent

Phy

sics

Med

icin

e

Ast

rono

my

Env

iron

men

t

Eng

inee

ring

Med

ia

Fin

ance

deve

lopi

ng n

ew m

iddl

ewar

e ke

rnel

s

deve

lopi

ng n

ew m

iddl

ewar

e ke

rnel

s

deve

lopi

ng r

esou

rce

brok

ers

deve

lopi

ng r

esou

rce

brok

ers

intr

oduc

ing

new

met

agri

d en

gine

s

intr

oduc

ing

new

met

agri

d en

gine

s

Dev

elop

ing

grid

ser

vice

API

s

Dev

elop

ing

grid

ser

vice

API

s

addi

ng n

ew m

iddl

ewar

e se

rvic

es

addi

ng n

ew m

iddl

ewar

e se

rvic

es

deve

lopi

ng g

rid

appl

icat

ion

tool

kits

deve

lopi

ng g

rid

appl

icat

ion

tool

kits

Rai

sing

gri

d aw

aren

ess

of b

usin

ess

conc

epts

Rai

sing

gri

d aw

aren

ess

of b

usin

ess

conc

epts

advi

sing

sem

antic

-aw

are

sol

utio

ns

advi

sing

sem

antic

-aw

are

sol

utio

ns

Dev

elop

ing

spec

ial p

urpo

se g

rid

mid

dlew

are

Dev

elop

ing

spec

ial p

urpo

se g

rid

mid

dlew

are

Key Issue Addressed

Grid Middleware Software Development Model

Bottom-up approach

Top-down approach


Bottom-up Approach

“Ecosystem” of Grid Components (Globus Alliance)

Individual parts of the system are specified in detail and the parts are then composed to form larger components, which are in turn composed until a complete system is formed

The exercise of building a Grid system or application is framed as a software integration problem, hopefully, leveraging existing grid components to reduce the development cost .


Top-Down Approach

In WebComG, applications are specified as Condensed Graphs in a manner which is independent of the execution architecture, thus separating the application and execution environments.


However …..

However, our survey reveals that the development cost of grid applications using a bottom-up approach is considerable.

By adopting a bottom-up approach, most of grid middleware development has evolved in isolation of application domain needs.


However …..

A top-down approach for grid middleware development may lead to narrow solutions and tend to be less generic


Adapting the development model of the grid anatomy


We propose marrying a top-down approach with a bottom-up approach for middleware development:

From a software engineering perspective, we need bridge bridge infrastructuresinfrastructures capable of merging a top down with a bottom-up approach. This relies on:

● A Unified Computational Model capable of expressing the workflow of operations

● Stateful Web Services for the implementation of the Bridge Infrastructures.

A Concrete Example


Border servicesBorder services involve :

Guardian services:Guardian services: have the duty to check whether all the conditions to cross the border are met

Translation services:Translation services: have the duty to translate information between the different protocols and languages used in WebCom-G and the various Grid Middlewares.

Proxy servicesProxy services have the duty to act in one middleware on behalf of a service residing in the other.

A Concrete Example


Border Region

Border Region

LCG2 RegionLCG2 Region

GT4 RegionGT4 Region

WebComRegion

WebComRegion

At most two technologies overlap

WebCom-G Interoperability with existing Grid MiddlewareWebCom-G Interoperability with existing Grid Middleware


Documents

Cracow Grid Workshop CGW'05 (Cracow, Poland)