ICENI Overview & Grid Scheduling

ICENI Overview&

Grid Scheduling

Laurie Young

London e-Science CentreDepartment of Computing, Imperial College

2

ICENI

IC e-Science Networked InfrastructureDeveloped by LeSC Grid Middleware GroupCollect and provide relevant Grid meta-data Use to define and develop higher-level services Interaction with other frameworks: OGSA, Jxta etc.

The Iceni, under Queen Boudicca, united the tribes of South-East England in

a revolt against the occupying Roman forces

in AD60.

3

ICENI (The Big Picture)

Private

Resource Manager

Policy Manager

CR

SR

Identity Manager

Domain Manager

CR

SR

Gateway between private and public regions

Public

Resource Browser

Public Computational Community

SR CR

Public Computational Community

SR

Private

Administrative

Domain

SR

CR

Resource Broker

Application Design Tools

Component Design Tools

Application Mapper

Web ServicesGateway

Application

Portal

Computational Resource

SoftwareResources

NetworkResources

StorageResources

JavaCoGGlobus

4

ICENI Stack

Portal Interface

Application Construction & Deployment

ICENI Middleware

Grid Fabric

5

Web Portals

• Handheld wireless devices become ubiquitous– Personal Digital Assistants, Mobile Phones– Secure access any time, any place, any where

• Use X.509 certificates embedded in a browser to authenticate user’s identity

• Integration portal infrastructure with ICENI– EPIC: Use component meta-data to

build portal application• Goal: Provide secure ‘one stop shop’ for e-

science

6

EPIC: e-Science Portal at Imperial College

• Collaborative LeSC industrial project with Sun Microsystems

• Develop a secure portal infrastructure to:– Access your own personal environment– Applications to support day-to-day e-science– Interaction with other Grid infrastructures

• Allow role based access to resources– Anonymous: public web pages– Students: internal pages, email, compute resources– Staff: restricted pages

7

ICENI Application Model

• Legacy code!• Component Applications

– Compose applications from many components– Component does work on data– Component communicates data

8

Component Motivation

• Logical application model• Collaborative software authoring • Promote component reuse and sharing• Simplify application construction• Enable deployment to diverse Grid resources:

– Communication Selection– Implementation Selection

9

Layered Abstraction

Meaning

BehaviourBehaviour

dataflowabstract data types

Implementation

control flowthreads etc.

performance,architectures,concrete data type

may have many

may each have many

Implementation

10

Component’s View of the Grid

OtherCode

SOAP

More Code

RMI

My Code

You must implementa provided interface

You may call methods

provided by the middleware

Context object

11

Visual Component Composition

12Grid

Container

Deployment of Components

ComponentDesignTools

Scientist

ApplicationDesignTools

EndUser

Application DescriptionDocument

Developer

ImplementationAnnotating

Tools

Code CodeCode

Code

Run-TimeRepresentation

ApplicationMapper

RTR

Code

Access ResourceInformation

APO

Application Proxy Object

Repository

13

SOAPRMI

Component Execution

Compute Resource Hardware

RTR

CodeCode Code

RTR RTR

NetworkResource

MPI

APO

Jini Jini

OGSA, Jxta, etc. OGSA, Jxta, etc.

14

Components as Services

Component

Service interface

SOAP (or other)protocol

Context object

15

ICENI & Jini: P2P

Service requester

Service LocatorService Provider

LookupService

ServiceMatches

Two- wayInteractionwith service

Register service

Discover services

Conceptual Model of peer- to- peer architecture

16

Web Services Architecture

Service requester(SOAP Client)

Service Locator(UDDI Registry)

Service Provider(SOAP Server)

UDDILookup

UDDIResponse, WSDL location

SOAPmessageexchange

UDDI registration

Web Services Crawler?

Web Service Model

17

Synergy

The Synergy

Service requester

Service Locator Web Service Proxy

Service Provider Web Service Proxy

Service RequesterWeb Service Client

P2P(J ini)

Web Services

J ini Lookup J ini Object

J ava App

RMI RMI

SOAP SOAP

SOAP

SOAPProxy

18

Grid Service Contracts

JiniLookupService

DRMAAResource

DRMAAClient

19


JiniLookupService

DRMAAResource

User: A+BDuration:1hr

DRMAAClient

User:B

ResourceBrowser

20


JiniLookupService

DRMAAResource


DRMAAResource

User:ADuration:10m

DRMAAClient

User:B

DRMAAClient

User:A

21

OGSA & Jini Integration

JiniLookupService

GatewayManager

GSI enabledWeb Service

HostingEnvironment

DRMAAResource


DRMAAResource

User:ADuration:10m

22


JiniLookupService

GatewayManager


HostingEnvironment

Jini ClientInterface

WSDLInterface

DRMAAResource


DRMAAResource

User:ADuration:10m

23


JiniLookupService

GatewayManager


HostingEnvironment

GSI + SOAPConnection Jini Client

Interface

WSDLInterface

DRMAAResource


DRMAAResource

User:ADuration:10m

24


JiniLookupService

GatewayManager


HostingEnvironment

GSI + SOAPConnection

User Info

SOAP->Java

Jini ClientInterface

WSDLInterface

DRMAAResource


DRMAAResource

User:ADuration:10m

25

Application Mapping (Scheduling)

• Architecture– How meta-data is collected– What meta-data is required

• Scheduling Algorithms– Map components onto resources for “best” results– Meta-data dependent decisions

26

Scheduling Architecture

Resources

ICENI

App Builder (GUI) Component Repository Performance Models

Scheduler Launcher

27

Multiple Metrics (1)

• “It is the goal of a scheduler to optimise one or more metrics” (Feitelson & Rudolph)

• Generally one metric is most important– Application Optimisation

• Execution time• Execution cost

– Host Optimisation• Host utilisation• Host throughput• Interaction Latency

28

• In a Grid Environment there are three application optimisation based important metrics– Start time ( )– End time ( )– Cost ( )

• Relative importance varies on a user by user and application by application basis

Multiple Metrics (2)

be

29

• A Benefit Function maps the metrics we are interested in to a single Benefit Value metric

• Different benefit functions represent different optimisation preferences

Combining Metrics – Benefit Fn

),,( ebBB

30

Optimisation Preferences

• Cost Optimisation

• Time Optimisation

• Cost/Time Optimisation

max max e and if

eB

max max e and if

eB

max max e and if

eB

31

Schedule Benefit

• Each component and communication has a benefit function

• Each resource and network connection has a predicted time & cost for each component or communication that could be deployed

• Fit the tasks onto the resources to get the maximum Total Predicted Benefit

),,( ebt BB

32

Graph Oriented Scheduling (1)

• Applications are described as a graph– Nodes represent application components– Edges represent component communication

• Resources are described as a graph– Nodes represent resources– Edges represent network connections

33


Condor pool

Atlas Saturn

Viking

Design Analyse

Scatter

Gather

Mesh

DRACS

Mesh

DRACS

Mesh

DRACS

Factory

34


Condor pool

ScatterGather

DesignAtlas

Factory

AnalyseSaturn

Viking

35

Summary

• Component framework provides:– Rich application meta-data– Decoupled component definition and implementation

• Meta-Data:– Exploit performance information to map component implementation to

the ‘best’ resources• Resource Broker:

– Resource selection through user defined policies:• Minimise cost using computational economics• Minimise execution time using the application mapper

36

Acknowledgements

• Director: Professor John Darlington• Technical Director: Dr Steven Newhouse• Research Staff:

– Anthony Mayer, Nathalie Furmento– Stephen McGough, James Stanton– Yong Xie, William Lee– Marko Krznaric, Murtaza Gulamali– Asif Saleem, Laurie Young, Gary Kong

• Contact:– http://www.lesc.ic.ac.uk/– e-mail: [email protected]

Documents

ICENI Overview & Grid Scheduling