Grid Computing Research @ Melbourne: Gridbus Perspective Dr. Rajkumar Buyya Grid Computing and...

Grid Computing Research @ Melbourne: Gridbus Perspective

Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS) LaboratoryDept. of Computer Science and Software EngineeringThe University of Melbourne, Australiawww.gridbus.org

Gridbus Sponsors

2

Grids @ Various Schools in Melbourne:Technology Creation -> Applications

Faculty of Engineering – CS/EE/Geomatics GRIDS Lab and the Gridbus Project – technology focus SensorWeb (NICTA) Natural Language Processing Virtual organisation for tsunami research

Faculty of Science - Physics Belle and LHC Grid Australian Virtual Observatory

Faculty of Medicine – HFI, Anatomy MRI Grid Distributed Kidney Modelling

Social Sciences/Humanities The development of a collaborative distributed environment for humanities

research projects whose primary data takes the form of ethnographic audiovisual media.

A Grid-enabled technology for the sharing of graphics, audio, video and text between social science research groups working internationally.

Information Tech. Services Division MARC (Melbourne Advanced Research Computing)/HPC

3

Classes of Grid Services

Computational Services – CPU cycles Pooling computing power: SETI@Home, TeraGrid,

AusGrid, ChinaGrid, IndiaGrid, UK Grid,… Data Services

Collaborative data sharing generated by instruments, sensors, persons: LHC Grid, Napster

Application Services Access to remote software/libraries and license

management—NetSolve Interaction Services

eLearning, Virtual Tables, Group Communication (Access Grid), Gaming

Knowledge Services The way knowledge is acquired, processed and

managed—data mining. Utility Computing Services

Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities.

Computational Grid

Data Grid

ASP Grid

Interaction Grid

Knowledge Grid

Utility Grid

infra

stru

ctu

re

Users

4

Grid Challenges

Security

Resource Allocation & Scheduling

Data locality

Network Management

System Management

Resource Discovery

Uniform Access

Computational Economy

Application Construction

5

GRIDS Lab @ Melbourne

Youngest and one of the rapidly growing research labs in our School/University:

Founded in 2002 Houses:

Research Fellows/PostDocs Research Programmers PhD candidates Honours/Masters students

Funding National and International organizations Australian Research Council & DEST Many industries (Sun, StorageTek, Microsoft,

IBM, Microsoft) University-wide collaboration:

Faculties of Science, Engineering, and Medicine

Many national and international collaborations.

Academics Industries

Software: Widely in academic and industrial users.

Publication: GRIDS Lab produces over 20% of our Dept’s

research output.

EducationR & D

+ Community Services: e.g., IEEE TC for Scalable Computing

6

Selected Funded Projects: 2004-2006

Australian Research Council: Discovery Projects:

"The Development of Data Grids for High Energy Physics" "Economic Scheduling for Efficient Management of Clusters and

their Cooperative Federation" Application Projects:

"Development of e-Research Tools for an MRI Grid Computing Facility"

"e-Research Grid Environment for Integration of Distributed Kidney Models and Resources”

7

The Gridbus Project @ Melbourne:Enable Leasing of ICT Services on Demand

WWG

Pushes Grid computing into mainstream

computing

Gridbus

8

The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying e-Research Applications on Utility Grids

The Gridbus Project @ GRIDS Lab, The University of Melbourne: The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying eToolkit for Creating and Deploying e--Research Applications on Utility GridsResearch Applications on Utility Grids

Gridbus

Distributed Data

http://www.gridbus.org

• Gridbus is a “open source” Grid R&D project with focus on Grid Economy, Utility Grids and Service Oriented Computing.

• Gridbus Middleware components include:– Alchemi: .NET-based Enterprise Grid

– Grid Market Directory and Web Services

– Grid Bank: Accounting and Transaction Management

– Visual Tools for Creation of Distributed Applications

– Grid Service Broker and Scheduling

– Workflow Management Engine

– GridSim Toolkit

– Libra: SLA-based Resource Allocation

9

What does Grid players require?

They need tools and technologies that help them in value expression, value translation, and value enforcement.

Grid Service Consumers (GSCs): How do I express QoS requirements ? How do I trade between timeframe & cost ? How do I map jobs to resources to meet my QoS needs? How do I manage Grid dynamics and get my work done? …

Grid Service Providers (GSPs) How do I decide service pricing models ? How do I specify them ? How do I translate them into resource allocations ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? …

10

Solution 1: Service Oriented Architecture (SOA)

A SOA is a contractual architecture for offering and consuming software as services.

There are four entities that make up an SOA service provider, service registry, and service consumer (also known as service requestor).

The functions or tasks that the service provider offers, along with other functional and technical information required for consumption, are defined in

the service definition or contract.

provider

registry

consumer

contract

11

Solution 2: Market-Oriented Grid Computing - (a) Sustained Resourced Sharing and (b)

Effective Management of Shared Resources

Grid Economy

12

Grid Node N

Service-Oriented Grid Architecture

Grid Service Consumer

Pro

gra

mm

ing

En

viro

nm

ents

Grid Resource Broker

Grid Service Providers

Grid Explorer

Schedule Advisor

Trade Manager

Job ControlAgent

Deployment Agent

Trade Server

Resource Allocation

ResourceReservation

R1

Misc. services

Information Service

R2 Rm…

Pricing Algorithms

Accounting

Grid Node1

…

Core Middleware Services

…

…

HealthMonitor

Grid Market Services

JobExec

Info ?

Secure

Trading

QoS

Storage

Sign-on

Grid Bank

Ap

pli

cati

on

s

Data Catalogue

13

Gridbus and Complementary Technologies – realizing Utility Grid

AIXSolarisWindows Linux

.NETGridFabricSoftware

GridApplications

Core GridMiddleware

User-LevelMiddleware

GridBank

Grid Exchange & Federation

JVM

Grid Brokers:

X-Parameter Sweep Lang.

Gridbus Data Broker

MPI

Condor SGE TomcatPBS

Alchemi

Workflow

IRIX OSF1 Mac

Libra

Globus Unicore ……Grid

MarketDirectory

PDB

CDB

Worldwide Grid

GridFabricHardware

……

PortalsScience Commerce Engineering ……Collaboratories

……

Workflow Engine

Grid Storage Economy

Gri

d E

con

om

y

NorduGrid XGrid

ExcellGrid

Nimrod-G

Gridscape

14

On Demand Assembly of Services: Putting Them All Together

ASP Catalogue

Grid Info Service

Grid Market Directory

GSP(Accounting Service)

GridbusGridBank

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)

(Globus)

Alchemi

GS

GTS

Cluster Scheduler

Job

8

GridResource Broker

2

Visual Application Composer

Application CodeExplore

data1

36

45

Resu

lts9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

1533

On Demand Assembly of Services: Putting Them All Together

ASP Catalogue

Grid Info Service

Grid Market Directory

GSP(Accounting Service)

GridbusGridBank

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)(Globus) Alchemi

GS

GTS

Cluster Scheduler

Job

8

GridResource Broker

2

Visual Application Composer

Application CodeExplore

data1

36

45

Res

ults

9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

16

Alchemi: .NET-based Enterprise Grid Platform & Web Services

InternetInternet

InternetInternet

Alchemi Worker Agents

Alchemi Manager

Alchemi Users

Web Services

Web Services

•SETI@Home like Model•General Purpose•Dedicated/Non-dedicate workers•Role-based Security•.NET and Web Services•C# Implementation•GridThread and Job Model Programming•Easy to setup and use• Widely in use!

17

Some Users of Alchemi

Tier Technologies, USALarge scale document processing using Alchemi framework

CSIRO, AustraliaNatural Resource Modeling

The Friedrich Miescher Institute (FMI) for Biomedical Research, SwitzerlandPatterns of transcription factors in mammalian genes

Satyam Computers Applied Research Laboratory, IndiaMicro-array data processing using Alchemi framework

The University of Sao Paulo, BrazilThe Alchemi Executor as a Windows Service

stochastix GmbH, GermanyServing clients in International Banking/Finance sector

Many users in Universities: See next for an example.

18

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

19

A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids.

It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users’ QoS requirements (deadline, budget, & T/C optimisation)

Key Features A single window to manage & control experiment Programmable Task Farming Engine Resource Discovery and Resource Trading Optimal Data Source Discovery Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & sharing of results Accounting

Grid Service Broker (GSB)

20

Gridbus Broker Architecture

Grid Middleware

Gridbus Client Gridbus ClientGribus Client

Grid Info Server

Schedule Advisor

Trading Manager

Gridbus Farming Engine

RecordKeeper

Grid Explorer

GE GIS, NWSTM TS

RM & TS

Grid Dispatcher

RM: Local Resource Manager, TS: Trade Server

G

G

CU

Globus enabled node.A

L

Alchemi enabled node.

(Data Grid Scheduler)

DataCatalog

DataNode

Unicore enabled node.

$

$

$

App, T, $, Opt

(Bag of Tasks Applications)

21

Gridbus Broker: Separating “applications” from “different” remote service access

enablers and schedulers

Alchemi

Gateway

UnicoreData Store

Access Technology

Grid FTPSRB

-PBS-Condor-SGE

Globus

Job manager

fork() batch()

Gridbusagent

Data Catalog

-PBS-Condor-SGE-XGrid

SSH

fork()

batch()

Gridbusagent

Single-sign on securityHome Node/Portal

GridbusBroker

fork()

batch() -PBS-Condor-SGE-Alchemi-XGrid

Application Development Interface

Sch

ed

ulin

gIn

terfa

ces

Alogorithm1

AlogorithmN

Plugin Actuators

22

Gridbus Services for eScience applications

Application Development Environment: XML-based language for composition of task farming (legacy)

applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine.

Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes

that meet user QoS requirements. Hide Low-Level Grid Middleware interfaces

Globus (v2, v4), SRB, Alchemi, Unicore, and ssh-based access to local/remote resources managed by XGrid, Condor, SGE.

23

Figure 3 : Logging into the portal.

Drug DesignMade Easy!

Click Here for Demo

24

Deadline (D) and Budget (B) Constrained Scheduling Algorithms

Algorithm Execution Time (D)

Execution Cost (B)

Compute Grid

Data Grid

Cost Opt Limited by D Minimize Yes Yes

Cost-Time Opt

Minimize if possible

Minimize Yes

Time Opt Minimize Limited by B Yes Yes

Conservative-Time Opt

Minimize Limited by B, jobs have guaranteed minimum budget

Yes

25

Gridbus Project: Some Applications and Users

Gridbus Project: Gridbus Project: Some Applications and UsersSome Applications and Users

http://www.gridbus.org

BioGrid: Molecular docking for Drug-discovery

BioGrid: Molecular docking for Drug-discovery

High Energy Physics: Particle Discovery

High Energy Physics: Particle Discovery

Melbourne University

NeuroScience: Brain Activity Analysis

NeuroScience: Brain Activity Analysis

Natural Resource ModelingNatural Resource Modeling

CSIRO Land and Water, Austraila.

Large Scale document processing

Large Scale document processing

Tier Technologies, USA.

Detection of patterns of transcription factors in mammalian genes

Detection of patterns of transcription factors in mammalian genes

26

GridSim: A Toolkit for Modeling and Simulation of Various Types of Grids

Basic Discrete Event Simulation Infrastructure

Virtual Machine (Java, cJVM, RMI)

PCs ClustersWorkstations

. . .

SMPs Distributed Resources

GridSim Toolkit

Application Modeling

InformationServices

Resource Allocation

Grid Resource Brokers or Schedulers’s Simulation

Statistics

Resource Modeling and Simulation (with Time and Space shared schedulers)

ClustersSingle CPU ReservationSMPs Load Pattern

Application Configuration

Resource Configuration

Visual Modeler

Grid Scenario

Network

SimJava Distributed SimJava

Resource Entities

Output

Application, User, Grid Scenario’s Input and Results

Add your own policy for resource allocation

AuctionFramework

27

Selected Funded Projects: 2006-2009

Australian Research Council: Discovery Project:

QoS-based Scheduling of e-Research Application Workflows on Global Grids

QoS-based Workflow Scheduling SLA-based Resource Allocation

Dept. of Science, Training and Education International Science Linkage

The Utility Grid Project: Autonomic and Utility-oriented Global Grids for Powering Emerging E-Research Applications

Links Gridbus Efforts to Many International Efforts

28

Grid Workflow Application Modeling & Definition Tools

Grid Workflow Specification and Verification

Grid Workflow Management System

Resource Info Service

Application Catalogue

Build Time

Run Time

Workflow Design & Monitoring

Workflow Execution Control & Monitoring

Interaction with Grid resources

Interaction with VO Info services

QoS-based Workflow Scheduling

Fault Management Data ManagementData Catalogue

Virtual Organization

R2 Rn……

SLA-based Resource Allocation System(Plug-in for Existing Local Resource Managers)

Negotiation Services

ExecutionMonitor

E-Researchers/Users

feedback

Core Grid Services

SynchrotronData source

Global Grid

29

SLA-based Allocation of Cluster Resources

Computation Node 1

Job Manager

Computation Node c

:::

Economy-based Admission

Control

Economy-based Resource Allocation

Computation Nodes

User u

:::

Manager Node

Node Status Monitor

Job Control

User Population

User 1job1

2 3

45

Utility-driven Job Scheduler

execution results

feedback

job submission

execution results

Pricing

Resource Manager

InterGrid: Internetworking of Islands of Grids

Rajkumar Buyya

Grid Computing and Distributed Systems(GRIDS) Laboratory

Dept. of Computer Science and Software EngineeringThe University of Melbourne, Australia

www.gridbus.org/gridsim

5th International Conference on Grid and Cooperative ComputingChangsha, China, October 2006

31

The Outcomes of Grid Computing thus Far: Islands of Grids

Organizational Gridor island of Grid

32

InterGrid: An architecture for Internetworking of “islands” of Grids

G rid

G rid G rid

IG G

IG G

IG G

RP

RP

RP

RP

RP

RP

RP

RP

RP

IRM

IRM

IRM

IntraG rid

G G

RP

RP IRM

IntraG rid

G G

G ridIG G

RP

RPIRM

IG G

RPRP

RP

IRM

G rid

IG G

RP

RP

IRM

G rid

IG G

RP RP

RP IRMPeering

InterGrid resourceallocationSites providing resources

Service deploym ent / usage

CL

G RB

33

The InterGrid: How to get there?

Organizational Gridor island of Grid

InterGrid G ateway

O rganizational Gridor island of G ridPeering arrangement

34

New Research Challenges

Decentralised “service” discovery from gateways-based to “leaf” nodes

Protocols and Business models for “Peering” between Grids

Application models that support growing and shrinking of application

Resource management and scheduling SLA based resource allocation.

35

Summary and Conclusion

Grids have emerged as enabler for Cyberinfrastructure that powers e-Science and e-Business applications.

Delivering ICT services as computing utilities. Australia is actively developing Grid technologies and

applications Recently Australian Govt. is pushing Grids heavily under

e-Research banner. Melbourne is leading a number of Grid projects nationally. SOA + Market-based Grid Management = Utility Grids Grids allow users to dynamically lease Grid services at

runtime based on their quality, cost, availability, and users QoS requirements.

Grids offer enormous opportunities for realizing e-Science and e-Business at global level.

36

Thanks for your attention!

We Welcome Cooperation in Research and Development!http:/www.gridbus.org