GCC 2006 Panel:Grid Research and Engineering Vs Standards

Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS) LaboratoryDept. of Computer Science and Software EngineeringThe University of Melbourne, Australiawww.gridbus.org

Gridbus Sponsors

2

Panel Questions

Q1: How can research creativity be reconciled with the need for, and use of, standards and standard software?

Q2: What are the most important challenges (and possible solutions) in managing in an integrated manner the middleware, data, users, and resources that make up a Grid?

Q3: How can Grid-user/developers be provided with an integrated “on the Grid” development environment?

Q4: What are the essential aspects of the “engineering science” needed for Grid systems that are interoperable, scalable, robust, sustainable, and maintainable?

3

Q1: Research Creativity Vs Standards

Standard are great as long as they are restricted to: Protocols Interfaces Formats

Successful ICT standards: ASCII (languages), IEEE (devices), IETF (networking), W3C (web and web

services), OGF (in progress, yet to settle) Yes, Research(ers) should embrace standard as long as

purpose is served, if not, “invent” new one. If we push for a specific software as a standard,

It creates “one world” rules all mentality! They can “kill” many creative works leads to loss of diversity. So called de-facto standards make extremely hard for creative ideas to get

in.

4

What does Grid promises

Resource sharing across multiple administrative boundaries Effective utilisation of existing resources Dynamic provisioning

Application Acceleration Scalability Reliability Virtualisation:

applications, services, resources,…

5

E.g., The Use of Grid as a Cyberinfrastructure for e-Science

Distributed instruments

Distributed computation

Distributed data

Peers sharing ideas and collaborative

interpretation of data/results

2100 2100 2100 2100

2100 2100 2100 2100

Remote Visualization

Data & Compute Service

Cyberinfrastructure

E-Scientist

6

Q2: Multi-institution Collaboration Challenges

Security

Resource Allocation & Scheduling

Data locality

Network Management

System Management

Resource Discovery

Uniform Access

Computational Economy

Application Construction

8

Some Open-Source Grid Middleware Solutions

9

What does Grid players want?

Grid Consumers Execute jobs for solving varying problem size and complexity Benefit by utilizing distributed resources wisely Tradeoff timeframe and cost

Strategy: minimise expenses

Grid Providers Contribute resources for executing consumer jobs Benefit by maximizing resource utilisation Tradeoff local requirements & market opportunity

Strategy: maximise return on investment

10

Solution 1: Service Oriented Architecture (SOA)

A SOA is a contractual architecture for offering and consuming software as services.

There are four entities that make up an SOA service provider, service registry, and service consumer (also known as service requestor).

The functions or tasks that the service provider offers, along with other functional and technical information required for consumption, are defined in

the service definition or contract.

provider

registry

consumer

contract

11

Solution 2: Market-Oriented Grid Computing - (a) Sustained Resourced Sharing and (b)

Effective Management of Shared Resources

Grid Economy

12

The Gridbus Project @ Melbourne:Enable Leasing of ICT Services on Demand

WWG

Pushes Grid computing into mainstream

computing

Gridbus

14

Q3 & Q4: “on the grid” app. dev. Environment and “Engineering Science”

Interoperability: Do use standards

Scalable Decentralised management, make sure “application” scale from

“desktop” to “global grids” Robust

Fault-management, persistence, state-saving (like Windows!)… Sustainable and Maintainable

Make a clear separation between different concerns E.g: application programming interface must be independent of

software used for deploying and executing “user” abstract services, resources, “device independent” “applications” and “execution management/scheduling engine”,

low-level middleware. Support “old” and “new” standard at the same time!

15

Gridbus Broker: Separating “applications” from “different” remote service access

enablers and schedulers

Alchemi

Gateway

UnicoreData Store

Access Technology

Grid FTPSRB

-PBS-Condor-SGE

Globus

Job manager

fork() batch()

Gridbusagent

Data Catalog

-PBS-Condor-SGE-XGrid

SSH

fork()

batch()

Gridbusagent

Single-sign on securityHome Node/Portal

GridbusBroker

fork()

batch() -PBS-Condor-SGE-Alchemi-XGrid

Application Development Interface

Sch

ed

ulin

gIn

terfa

ces

Alogorithm1

AlogorithmN

Plugin Actuators

InterGrid: Internetworking of Islands of Grids

Rajkumar Buyya

Grid Computing and Distributed Systems(GRIDS) Laboratory

Dept. of Computer Science and Software EngineeringThe University of Melbourne, Australia

www.gridbus.org/gridsim

5th International Conference on Grid and Cooperative ComputingChangsha, China, October 2006

17

The Outcomes of Grid Computing thus Far: Islands of Grids

Organizational Gridor island of Grid

18

InterGrid: An architecture for Internetworking of “islands” of Grids

G rid

G rid G rid

IG G

IG G

IG G

RP

RP

RP

RP

RP

RP

RP

RP

RP

IRM

IRM

IRM

IntraG rid

G G

RP

RP IRM

IntraG rid

G G

G ridIG G

RP

RPIRM

IG G

RPRP

RP

IRM

G rid

IG G

RP

RP

IRM

G rid

IG G

RP RP

RP IRMPeering

InterGrid resourceallocationSites providing resources

Service deploym ent / usage

CL

G RB

19

The InterGrid: How to get there?

Organizational Gridor island of Grid

InterGrid G ateway

O rganizational Gridor island of G ridPeering arrangement

20

New Research Challenges

Decentralised “service” discovery from gateways-based to “leaf” nodes

Protocols and Business models for “Peering” between Grids

Application models that support growing and shrinking of application

Resource management and scheduling SLA based resource allocation.

21

Thanks for your attention!

We Welcome Cooperation in Research and Development!http:/www.gridbus.org

22

Layers of Grid Architecture

Grid resourcesDesktops, servers, clusters, networks, applications,

storage, devices + resource manager + monitor

Security ServicesAuthentication, Single sign-on, secure communication

Job submission, info services, Storage access, Trading, Accounting, License

Resource management and scheduling

Grid programming environment and toolsLanguages, API, libraries, compilers, parallelization tools

Grid applicationsWeb Portals, Applications,

Adaptiv

e M

anagem

ent

Application Development and Deployment Environment

Distributed Resources Coupling Services

CoreMiddleware

User-LevelMiddleware

System level

User level

Au

tonomic/ G

rid Econ

omy