© 2005 IBM Corporation

IBM India

From Grid Computing to Cloud Computing – The IBM ApproachGaruda Partner Meet ,4th March 2008,Bangalore,India

P. Sambath Narayanan Ph.DIndia Systems & Technology LabIBM

2

Agenda

I. Important Challenges

III. Important Technologies, Trends and Standards

V. Case Studies

VII.From GRID Computing to Cloud Computing

IBM India

© 2005 IBM Corporation

Terminology

Virtualization Service Orientation Data management information Data Service Policy Management Interoperable Automation Lifecycle

IBM India

© 2005 IBM Corporation

I. Some Important Challenges

Data Management – Right Time & Right Data

Network Bandwidth and Latency

Security

Software and Standards

Need for many Grid based Scientific and Commercial Applications

Enable smooth scaling in many dimensions

Integration with the physical world

IBM India

© 2005 IBM Corporation

IBM India

© 2005 IBM Corporation

IBM India

© 2005 IBM Corporation

Few Important Challenges(contnd.)

Data Management is a Challenge Diverse usage scenarios

Volume of data - TBs

Right data at right time

Format of data

Heterogenity of systems at all level

Bandwidth, transfer, manipulation and analysis of large volume of data

IBM India

© 2005 IBM Corporation

Few Important Challenges (contnd.)

Network Bandwidth

Large volume of data needs to be transferred across the network

Ensuring right data to be available at the right time

Latency, Bandwidth, transfer, manipulation and analysis of large volume of data

Cost of Bandwidth

IBM India

© 2005 IBM Corporation

Grid Security

Intrusion Detection

Anti VirusManagement

PolicyManagement(Auth,Privacy,federation

User Managmnt.

KeyManagmnt

Tru

st Mod

el

Secu

re logg

ing

Policy Expression and exchange

Bindings security(transport, protocol,messagesecurity

Secure conversations

Credential & Id Translation

Access controlenforcement Audit

Service/end-pointpolicy

Mappingrules

AuthorizePolicy

PrivacyPolicy

IBM India

© 2005 IBM Corporation

II. Key Technologies

Virtualization

Storage / Data Management

Grid Security

Grid Software

IBM India

© 2005 IBM Corporation

Keywords

Virtualization Service SOA Data management information Data Service Policy Management Interoperable Automated Lifecycle

IBM India

© 2005 IBM Corporation

Grid Technology Evolution

OGSA

Managed,SharedVirtual System

Globus Toolkit

Standards,GT2Many Deployments

Many DeploymentsScientific Applications

1990 1995 2000 2005 2007

Grid

Ad

optio

n &

A

ccep

tanc

e

IBM India

© 2005 IBM Corporation

Grid Open Standards

OASIS (organization for the advancement of structured information standard)

WS – Resource Framework

WS – Notification

Open Grid Services Architecture-GGF OGSA Basic Profile

OGSA Security Profile

Basic Execution Services (OGSA-BES)

Job Submission Description Language (JSDL)

Data Access and Integration Services (DAIS)

Configuration Description, Deployment, and Lifecycle Management (CDDLM)

OGSA Byte I/O (Byte IO)

IBM India

© 2005 IBM Corporation

OGSA Design Principles

Service Orientation to virtualize resources Everything is a service

From Web service Standard interface service mechanisms, multiple protocols

bindings, local/remote transparency

From Grids Service semantics, reliability and security models

Life cycle management, discovery and other services

Multiple hosting environments C,J2EE,.NET

IBM India

© 2005 IBM Corporation

Technology Classification & Trends

Serial Applications

Parallel Applications

•Multi-threaded

•MPI

•open

Client Server

•CORBA

•COM/DCOM

•.NET, J2EE

•Home grown work

• distribution

P2P

•App Integration

•Reliable Messaging

•Reliable execution

Service virtualization

•Web services

•Service registration,

Discovery, invocation

•Location independent

•Lift App off the servers

Mainframes

Storage

•DAS

Open Systems

•Unix, Linux,Windows

Storage

•DAS

Clusters

•DRM

Storage•DAS

Infra. Virtualization

•Grid

•OGSA

•Data Grid

•Service provisioning

Infrastructure

Application technologies

Monolithic Open Distributed Virtualized

16

Virtual I/O paths

Virtualization-Single system & partitioning

Features Micro-partitioning

Share processors across multiple partitions

Minimum Partition: 1/10 processor

AIX 5L V5.3 or Linux*

Virtual I/O ServerShared EthernetShared SCSI & Fiber ChannelInt Virtualization ManagerAIX 5L V5.3 & Linux partitions

Partition LoadManagerAIX 5L V5.2 & V5.3 supportedBalances Processor & memory request

Partition Mobility

* = SLES 9 or RedHat v3 with update 3

POWER Hypervisor

LinuxAIX 5L

V5.2

Dynamically Resizable

3 Cores

2Cores

ManagerServer

LPAR 2AIX 5L V5.3

LPAR 1AIX 5L V5.2

LPAR 3Linux

PLM PartitionsUnmanaged Partitions

POWER Hypervisor

PLM Agent PLM Agent

AIX 5L

V5.3

6Cores

6 Cores

Lin

ux

AIX

5L

V5.

3

AIX

5L

V5.

3

AIX

5L

V5.

3

AIX

5L

V5.

3

AIX

5L

V5.

3

Lin

ux

Micro-partitioning

1Cores

Linux Linux

3 Cores

AIX 5L

V5.3

3Cores

EthernetSharing

Virtual I/O Server

Partition

StorageSharing

Int VirtManager

IBM India

© 2005 IBM Corporation

Virtualization – Information/StorageTechnology

Helps in addressing Data Management Challenges Integrated view of storage, fs and DB driven by

standard

Data transformation, security and replication

IBM India

© 2005 IBM Corporation

Virtualization - WorkloadTechnology

Workload Management Challenge

Single logical view of workload scheduling

Different type of scheduling environments and domains

Workload virtualization strategy is to create a single, logical view of workload scheduling.

This will enable users to accelerate performance of multiple large application workloads across their organization, leveraging and orchestrating IT resources in a flexible and dynamic fashion.

IBM India

© 2005 IBM Corporation

Virtualize Unlike Resources

Virtualize the

Enterprise

Virtualize Outside the Enterprise

Virtualize Like Resources

Single Systems& Partitioning

Cluster

Simple(2-4)

Sophisticated(4+)

Workload Management

SAN Volume Controller

Information Integrator

GPFS

NFS V4

SAN FS

Integrated Cluster Environment

XDExtended Deployment

LoadLeveler

LoadLeveler MultiCluster

CSMCluster Systems Manager

IntelligentOrchestrator

Systems Edition

Enterprise Edition

Provisioning Manager

EnterpriseWorkloadManager

Information

IBM Grid Toolbox

LSF

GridServer

Symphony

MultiCluster

MP Enterprise

Who does virtualization

IBM India

© 2005 IBM Corporation

Overcoming Network Challenges

Through efficient utilization of the Network. IBM Download Grid example. Explained in later slides.

Integrating with Global Research Networks

National Research & Education Networks. Supporting Research and Education communities

Specialized ISP

Lambda Grid

IBM India

© 2005 IBM Corporation

IBM India

© 2005 IBM Corporation

Overcoming Grid Security Challenges

Three key attributes of Grid Security Model Enables integration and interoperability

Creation and management of dynamic trust domains

Supports dynamic creation of services

OGSA Security Web services security standard

Grid Security Infrastructure (GSI) Portion of the Globus tool kit that implements security function

IBM India

© 2005 IBM Corporation

Virtualize Unlike Resources

Virtualize the Enterprise

Virtualize Outside the Enterprise

Virtualize Like Resources

Single Systems & Partitioning

Cluster

Simple(2-4)

Sophisticated(4+)

China GridMinistry of Education

People’s Republic of China

National Digital Mammography Archive

IBM

Across the Spectrum: Real Life References

IBM India

© 2005 IBM Corporation

Earth System Grid(ESG) - Case Study Overcoming Data Management Challenges

Service = RepositoryStorage Repository for Model generated atmospheric data3200 Users91,000 filesMore than 150 TB of data downloadedMore than 300 research papers

0

100

200

300

400

500

600

11/1

/04

12/1

/04

1/1/

05

2/1/

05

3/1/

05

4/1/

05

5/1/

05

6/1/

05

7/1/

05

8/1/

05

9/1/

05

10/1

/05

11/1

/05

12/1

/05

1/1/

06

2/1/

06

3/1/

06

4/1/

06

5/1/

06

6/1/

06

7/1/

06

8/1/

06

9/1/

06

10/1

/06

GB

/da

y

Daily 7-Day Average

GB

/day

600

IBM India

© 2005 IBM Corporation

ESG Architecture & Technologies

Climate data Metadata catalog

NcML (metadata schema)

OPenDAP-G (aggregation and subsetting)

Data management Data Mover Lite

Storage Resource Manager

Globus toolkit Globus Security Infrastructure

GridFTP

Monitoring and Discovery Services

Replica Location Service

Security Access control

MyProxy

User registration

DataSubsetting

AccessControl

UserRegistration

OPeNDAP-GMyProxy SRM DISKCache

ESG Web Portal

NCARCache

NCARMSS

RLS SRM

ORNLHPSS

RLS SRM

RLS

SRM

RLS LANLCache

searchbrowse

download

WebBrowser

DML DataUser

publishWebBrowser

DataProvider

MonitoringServices

DataPublishing

ClimateMetadata

CatalogsBrowsing

UsageMetrics

DataDownload

DataSearch

NERSC

MSS, HPSS: Tertiarydata storage systems

IBM India

© 2005 IBM Corporation

Energy Exploration – Case Study

Service = Seismic Computing 3-D Seismic imaging is the most resource intensive Grid Enabled system for Seismic Imaging Gulf of Mexico 3-D Marine Surveys Estimated run times on a cluster(128 cpu,2.4 GHz,Pentium) Compute intensive wave equation provides better accuracy

Slides are based on the work done by 3DGeo Team. See Reference Material

IBM India

© 2005 IBM Corporation

Parallelization of PSDM on Multiple Clusters

Clusters from 3DGeo processing centres and Clusters from SDSC

MPICH-G2 / MPICH-GP(Kum Rye Park)

Globus Tool kit DCs –

SantaClara,Houston,SanDiego Supercomputing Centre

This slide is based on the work done by 3DGeo Team. See Reference Material

IBM India

© 2005 IBM Corporation

Medical Education Over Access GridWork Done by J.Silverstein, U. Chicago

IBM India

© 2005 IBM Corporation

National Digital Mammography ArchiveElectronic Medical Record data grid and repository

Motivation To help doctors and medical students learn more about breast

cancer and related diseases Challenges

Managing and storing of huge files for fast retrival

Annual NDMA volume could exceed 5.6 peta bytes per year– Image size 160 MB per study

Minimum daily traffic estimated 28 TB

NETwork bandwidth and response

Encryption of patient data and transmission across public networks

IBM India

© 2005 IBM Corporation

caBIG: sharing of infrastructure, applications, and data.

DataIntegration!

Service Oriented Science – Cancer & Biology

IBM India

© 2005 IBM Corporation

IBM India

© 2005 IBM Corporation

Technology Evolution

OGSA

Managed,SharedVirtual System

Globus Toolkit

Standards,GT2Many Deployments

Many DeploymentsScientific Applications

1990 1995 2000 2005 2007

Grid

Ad

optio

n &

A

ccep

tanc

e

IBM India

© 2005 IBM Corporation

Business Challenges

With demand for IT resources hard to predict, service providers usually over-provision resources in order to support peak demands and ensure continuous service availability and quality, while other systems run at lower capacity,

IBM India

© 2005 IBM Corporation

Cloud Computing Defined

Large pools of systems are linked together to provide IT services

Service-based online economy resources and services are transparently provisioned

and managed.

IBM India

© 2005 IBM Corporation

The Need - Cloud Computing

Dramatic growth in connected devices

Real-time data streams

the adoption of service oriented architectures

Web 2.0 applications

Open collaboration, social networking and mobile commerce.

Massive increase in the scale of IT environments

driving the need to manage them as a unified cloud.

IBM India

© 2005 IBM Corporation

Business SolutionAfter Cloud Computing

Cloud-computing-based technologies that will enable the borderless delivery of IT services based on actual demands to keep costs competitive.

Seamless delivery of services to consumers regardless of demand or available computing resources

Virtualization and Grid Technologies

IBM India

© 2005 IBM Corporation

Cloud Computing Example

Delivery of online entertainment. Distribution of television shows, movies and other videos

are moving to the Web the cloud computing technologies would enable a

network of service providers to host the different media. Using cloud computing technology, the broadcasters can join forces to reach a service cooperation contract that enables them to tap into advanced services including content distribution, load balancing, and overlay networking across different platforms in different countries.

If there is large demand for a show hosted by a particular site, it can dynamically 'hire' additional servers and services from other sites that are not being used.

IBM India

© 2005 IBM Corporation

Blue Cloud

Series of cloud computing offerings

Allow corporate data centers to operate more like the Internet

Enable computing across a distributed, globally accessible fabric of resources, rather than on local machines or remote server farms.

IBM India

© 2005 IBM Corporation

Reference Material

A Virtualization Experience: IBM Worldwide Grid Implementation, Moon Kim et al., IBM Red Books, IBM

Grid 2, Edited by Ian Foster & Carl Kesselman,Elsevier,2004 Grid computing for energy exploration,

D.Beve,S.E.Zarantonello,N.Kaushik,I.Musat

IBM India

© 2005 IBM Corporation

Summary

1) Storage, Security, Network and Application availabilty are major Grid challenges

2) Virtualization is an important technology for the Grid3) Many large Grid projects have been working successfully4) Think of Grid for variety of services, not just for computing

alone5) Grid, virtualization and service orientation have many things in

common

© 2005 IBM Corporation

IBM India

From Grid Computing to Cloud Computing – The IBM ApproachGaruda Partner Meet ,4th March 2008,Bangalore,India

P. Sambath Narayanan Ph.DSambath.narayanan@in.ibm.com