Grid Programming Environment: Implementation & Globus Interface

Grid Programming Environment: Implementation & Globus Interface

Ralf RateringThomas KentemichAlexander LukichevIgor Odintsov

Grid Day in CGSLNovember 3, 2005

November 3 2005

- 2 -

*Other brands and names are the property of their respective owners © Copyright 2005 Intel Corporation. All Rights Reserved.

Agenda

Intel GPE Component Overview

Intel GPE Implementation Setting up a GPE Grid Globus Interface Current State and Next Steps

November 3 2005

- 3 -


Intel GPE Component Overview

November 3 2005

- 4 -


Application Programming Layer

GPE Component Overview

ApplicationsFSI, Energy, Mfg, Telco, Pharma, Retail, ...

Resource Virtualization Layer

Service OrientedArchitecture/Infrastructure

(SOA & SOI)

Grid Middleware

Compute Storage Network

Operating System

Compute Storage Network

Operating System

...

Man

ag

em

ent

Secu

rity

Application Programming LayerGrid Programming Library

Grid SDK Client FrameworkGridBeans

Globus Toolkit-based Open

Source implementation

(GPE4GTK)

Prototype

Examples and Templates

November 3 2005

- 5 -


Grid Platform

Grid Application I/F (WSRF, JSDL, ...)

Mg

mt

Co

nso

le

Application Programming Layer

Grid Middleware

Grid Platform I/F(WSRF, WSDM,

JSDL, …)

Workflow Engine

Scheduler / BrokerTarget System

Registry

Database Access

H/W Platform I/F(CIM & WS–based)

Service OS

JobManagement

Service(JMS)

TargetSystemService(TSS)

StorageManagement

Service(SMS)

FileImportService

(FIS)

FileExportService(FES)

TargetSystemFactory(TSF)

MRTE

Virtual Machine

OS

Target System

Interface (TSI)

Virtual Machine

OS

Target System

Interface (TSI)create & manage

Application Deployment Service

Application Client Web portals

Grid Platform

...

...

Ad

min

Clie

nt

GPE Architecture (Fitting in Grid SW Architecture)

1

2

3

= GPE components under development

Hosting Environments and Server Implement. (UNICORE/GS, GTK4, …)

Expert Client

Firmware ( Platform S/W)

Platform H/W & Platform *Ts(Compute, Storage, Network)

Remote File Manager

GridBeans

= GPE components (not started)

Firmware ( Platform S/W)

Platform H/W & Platform *Ts(Compute, Storage, Network)

Applications

November 3 2005

- 6 -


OGSA (Open Grid Services Architecture)– Share components with other architectures

WS* (WS-Addressing, WSRF, WSN, etc.)– Interoperation with other Grid Middleware

JSDL (Job Submission Description Language)

– High level job description that can be submitted to all target systems offering a JSDL interface

BPEL (Business Process Execution Language)

– Integration of Grid Bean services into larger business process workflows

CIM (Common Information Model)– Used to describe resources

– Usage of CIM management interfaces for Grid administration

Emerging Standards

November 3 2005

- 7 -


Intel GPE Implementation

November 3 2005

- 8 -


Expert Client

GPE Client Framework Full access to Grid for expert

users and administrators Workflow Editor Supports multiple applications

and user identities

Application Client

Light-weight Java application running on mobile devices

Simple application specific interface

Portal Client

JSR-168 compliant application portlets that can be integrated into existing portals like UPortal, Jetspeed or GridSphere

November 3 2005

- 9 -


Expert Client

Application Client

Portal Client

Write GridBeans for your applications

GridBean Service

GridBean

GridBean

GridBean

GridBean

GridBean

GridBean

GridBean

GridBean SDK

GPE Client API, libraries and tools

GridBean

GridBean

GridBean

A GridBean dynamically integrates application-specific user interfaces in different clients

November 3 2005

- 10 -


Expert Client

Application Client

Portal Client

GPE as interoperability framework

UNICORE/GS

Globus Toolkit 4

Other OGSA-compliant Grid

servers

China Grid Support Package

SOAP, WSDL,WSRF, WS-Addressing, WS-Security,JSDLand other specifications

Atomic Services

Atomic ServiceClient API

November 3 2005

- 11 -


Atomic Services Overview

Target System Factory(TSF)

Atomic service interfaces define basic set of operations and properties that have to be available on a Grid system

– Operations describe the functionality

– Properties describe the state (WS-Resource Framework)

Different implementations of interfaces for different infrastructures

– Protocols, languages, security models, ...

Target SystemService(TSS)

JobManagement

Service(JMS)

StorageManagement

Service(SMS)

FileTransferService(FTS)

Managefiles on storage

Add a new target systemto the Grid

Manage transfers to/from storage

ManageJobs on target system

Managetarget system

November 3 2005

- 12 -


JobManagement

Service(JMS)

StorageManagement

Service(SMS)

FileTransferService(FTS)

Operations Service Resource Type Properties

Atomic Service Interfaces based on WSRF

Extends WS-RP/LTstart()abort()hold()resume()

Extends WS-RP/LTJob-EPR submit(JSDL, initialTT)

Job3Job2

Status=RunningOriginalJSDL=<JSDL..>ExecutionJSDL=<JSDL..>etc.

Job1

create

Extends WS-RP/LTlistDirectory(path)listFileProperties(path)createDirectory(path)changePermissions(path, perm)copy(path, path)rename(path, path)delete(path)import(path, protocol, isPipe)export(path, protocol, isPipe)

HomeRoot

AvailableFileSpace=127kbSupportedProtocols= {GridFTP, UPL, scp}etc.

TS-Property

Temp

Work

create

Target System Factory(TSF)

Target SystemService(TSS)

Target System

Memory=1GbRunning Jobs=23CPU Type=ia64Storage=Temp, Rootetc.

create

TSR-EPR create(config)

Extends WS-RP/LT

FileTransfer 1

FileTransfer 2

Sourcefile=c:/tmp/test.txtDestFile=/tmp/test.txtTransferred=300kbetc.

create

November 3 2005

- 13 -


Hiding platform-specific information with Application Resources

Abstract job concept borrowed from UNICORE– No concrete platform specific information in

job decription (paths, libraries, etc.)

– Job will be incarnated on target system

– For security and portability reasons!

Available applications can be queried from target system resource properties

Use JSDL Posix extensions to specify required application resources in submitted job

November 3 2005

- 14 -


Transparency

Virtualization of the computational resources requires the job definitions to be abstract.

Non-abstract definition:

/usr/bin/povray /var/uspaces/ec82a0f0-4b7d-bf2585/povray.pov

Abstract definition:

<Job><Application name=“POVRay”/><Field name=“Input”>povray.pov</Field>

</Job>

November 3 2005

- 15 -


Abstract Job Definitions

Advantages of abstract job definitions:

Easy to use with brokers Don’t need to disclose private

information about target system configuration

One step job submission

November 3 2005

- 16 -


File Transfers Atomic Services support different

protocols–FTP, GridFTP, plain HTTP(S), SOAP with

Attachements, any other Storage management announces

available protocols via its resource properties–Client queries available protocols and

selects appropriate one–GridFTP for large high-performance

transfers–HTTPS or SOAP w/a to work with

firewall limitations

November 3 2005

- 17 -


Client

Advanced Services: Workflow Execution

Workflow Execution

Service(TSS)

BPEL-enabled JSDL

Target System

Target System

CustomWeb-service

orchestrate

First implementation combining BPEL, JSDL and WSRF

Use Grid-specific BPEL subset– Allows integration into larger

business processes

Implemented as WSRF service itself– Information about workflow (state,

input/output files, etc.) is available through resource properties

November 3 2005

- 18 -


Advanced Services: Target System Registry

Registry keeps track of static and dynamic information– Hardware properties, available

software, ...

– Workload, available disk space, ...

TSS contacts registry on startup and when properties change (WS-Notification)

Clients and services query informations about target systems from registry

Client Target System

Target System

Target System

Register/update

Query

Resource Broker

Target System Registry

November 3 2005

- 19 -


GridBeans InternalsG

rid

Bea

n

Job DefinitionJob Definition

GPE Workflow Engine:

Atomic Target System:

• Contact arbitrary web-services

• Perform XML evaluations

• Incarnate user’s task

• Perform computations

November 3 2005

- 20 -


GridBeans

POVRay

Visit

Compiler Gaussian

GridBean API

FinancialSimulationInterface

GPE Client API

GPE Client Framework

Web Portals

GPE Application Client

GPE Expert Client

Higher-level application services

EngineeringApplications

FinancialSimulations

etc.

Grid Programming API•Easy to implement•Portable•Easy to maintain

Advanced servicesAtomic Services

China Grid Support Package

Target SystemRegistry

UNICORE/GS

Globus ToolkitResource

Broker

Workflow

•Higher-level custom services take advantage of basic functionality provided by atomic and advanced services•Optional deployment of user interfaces through GridBeans

November 3 2005

- 21 -


Setting up a GPE Grid

November 3 2005

- 22 -


Adding a target system to a Grid

Target SystemStatic

Properties

Incarnation Database(IDB)

Target System

Interface (TSI)

Lightweight Java component to perform the actual work

Service container

Configuration file describing hardware and software properties

Configuration file defining systems specific paths to commands, libraries, etc.

1. Start Target System Interface

2. Define Static Properties and Incarnation Rules3. Start Service Container

Atomic services

November 3 2005

- 23 -


Target System

TSI

Web Server

Service container

Target System

TSI

Target System

TSI

Atomic services

Site A

Target System

TSI

Service container

Atomic services

Target System

TSI

Service container

Atomic services

Site B

Web Server

Service container

Advanced services

Application services

Site C

A GPE Inter-Grid

Clients

Globus Interface

November 3 2005

- 25 -


Why Globus?

De-facto standard, widely used Powerful, fast evolving Has enough features to host GPE services

November 3 2005

- 26 -


Differences between Globus and UNICORE/GS hosting environments

Globus Toolkit 4 UNICORE/GS

Transport-level security

TLS v1 (SSL v3) SSL v2

Delegation Proxy certificates Extended Trust Delegation

WSRF specs Draft v1.0 Latest specs

de-facto standard

November 3 2005

- 27 -


Portability

GPE client core

Globus client API implementation

Unicore/GS client API implementation

Globus atomic services interface

GPE service core

UNICORE/GS

November 3 2005

- 28 -


Incarnations Database

povray +I<SOURCE> +O<TARGET> +F<OUTPUT_FORMAT> +W<WIDTH> …

fields

Fields are taken from the abstract job definition and are merged with the default values for user or application. The system is very flexibly configurable.

GPE uses incarnations database to store the templates of user’s jobs to incarnate.

November 3 2005

- 29 -


Current State and Next Steps

November 3 2005

- 30 -


GPE & GPE4GTK

GPE Alpha version has been released at UNICORE SourceForge project on June 1, 2005– Full working version for demos and developers

GPE-enhanced GTK4 implementation– SourceForge project „GPE4GTK“

– First release on October 28, 2005

– Beta release on December 31, 2005

UNICORE/GS support in European UniGridS project

Cooperation with China Grid to integrate GPE until end of 2006

Possible cooperation with SimDAT

November 3 2005

- 31 -


Summary

Intel GPE......provides easy portable solution for

different Grid infrastructures including UNICORE/GS and GTK4

...provides a client framework to give users access to the infrastructure

...provides the GridBean concept and a programming API for Grid developers

...is available as Open Source today

Thank you !

Documents

Grid Programming Environment: Implementation & Globus Interface