Running Workflowson Clouds and Grids

Gabor Terstyanszky, University of Westminster T. Fahringer, P. Kacsuk, J. Montagnat, I. Taylor

e-Science Workshop, Oxford, 15 March 2012

SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585

2

Workflow Interoperability Challenge

33

Computer and Automation Research Institute, Hungarian Academy of Sciences

MTA SZTAKI Hungary

University of Innsbruck UIBK Austria

Charité - Universitätsmedizin Berlin C-UB Germany

French National Centre for Scientific Research CNRS France

University of Westminster UOW United Kingdom

Cardiff University CU United Kingdom

Academic Medical Centre of the University of Amsterdam AMC Netherlands

University of Southern California USC USA

SHIWA ProjectPartners:

Duration:July 2010 – June 2012 (extended to September 2012)

Project Objectives and Services

Objectives:• To create an environment which enables seamless execution of

workflows of different workflow systems through workflow interoperability

Services:• To support the whole workflow lifecycle: editing, uploading, browsing

downloading and executing workflows

• To provide coarse- and fine-grained workflow interoperability solutions

• To offer Distributed Computing Infrastructure interoperability solution

• To provide desktop computer and portal interfaces to manage workflows

Key actors:• researchers workflow engine developers

workflow developers

4

5

Workflow description abstract workflow•plain text which describes the inputs and outputs and explains what the workflow doesWorkflow implementations concrete workflow

• It contains all implementation files or references to these files (via eg. URLs) and also holds other data/metadata necessary to run the workflow on its associated workflow engine.

Workflow configurations• It contains pre-defined input files and parameters or references to these files and parameters and other

data/metadata of a workflow.Workflow engines• It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references

to them

SHIWA Data Structure

6

J2 WF4

J1

J3

GEMLCA client

GEMLCA Service

Workflow Engine B

Workflow of Workflow Engine B

Workflows of Workflow Engine A

Coarse-Grained Interoperability

DCI

• native jobs (or workflows): J1, J2, J3• non-native workflows: WF4

- black boxes which are managed as legacy code applications

Workflow Engine A

7

SHIWA Science Gateway Resourcesnative WE WS-PGRADE local resources: invocation of locally deployed WEs portal WS-PGRADE v3.3.2WE submission to local cluster repository GEMLCA + SHIWA repo remote resources: through remotely pre-deployedsubmitter GEMLCA with GIB WEs to gLite and Globus DCIsproxy management SHIWA Proxy Server

SHIWA Portal

Legacy Code GUI

SHIWA Science Gateway

WE1 WEp

GEMLCA Repository

WF1 WFm

SHIWA Repository

WF1 WFn

WS-PGRADE Workflow

engine

WS-PGRADE Workflow

editor

gLite DCI Globus DCI

pre-deployed-WEs

MOTEUR WE

Kepler WE

Taverna WE

Triana WE

local cluster

MOTEUR WE

GWES WE

ASKALON WE

SHIWA VO

ASKALON WE

GEMLCA Service

GEMLCA with GIB

SHIWA Proxy Server

Proxy Server

CGI Infrastructure

ProActive WE

8

SHIWA Portal: Editing Workflows

9

SHIWA Portal: Configuring Workflows

10

SHIWA Portal: Executing Workflows

11

SHIWA Workflow Repository

Browse Workflows

Browse Implementations

Search Sort

12

SHIWA Workflow Repository

13

SHIWA RepositorySHIWA Portal

WF1

SHIWA Science Gateway

GEMLCA Service

WFn

WE1 WEp

GEMLCA Repository

WE + WF

WF1 WFm

GEMLCA with GIB

WF list

WS-PGRADE Workflow

engine

WS-PGRADE Workflow

editor

edit WF

s2

search WF

s1

s5

s4

gLite DCI

MOTEUR WE

GWES WE

Globus DCI

pre-deployed-WEs

MOTEUR WE

Kepler WE

Taverna WE

Triana WE

local cluster

ASKALON WE

SHIWA VO

ASKALON WE

user

invoke WEs7

CGI User Scenario: Native WE

Legacy Code GUI

SHIWA Proxy Server

Proxy Server

s3

s7

submit WE

s6

ProActive WE

141414

SHIWA RepositorySHIWA Portal

WF1

SHIWA Science Gateway

GEMLCA Service

WFn

WE1 WEp

GEMLCA Repository

WF1 WFm

GEMLCA with GIB

WS-PGRADE Workflow

Engine

WS-PGRADE Workflow

Editor

search WF

s1

WE + WF s5

gLite DCI

MOTEUR WE

GWES WE

Globus DCI

MOTEUR WE

Kepler WE

Taverna WE

Triana WE

local cluster

ASKALON WE

SHIWA VO

ASKALON WE

user

invoke WE s7

CGI User Scenario: Non-native WE

Legacy Code GUI

GEMLCA Client

MOTEURWorkflow

Engine

MOTEUR Workflow

editor

s2

s3

GEMLCA UI

SHIWA Proxy Server

Proxy Server

s6

s4

ProActive WE

15

DCI Interoperability

JSDL Translator

Workflow Engine

DCI Bridge

Metabroker

Proxy Server

DCI

J2

J1

J4

J3

jobs in JSDL

J2

J1

J4

J3

jobs in non-JSDL

16

DCI BridgeFeatures:• web service based app which provides standard access to different

DCIs: clouds - EC2, OpenNebula clusters - LSF, PSB desktop grids - BOINC, OurGrid, XtremWeb service grids - ARC, gLite, Globus, Unicore

• workflow jobs are submitted to the BES Factory Service of the DCI Bridge where jobs are described in JSDL

Components:• resource registry - configuration interface of DCIs

• application manager - BES management

• runtime system - job execution management via BES Factory

• monitoring - logs and messages handling and visualisation

17

DCI Bridge

WS-PGRADE Workflow

Engine

WS-PGRADE Workflow

Editor

Portal

BES Factory Service

Job Registry

Input Queue

Upload Marnager

Plug-in Manager

PluginA

PluginB

PluginC

inputs & executables

outputs

clouds

desktop grids

service grids

DCI Bridge

proxy server

s1

s2 s3

s4s5

s6

s7

s8s10

s9

s11

181818

Fine-Grained Interoperability

• front-end plug-in: converts workflow of workflow engine A into IWIR (Intermediate Workflow Interpreter Representation)

• back-end plug-in: converts from IWIR into workflow of workflow engine B

IWIR workflow

J2

J1

J4

J3

WFA ->IWIR frontend plug-in

J2

J1

J4

Workflow of Workflow Engine A

J3

WFA ->IWIR backend plug-in

Workflow Engine B

DCI

J2

J1

J4

J3

Workflow of Workflow Engine B

ASKALON

MOTEUR

PGRADE

Triana

SHIWA RepositorySHIWA Portal

WF1

SHIWA Science Gateway

WFn

WE1 WFp

GEMLCA Repository

WF1 WFm

gLite DCI Globus DCI

DCIs

ARC DCI Unicore DCI

GEMLCA submitter

Kepler WEGWES WE

Taverna WE

Triana WE

MOTEUR WE

ASKALON WE

WS-PGRADE Workflow

engine

WS-PGRADE Workflow

editor PGRADE WE

pre-deployed-WEs

SSP DCI & FGI Infrastructure

SHIWA Science Gateway Resourcesnative WE ASKALON, MOTEUR, WS-PGRADE

local WEs: Kepler, MOTEUR , Taverna & Triana WEsportal WS-PGRADE v3.4.1 submit to the local cluster repository GEMLCA + SHIWA remote WEs: ASKALON, GWES & MOTEUR WEsGEMLCA

ARC, gLite, Globus and Unicore DCIsproxy server

DCI Bridge

Meta Broker

Proxy Server

JSDL Translator

ProActive WE

2020

SHIWA RepositorySHIWA Portal

WF1

SHIWA Science Gateway

WFn

WE1 WFp

GEMLCA Repository

WF1 WFm

gLite DCI Globus DCI

DCIs

ARC DCI Unicore DCIWS-PGRADE

Workflow engine

DCI Interoperability Scenario

DCI Bridge

Meta Broker

Proxy Server

JSDL Translator

search WF

s1

s6

s4s2

edit workflow

s7

submit workflow

s5

s3

s3

WS-PGRADE Workflow

editor

2121

SHIWA RepositorySHIWA Portal

WF1

SHIWA Science Gateway

WFn

WE1 WFp

GEMLCA Repository

WF1 WFm

DCIs

WS-PGRADE Workflow

Engine

WS-PGRADE Workflow

Editor

FGI Interoperability Scenario

ccDCI

Bridge

Meta Broker

Proxy Server

JSDL Translator

search WF

s1

Workflow Engine

Workflow Editor

s4FP1

FP2

FPn

cc

BP1

BP2

BPm

s3s2 edit WF

submit WFs7

gLite DCI Globus DCI

ARC DCI Unicore DCI

s5

s6

2222

SHIWA Simulation Platform:Access

SHIWA Science Gateway (SHIWA Portal + SHIWA Repository)

• joint portal and repository account given by the science gateway administrator

DCI resources

• certificate given by the relevant DCI administrator

Access to the SHIWA Portal

shiwa-portal.cpc.wmin.ac.uk/liferay-portal-6.05

Access to the SHIWA Repository

shiwa-repo.cpc.wmin.ac.uk

23

Conclusions

• the simulation platform supports the whole lifecycle of workflows, i.e. creating, testing, uploading, browsing, downloading and running workflows

• users can use workflows of their own and of other workflow systems through the same user interface, i.e. they can re-cycle or share workflows created by other research teams or even other research communities

• workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform

• researchers (astro physicists, earth scientists, bio scientists, etc.) can browse the workflow repository, select and run workflows through the simulation platform

• the simulation platform provides access to major European e-infrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources