Ogce Workflow Suite Tg09

OGCE Workflow Suite for Science Gateways

Suresh Marru, Raminder Singh,Chathura Herath & Marlon Pierce

Indiana University

OGCE

Generalize, Harden, Build

Test

Requirements from gateways

• Gateways demand scientific workflow systems to be:– Flexible– Dynamic– Interactive – Technology Adaptive– Interoperable with Emerging Computational

Resources and their job management interfaces

OGCE Workflow Suite

• Generic Service Toolkit– Tool to wrap command-line applications as web services– Handles file staging&job submissions– Extensible runtime for security, resource brokering& urgent computing– Generic Factory service for on-demand creation of application services

• XRegistry– Information repository for the OGCE workflow suite– Register, search, retrieve&share XML documents– User & hierarchical group based authorization

• XBaya– GUI based tool to compose&monitor workflows– Extensible support for compiler plug-ins like BPEL &Jython– Dynamic Workflow Execution support to start, pause, resume, rewind

of workflow executions

OGCE Workflow Tutorial

Features

• Security– Authentication and authorization– Secure invocations between services– Support for gateway community accounts– Support for multiple user accounts

• Reliability– Retry job submissions and file staging– Fault Tolerance and Recovery service

• Over-provisioning and migration

• Compatibility– Taverna, Kepler and Trianna


Application Services• Workflows are built by composing web

services– Fortran applications are “wrapped” by a

Application Factory which generates a web service for the app.

• Registers WSDL for the service with a registry

– Each service generates a stream of notifications that log the service actions back to the XMC Cat Metadata Catalog.

c

Application Factory

App Service

Run program& publish events

Workflow Composition, Execution & Monitoring

XBaya enables users to construct, share, execute and monitor sequence of tasks executing on their local workstations to high-end compute resources.

Service Monitoring via Events• The service output is a stream of events

– I am running your request– I have started to move your input files.– I have all the files– I am running your application.– The application is finished– I am moving the output to you file space– I am done.

• These are automatically generated by the service using a distributed event system(WS-Eventing / WS-Notification)– Topic based pub-sub system with

a well known “channel”.Notification

Channel

listener publisher

SubscribeTopic=x

xx

ApplicationService

Instance

12

34

56

WRF-Static running on Tungsten

OGCE Workflow Tools

Workflow Suite Architecture

“A significant need exists in many disciplines for long-term, distributed, and stable data and metadata repositories”

“Metadata is key to being able to share results”

Metadata Catalog

Query For W

orkflows

Mon

itor W

orkf

lows

Sea

rch

Res

ults

Compose Workflow

Workflow

Portal

Message Bus

Wor

kflo

w

Not

ifica

tions

Record Workflow Inputs

Workflow Configuration and Intermediate Results

Record Workflow Outputs

Workflow Notifications

XML Metadata Catalog (XMC Cat)Taming Complex Scientific Metadata Schemas

– NSF Blue-Ribbon Advisory Panel on Cyberinfrastructure

– UK e-Science Core Programme Study

More Info: Scott Jensen

Applications• LEAD

– Lower entry barrier to using weather analysis tools– Improve detection, analysis & prediction of mesoscale weather

• Motif-Network– Transformation of sequenced genomes to “domain-space”

• Cyber-Infrastructure Evaluation– Performance evaluation of future supercomputer architectures

• ADAM– Algorithms for feature extraction, data normalization, classification

and normalization• GridChem

– Molecular Chemistry Grid helping researchers run chemistry applications on Grid Environment


88D Radar Re-

mapper

Satellite Data Re-mapper

NIDS Radar Re-mapper

Radar data (Level II)

Surface data, upper air mesonet data and

wind profiler data

Radar data (Level III)

Satellite data

ADAS

Terrain Preprocessor

3D Model Data

Interpolator (Initial

Boundary Conditions)

3D Model Data

Interpolator(lateral Boundary

Conditions)

Terrain data files NAM, RUC, GFS data

WRF Static Preprocessor

ARPS to WRF Data

Interpolator

ARPS Plotting Program

IDV

Surface, terrestrial data files

1

WRF to ARPS DataInterpolator

2

3

4

5

6

7

9

11

12

13

Run once per forecast region

Repeat periodically

for new data

Triggered if a storm is detected

Visualization on users request

ADAM

Data mining:look for storm

signature

WRFWRF

WRFWRF

14

ARPS Ensembl

e Generato

r

15

Static data Initialization Forecast VisualizationReal time data Analysis Data Mining

10

8

LEAD: A Weather Forecasting Workflow (1/2)

13

WRF-Static running on Tungsten

LEAD: A Weather Forecasting Workflow (2/2)


Motif-Network: Whole Genome workflow

• Domain webs of large genomes– Input list of amino acid sequences– Identify all known domains– Construct webs

Parallel processingCapability-type computing

Ensemble-type processing(minimal network reqs)

Capacity-type computing

Jeff Tilson, RENCIOGCE Workflow Tutorial

CI: Execute Sub-Workflow

• Input a campaign step filename• Execute GAMESS per step

specification

Jeff Tilson, RENCIOGCE Workflow Tutorial

On-DemandGrid Computing

Example: “Optimal” Weather Prediction Using Dynamic Adaptivity

StreamingObservations

Storms Forming

Forecast Model

Data Mining

Refine forecast grid

Instrument Steering

Live Demo & Questions?
