Pegasus: Mapping Scientific Workflows onto the Grid Ewa Deelman Center for Grid Technologies USC...

Pegasus: Mapping Scientific Workflows onto the Grid

Ewa Deelman

Center for Grid Technologies

USC Information Sciences Institute

Ewa Deelman Information Sciences Institute

Pegasus Acknowledgements

Ewa Deelman, Carl Kesselman, Saurabh Khurana, Gaurang Mehta, Sonal Patil, Gurmeet Singh, Mei-Hui Su, Karan Vahi (Center for Grid Computing, ISI)

James Blythe, Yolanda Gil (Intelligent Systems Division, ISI)

Collaboration with Miron Livny (UW Madison) http://pegasus.isi.edu Research funded as part of the NSF GriPhyN, NVO

and SCEC projects and EU-funded GridLab

Ewa Deelman Information Sciences Institute

Outline

Workflow Management in Grids Pegasus, Planning for Execution in Grids Applications Using Pegasus In-time planning Future Research Directions

Ewa Deelman Information Sciences Institute

Grid Applications

Increasing in the level of complexity Use of individual application components Reuse of individual intermediate data products (files) Description of Data Products using Metadata Attributes

Execution environment is complex and very dynamic Resources come and go Data is replicated Components can be found at various locations or staged in on

demand

Separation between the application description the actual execution description

Ewa Deelman Information Sciences Institute

FFT

FFT filea

/usr/local/bin/fft /home/file1

transfer filea from host1://home/filea

to host2://home/file1

ApplicationDomain

AbstractWorkflow

ConcreteWorkflow

ExecutionEnvironment

host1 host2

Data

Data

host2

App

licat

ion

Dev

elop

men

t and

Exe

cutio

n P

roce

ss

DataTransfer

Resource SelectionData Replica Selection

Transformation InstanceSelection

ApplicationComponentSelection

Retry

Pick different Resources

Specify aDifferentWorkflow

Failure RecoveryMethod

Abstract Workflow

Generation

ConcreteWorkflow

Generation

Ewa Deelman Information Sciences Institute

Why Automate Workflow Generation?

Usability: Limit User’s necessary Grid knowledge Monitoring and Directory Service Replica Location Service

Complexity: User needs to make choices

Alternative application components Alternative files Alternative locations

The user may reach a dead end Many different interdependencies may occur among

components Solution cost:

Evaluate the alternative solution costs Performance Reliability Resource Usage

Global cost: minimizing cost within a community or a virtual organization requires reasoning about individual user’s choices in light of

other user’s choices

Ewa Deelman Information Sciences Institute

GriPhyN’sExecutable Workflow Construction

Build an abstract workflow based on VDL descriptions (Chimera)

Build an executable workflow based on the abstract workflows (Pegasus)

Execute the workflow (Condor’s DAGMan)

AbstractWorfklow

Concrete Workflow

Jobs

Chimera Pegasus DAGMan

VDL

RLS TC MDS

Ewa Deelman Information Sciences Institute

VDL and Abstract Workflow

d1

d2

ba

cb

VDL descriptions

User request data file “c”

d1 d2

ba cAbstract Workflow

Ewa Deelman Information Sciences Institute

Condor’s DAGMan Developed at UW Madison (Livny) Executes a concrete workflow Makes sure the dependencies are followed Execute the jobs specified in the workflow

Execution Data movement Catalog updates

Provides a “rescue DAG” in case of failure

Ewa Deelman Information Sciences Institute

Pegasus:Planning for Execution in Grids

Maps from abstract to concrete workflow Algorithmic and AI-based techniques

Automatically locates physical locations for both components (transformations) and data

Finds appropriate resources to execute Reuses existing data products where applicable Publishes newly derived data products

Chimera virtual data catalog Provides provenance information

Ewa Deelman Information Sciences Institute

Information ComponentsUsed by Pegasus

Globus Monitoring and Discovery Service (MDS) Locates available resources Finds resource properties

Dynamic: load, queue length Static: location of gridftp server, RLS, etc

Globus Replica Location Service Locates data that may be replicated Registers new data products

Transformation Catalog Locates installed executables

Ewa Deelman Information Sciences Institute

Example Workflow Reduction

Original abstract workflow

If “b” already exists (as determined by query to the RLS), the workflow can be reduced

d1 d2ba c

d2b c

Ewa Deelman Information Sciences Institute

Mapping from abstract to concrete

Query RLS, MDS, and TC, schedule computation and data movement

Execute d2 at B

Move b from A

to B

Move c from B

to U

Register c in the

RLS

d2b c

Ewa Deelman Information Sciences Institute

Montage Montage (NASA and NVO) Deliver science-grade

custom mosaics on demand

Produce mosaics from a wide range of data sources (possibly in different spectra)

User-specified parameters of projection, coordinates, size, rotation and spatial sampling.

Mosaic created by Pegasus based Montage from a run of the M101 galaxy images on the Teragrid.

Ewa Deelman Information Sciences Institute

Small Montage Workflow

~1200 nodes

Ewa Deelman Information Sciences Institute

Montage Acknowledgments Bruce Berriman, John Good, Anastasia Laity,

Caltech/IPAC Joseph C. Jacob, Daniel S. Katz, JPL http://montage.ipac. caltech.edu/

Testbed for Montage: Condor pools at USC/ISI, UW Madison, and Teragrid resources at NCSA, PSC, and SDSC.

Montage is funded by the National Aeronautics and Space Administration's Earth Science Technology Office, Computational Technologies Project, under Cooperative Agreement Number NCC5-626 between NASA and the California Institute of Technology.

Ewa Deelman Information Sciences Institute

Applications Using Chimera, Pegasus and DAGMan

GriPhyN applications: High-energy physics: Atlas, CMS (many) Astronomy: SDSS (Fermi Lab, ANL) Gravitational-wave physics: LIGO (Caltech, AEI)

Astronomy: Galaxy Morphology (NCSA, JHU, Fermi, many others,

NVO-funded) Biology

BLAST (ANL, PDQ-funded) Neuroscience

Tomography for Telescience(SDSC, NIH-funded)

Ewa Deelman Information Sciences Institute

Current System

Original Abstract Workflow

Current Pegasus

Pegasus(Abstract Workflow)

DAGMan(CW))

Co

ncre

te W

orfklo

w

Workflow Execution

Ewa Deelman Information Sciences Institute

time

Levels ofabstraction

Application-level

knowledge

Logicaltasks

Tasksbound toresources

and sent forexecution

User’sRequest

Relevantcomponents

Fullabstractworkflow

Partialexecution

Not yetexecuted

executed

Workflow refinement

Taskmatchmaker

Workflow repair

Policyinfo

Workflow Refinement and execution

Ewa Deelman Information Sciences Institute

Incremental Refinement Partition Abstract workflow into partial

workflows

PW A

PW B

PW C

A Particular PartitioningNew Abstract

Workflow

Ewa Deelman Information Sciences Institute

Meta-DAGMan

Pegasus(A)

Pegasus(B)

Pegasus(C)

DAGMan(Su(A))

Su(B)

Su(C)

DAGMan(Su(B))

DAGMan(Su(C))

Pegasus(X) –Pegasus generates the concrete workflow and the submit files for X = Su(X)

DAGMan(Su(X))—DAGMan executes the concrete workflow for X

Ewa Deelman Information Sciences Institute

Conclusions

Pegasus maps complex workflows onto the Grid

Uses Grid information services to find resources, data and executables

Reduces the workflow based on existing intermediate products

Used in many applications Part of GriPhyN’s Virtual Data Toolkit

Ewa Deelman Information Sciences Institute

Future Directions

Investigate various scheduling techniques Investigating fault tolerance issues Enable flexible interactions between workflow

refiners (GriPhyN-wide scope: Pegasus, DAGMan)

http://pegasus.isi.edu

GGF10 workshop on workflow management GGF Workflow management research group

deelman@isi.edu

Ewa Deelman Information Sciences Institute

Summary:

The Grid Now Syntax-based matchmaking

of resources to job requirements

Condor matchmaker Attribute based discovery

and selection Scheduling of jobs based on

Grid-able users that specify job execution sequences and computing requirements

Scripting languages Workflow languages, Task graphs

Explicit mappings from task to jobs, simple job brokers

Explicit service negotiation and recovery strategies

The Future Grid Knowledge-based reasoning

about resources enables Semantic matchmaking Aggregate resource reasoning

Task-level reasoning to plan and schedule jobs and resources

More agility and coordination Wide range of users can specify

high level requirements in a mixed-initiative mode

Mapping of high-level requirements to details required for execution

End-to-end resource negotiation and adaptive strategies to accommodate failure