Swift: A Scientist’s Gateway toCampus Clusters, Grids and

Supercomputers

Swift project: www.swiftlang.org

Presenter contact: davidkelly@uchicago.edu swift-support@ci.uchicago.edu

David KellyComputation Institute, University of Chicago

and Argonne National Laboratory

Parallel scripting language for clusters, clouds & grids– For writing loosely-coupled scripts of application programs

and utilities linked by exchanging files– Can call scripts in shell, python, R, Octave, MATLAB, …

Swift does 3 important things for you:– Makes parallelism transparent – with functional dataflow– Makes basic failure recovery transparent– Makes computing location transparent – can run your script

on multiple distributed sites and diverse computing resources (from desktop to petascale)

2www.swiftlang.org

MODIS script excerpt

a

3

www.swiftlang.org

$ cat modis.swifttype file;

app (file output) getLandUse (file input){ getLandUse @input;}

file getLandUse_inputs[] <filesys_mapper; pattern=”images/*.rgb">;

foreach i in getLandUse_inputs { file output <single_file_mapper; file=@strcat(i, “.output”)>; output = getLandUse(i);}

Same script runs on broad range of resources; separate throttles can be set for each site.

User can run on multiple resources:

4www.swiftlang.org

Cluster interactive node

swift

Output

Logs

Cluster file server

Input

Swiftscript

Applications

configfiles

UChicago UC3

UC3 seeder nodes

Midwest T2 nodes

OSG VO nodes

Beagle CrayCray XE 24-core nodes

Uchicago Midway

Sandybridge nodes

Westmere nodes

Department nodes

Swift’s location-independent scripting lets the user focus on science Example of running 3,000 jobs to 3 hosts including the UC3 campus

collective:

The user started on a basic login host processing 10 files and moved up to a 3,000 file dataset, changing only the dataset name and a site-specification list to get to the resources above

Expanded the scope of their computations from one node to hundreds or thousands of cores

User didn’t need to look at what sites were busy, or adjust arcane scripts, to get to these resources.

5

www.swiftlang.org

Midway 289Beagle   1070UC3      1641Total    3000

Swift is a parallel scripting system for grids, clouds and clusters– for loosely-coupled applications - application and utility programs linked by

exchanging files Swift is easy to write: simple high-level C-like functional language

– Small Swift scripts can do large-scale work Swift is easy to run: contains all services for running Grid workflow - in one

Java application– Untar and run – acts as a self-contained Grid client

Swift is fast: uses efficient, scalable and flexible distributed execution engine.– Scales to range of 1M tasks per script run

Swift usage is growing:– applications in earth systems sciences, neuroscience, proteomics, molecular

dynamics, biochemistry, economics, statistics, and more. Try Swift: www.swiftlang.org

6www.swiftlang.org

7Parallel Computing, Sep 2011www.swiftlang.org

Face-IT: Framework to Advance Climate, Economic, and Impact Investigations with Information Technology

8

www.ci.uchicago.edu/swift www.mcs.anl.gov/exm

• Collaboration between University of Chicago, Columbia University, Purdue University, and the University of Florida

• Galaxy – scientific workflow system

• Combine the best parts of Swift and Galaxy, and apply to Earth Systems

Galaxy Screenshots

9

www.swiftlang.org

Galaxy Screenshots

10

www.swiftlang.org