Multi-component Modeling with Swift at Extreme Scale

www.ci.anl.govwww.ci.uchicago.edu

1www.ci.anl.govwww.ci.uchicago.edu

Multi-component Modeling with Swift at Extreme Scale

Daniel S. Katz, Justin Wozniak, Michael Wilde, and Ian Foster

Computation Institute, University of Chicago & Argonne National Laboratory

http://swift-lang.org/

Supercomputing Frontiers 2015

March 17, 2015, Singapore

http://slideshare.com/danielskatz


2

Swift Domain of Interest

“Time”

“Co

mp

lexi

ty”

Increasing capabilities in computational science



3

Extreme-scale application space

Single large problem; e.g. PDE solution over single domain

Problem composed of large number of sub-problems; e.g. parameter sweep

Problem composed of small number of sub-problems; e.g. components, ensemble

Extreme-scale application space

Application space Swift addresses


Today’s hero problems

Yesterday’s hero problems


4

Climate models

• Built from components


Ocean

Atmosphere

Land

Sea ice


5

Methods of coupling

• Coupling library – add to each component– E.g. Model Coupling Toolkit

(http://www.mcs.anl.gov/research/projects/mct/)

• Coupling component – add new component– E.g. CSM Flux Coupler

(http://www.cesm.ucar.edu/models/cpl/)

• Using data store (in RAM) or file system (on disk)– E.g., DataSpaces

(http://personal.cac.rutgers.edu/TASSL/projects/data/index.html) and ADIOS (https://www.olcf.ornl.gov/center-projects/adios/)

• Traditional usage: climate, fusion, REMD• Newer usage: in situ analysis



6

Climate component models


Credit: Kaitlin Alexander & Steve Easterbrook, “The Software Architecture of Global Climate Models,” U. Toronto


7

One climate model


Credit: Kaitlin Alexander & Steve Easterbrook, “The Software Architecture of Global Climate Models,” U. Toronto


8

Multicomponent models for multiscale

Finite Element (microstructure, solve for k)

Phase Field (evolve microstructure)

),,( 11 iii tTrk

TTrk ),(0

QTkt

TCP

Finite Element (fuel element model, solve for T)

),( ii trT

newoldold tkT ,,

ii kk 1

ii TT 1

newnewnew tkT ,,

newkYes

Yes

No

No

newT


Credits:

FEM: B. Mihaila, et al., J. Nucl. Mater., 394 (2009) 182-189

PF: S.Y. Hu et al., J. Nucl. Mater. 392 (2009) 292–300

Nuclear fuel rod workflow: Marius Stan, Argonne


9

Types of coupling

• Loose (through files, sequential order to components)

• Tighter (through messages over WAN, distributed simultaneous components)

• Tight (through [MPI] messages over LAN/local IC, parallel simultaneous components)

Model1(MS A) File

Model2(MS A|B)File


Model1(MS A)

Model2(MS B)

Msg

Model1(MS A)

Model2(MS A)

Msg


10

Component state

• Do components have state?– No

o Init, run, stop

o Communication with other components before and after run, communication can be outside component lifecycle

– Yeso Init, run, run, run, run, stop

o Communication with other components before and after run, communication must be inside component lifecycle

• Can this be distinct from component execution?– “Stateless” component can remain in memory, if

each “run” depends fully on external inputs



11

Swift http://swift-lang.org/

• Swift is a parallel scripting system for grids, clouds and clusters

• F() and G() are computed in parallel– Can be Swift functions, or leaf tasks (executables or scripts in

shell, python, R, Octave, MATLAB, ...)

• r computed when they are done

• This parallelism is automatic

• Works recursively throughout the program’s call graph


(int r) myproc (int i)

{

int f = F(i);

int g = G(i);

r = f + g;

}


12

Swift Variables, tasks, files, concurrency

• Variables are single assignment futures– Unassigned variables are open

• Variables can represent files– When a file doesn’t exist, the variable is open

– When a file exists, the variable is closed

• All tasks found at runtime

• Tasks with satisfied dependencies (all inputs are closed variables) are run on whatever resources are available

• These runs create files/variables that allow more tasks to run



13

Advanced Photon Source (APS)

• Moves electrons at electrons at >99.999999% of the speed of light.• Magnets bend electron trajectories, producing x-rays, highly focused

onto a small area• X-rays strike targets in 35 different laboratories – each a lead-lined,

radiation-proof experiment station• Beam time allocated in multi-day chunks; very valuable!



14

High Energy Diffraction Microscopy (HEDM)

Used to non-destructively find crystal structure of metals at Argonne Advanced Photon Source (APS)

Near Field (NF)-HEDM grain identification for cross section of gold wire.L: colors are grains, R: colors are confidence

Confidence obtained by larger amount of analysis

Far field (FF)-HEDM grain identification for cross section of

experimental material.Colors are grain centers

Credit: Hemant Sharma, Jonathan Almer, Materials Physics and Engineering Group, X-Ray Science Division, ANL. Justin Wozniak, MCS and CI, ANL



15

NF-HEDM Using Mira via SwiftSingle integrated cross-system script – 4 GB processed every 4-10 mins, during beam time

320-node X86 cluster

Up to 8192 nodes used

Credit: Hemant Sharma, Jonathan Almer, Materials Physics and Engineering Group, X-Ray Science Division, ANL. Justin Wozniak, MCS and CI, ANL



16

Parallel tasks in Swift


• Swift expression: z = @par=8 f(x,y);

• When x, y are stored, Turbine releases task f with parallelism=8

• Performs ADLB_Put(f, parallelism=8)

• Each worker performs ADLB_Get(&task, &comm)

• ADLB server finds 8 available workers

• Workers receive ranks from ADLB server, which performsMPI_Comm_create_group()

• ADLB_Get() returns:task=f, size(comm)=8

• Workers perform user task– Communicate on comm

• comm is freed by Turbine

• Task data can be– in-memory objects stored in ADLB

– file stored on disk, file vars in ADLB

• Files can be stored in shared disk, or locally using MosaStore, Hercules, etc.


17

Simple climate model in Swift

• Swift yesterday (pseudocode)

• atm_extract and ocean_extract can run in parallel, e.g. each on n/2 cores

• atm_step and ocean_step can run in parallel, each on n/2 cores (or load balanced), or in sequence, each on n cores

• May use ADIOS or similar to handle MxN I/O


file: atm_state[], ocean_state[], wind[], sst[];

for t ...

{

wind[t] = atm_extract(atm_state[t]);

sst[t] = ocean_extract(ocean_state[t]);

atm_state[t+1] = atm_step(atm_state[t], stt[t]);

ocean_state[t+1] = ocean_step(ocean_state[t],

wind[t]);

}


18

More complex climate model in Swift

• Swift today (pseudocode)

• Functions still need to init, run, exit each step

• Functions can be compiled into single executable, rather than re-launched


file: atm_state0[x,y,z], ocean_state0[x,y,z];

float: atm_state[x,y,z][], ocean_state[x,y,z][], wind[x,y][],

sst[x,y][];

atm_state[][0] = read_atm(atm_state0[]);

ocean_state[][0] = read_ocean(ocean_state0[]);

for t ...

{

wind[][t] = atm_extract(atm_state[][t]);

sst[][t] = ocean_extract(ocean_state[][t]);

atm_state[][t+1] = atm_step(atm_state[][t], stt[][t]);

ocean_state[][t+1] = ocean_step(ocean_state[][t], wind[][t]);

}


19

More complex climate model in Swift

• Swift tomorrow – work on language and runtime needed (pseudocode)

• Function state & variables cached in node memory

• Maybe make use of local FLASH/storage


file: atm_state0[x,y,z], ocean_state0[x,y,z];

float: atm_state[x,y,z][], ocean_state[x,y,z][], wind[x,y][],

sst[x,y][];

atm_state[][0] = read_atm(atm_state0[]);

ocean_state[][0] = read_ocean(ocean_state0[]);

for t ...

{

wind[][t] = atm_extract(atm_state[][t]);

sst[][t] = ocean_extract(ocean_state[][t]);

atm_state[][t+1] = atm_step(atm_state[][t], stt[][t]);

ocean_state[][t+1] = ocean_step(ocean_state[][t], wind[][t]);

}


20

NAMD/VMD and Swift/T


Typical Swift/T Structure

MD1.cMD1.c MD2.cppMD2.cpp viz.cppviz.cpp

SWIG-generated Tcl wrappersSWIG-generated Tcl wrappers

Swift/T runtimeSwift/T runtimeMPI

Top-level dataflow scriptexchange.swift


NAMD/VMD Structure

Swift/T runtimeSwift/T runtime

NAMD (C++)NAMD (C++)

Tcl Evaluation (uplevel-eval)Tcl Evaluation (uplevel-eval)



Credit: Phillips et al., Petascale Tcl with NAMD, VMD, and Swift/T, HPTCDL 2014

• NAMD and VMD embed Tcl, and thus can run Swift dataflow programs

• Swift has been used to demonstrate n:m multiplexing of n replicas across a smaller arbitrary number m of NAMD processes

• This is very complex to do with normal NAMD scripting but can be expressed naturally in under 100 lines of Swift/T code


21

Conclusions

• Multicomponent models increasingly important for science

• Swift: effective means of programming and running many task computing (MTC) applications– MTC model traditionally based on read (file), run

(executable), write (file) model

– Now extended to many parallel tasks, read & write to data store, run functions

• Swift demonstrated for multicomponentmodeling in MD, material science



22

Future work

• Abstractions– For hierarchically complex, concurrent application components– For stateful tasks across functional programs

o Build on Swift's growing location and topology awareness

– For events and exceptions

• Language– How to express/hint that a function and/or its state should remain

local?– Relearn lessons from CCA, Babel, etc.– Expose topology features with dynamic programming constructs

• Runtime– How to implement the above?

o Cache/swap process and data in memory, FLASH, local disk?



23

Further research directions

• Orchestrating in-situ analytics for extreme-scale simulations

In-situ analytics of a climate simulation



24

Acknowledgements

• Swift (http://swift-lang.org/) is supported in part by NSF grant OCI-1148443

• Extreme scaling research on Swift (ExM project) was supported by the DOE Office of Science, ASCR Division, X-Stack program

• Additional support is provided by the Argonne Leadership Computing Facility and the Argonne Director’s Grand Challenge Project “Discovery Engines for Big Data”

• The Swift team (mostly UC/ANL):– Tim Armstrong, Yadu Nand Babuji, Ian Foster, Mihael

Hategan, Dan Katz, David Kelly, Ketan Maheshwari, Ioan Raicu(IIT), Michael Wilde, Justin Wozniak, Sophia Yang

• Special thanks to all Swift users and collaborators



25

Thank you!

Questions?


Technology

Multi-component Modeling with Swift at Extreme Scale