Histograms detect imbalances

Variable sizes capture variance

Lawrence LivermoreNational Laboratory

Center for Applied Scientific Computing

NC STATE UNIVERSITYDepartment of Computer Science

An Open Framework for Scalable, Reconfigurable Performance AnalysisTodd Gamblin‡, Prasun Ratn*†, Bronis R. de Supinski*, Martin Schulz*, Frank Mueller†, Robert J. Fowler‡, Daniel Reed‡

*Lawrence Livermore National Laboratory, †North Carolina State University, ‡Renaissance Computing Institute

This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.This work was also supported in part by NSF grants CNS-0410203, CCF-0429653 and CAREER CCR-0237570, as well as the SciDAC Performance Engineering Research Institute, DE-FC02-06ER25764.

UCRL-POST-236200

ScalaTrace: Reconfigurable,Scalable Performance Analysis

Size of machines is rapidly increasing(130,000+ processors)

Tools will be overwhelmed with data Need scalable, online measurement and analysis

Future Directions

Large Scale MPI Applications Implicit Program Behavior

Interactive User Tools and Visualization Understanding of Program Behavior and Optimizations

Feedback, Steering & G

lobal Configuration

Dat

a C

olle

ctio

n &

Ana

lysi

s

…MPI tasks

User Interaction& Storage

ReductionNodes

Local Configuration

Data Input: from Tracer, Profiler, Child Nodes

Data Output: to Parent Node, Tool

Communication pathsOPS Attr.1 Attr.2 Attr.N…

extensible, self-describing formatAttribute examples: PRSDs, timing, …Reducer

Transformer

Annotater

Analyzer

Compressor PRSDs Timing

Loop Detection

Annotator

MPI PRSD Time Loop

MPI PRSD Time

MPI PRSD Time

MPI PRSD Time

LoopInformation

Time

PRSD

Types of tool components

LOCAL Node INSTANTIATION

GLOBALREDUCTION

Trace representation using Power Regular Section Descriptors (PRSDs)

Idea: preserve time in compressed traces

• Encode time deltas instead of timestamps

• Create delta histograms automatically

• Dynamically balance histograms

•Time depends on path taken

•Distinguish histograms by path

Description of problem

Bins generated for synthetic input spanentire range with similar sample counts

Number of histograms per record depends onnumber of possible call paths

Sample bimodal distribution from UMT2K collectives

... MPI _Allreduce ( ... ) ; for ( ... ) { for ( ... ) { MPI _Send ( ... ) ; MPI _Recv ( ... ) ; } MPI _Barrier ( ... ) ; } ...

Replay accuracy (NAS Benchmarks and UMT2K)

IS

LU

FT

IS

FT

Trace sizes (NAS Benchmarks and UMT2K)

MG

BT

Near-constant trace size

DT, EP, LU

Sub-linear trace sizeCG, MG, FT

Non-scalable trace sizeBT, IS, UMT2K

Accurate replay: DT, EP,FT, LU, IS, UMT2K

Replay inaccurate in MPITime: CG, MG

Replay inaccurate inCompute Time: BT

ScalaReplay: Replay UsingHistogram Timing Annotations

Evolutionary Load-Balance Analysiswith Scalable Data Collection

Idea: Normalize measurements andmodels based on applicationsemantics.

Progress loops• Typically outer loops in SPMD codes• Indicate absolute progress towards some

domain-specific goal• Basis for comparison of load over time

Effort Loops• Variable-time loops, represent load• Data-dependent execution

Load Modeling Framework

Visualization

Effort Region Analysis

Progress EventAnnotator

MPI PRSD

Compressor

Trace merge WaveletCompression

MPI PRSD Progress

LocalEffort

Records

PRSDs

MPI

PRSD

MPI Tracer

Load-annotatedTrace

MPI PRSD Progress

Progress instrumentation• User marks progress loop explicitly

Effort modeled with code regions• Dynamically detected at runtime• Split MPI-op trace at collectives and wait operations• User can further divide code into phases with instrumentation

Models dislocation dynamics in crystals• Dislocations discretized as nodes & arms• Recursive spatial domain decomposition• Balancer subdivides nodes/arms along x, then y, then z

Com

press

. . .

. . .

. . .

Scalable Compression Using Wavelet TransformDistributed per-process effort measurements areconsolidated onto fewer processes

Parallel wavelet transform

Transformed data is locally compressed bythresholding, run-length and entropy coding

Compressed representation is merged in parallel

• Load measurements are reconstructed on client• Visualization tool allows intuitive data browsing

Com

press

Com

press

Near-constant low-overhead MPI traces

Annotate with additional reconfigurable data, e.g.

Time using adaptive histograms (left)

Progress rates, load imbalance (right)

Compression Ratio

Merge Time

Error

Process IdProgress

Process IdProgress

Process IdProgressProcess IdProgress

Process IdProgress

Process IdProgress

Checkpoint

ForceComputation

Effort for 64-node ParaDiS RunExact Reconstructed

Re-partition

Load Balance in ParaDiSz

x

y

Flexible framework for application-specific tools

Ability to select compression schemes fordifferent fields, data types

Foster combination, interaction between datacollection, analysis mechanisms

Adaptive, self-tuning runtime systems

Near term:

Adaptive wavelet transform topology,equivalence class detection

Full time-sequence compression