[Harvard CS264] 14 - Dynamic Compilation for Massively Parallel Processors (Gregory Diamos, Georgia Tech)

Dynamic Compilation for Massively Parallel

Processors

Gregory Diamos

PhD candidate

Georgia Institute of Technology and NVIDIA Research

April 14, 2011

Gregory Diamos CS264 - Dynamic Compilation 1/62

What is an execution model?


Goals of programming languages

Programming languages are designed for productivity.

Efficiency is measured in terms of:1 cost - hardware investment, power consumption, area requirement2 complexity - application development effort3 speed - amount of work performed per unit time


Goals of processor architecture

Hardware is designed for speed and efficiency.


Goals of processor architecture - 2

[1] - M. Koyanagi, T. Fukushima, and T. Tanaka. "High-Density Through Silicon Vias for 3-D LSIs" [2] - Novoselov et al. "Electric Field Effect in Atomically Thin Carbon Films." [3] - Intel Corp. 22nm test chip.

It is constrained by the limitations of physical devices.


Execution models bridge the gap


Goals of execution models

Execution models provide impedance matching between applications andhardware.

Goals:

leverage common optimizations across multiple applications.

limit the impact of hardware changes on software.

ISAs have traditionally been effective execution models.


Programming challenges of heterogeneity

The introduction of heterogeneous and multi-core processors changes thehardware/software interface:

Intel Nehalem IBM PowerEN AMD Fusion NVIDIA Fermi

1 multi-core creates multiple interfaces.2 heterogeneity creates different interfaces.3 these increase software complexity.


Program the entire processor, not individual cores.(new execution model abstractions are needed)


Emerging execution models


Bulk-synchronous parallel (BSP)

[1] - Leslie Valiant. A bridging model for parallel computing.


The Parallel Thread eXecution (PTX) Model

PTX defines a kernel as a 2-level grid of bulk-synchronous tasks.


Dynamically translating PTX

Dynamic compilers can transform this parallelism to fit the hardware.


Beyond PTX - Data distributions


Beyond PTX - Memory hierarchies

[1] - Leslie Valiant. A bridging model for multi-core.[2] Fatahalian et al. Sequoia: Programming the memory hierarchy.


Dynamic compilation/binarytranslation


Binary translation


Binary translators are everywhere

If you are running a browser, you are using dynamic compilation.


x86 binary translation


Low Level Virtual Machines

Compile all programs to a common virtual machine representation (LLVMIR), keep this around.

Perform common optimizations on this IR.

Target various machines by lowering it to an ISA.

Statically or via JIT compilation.


Execution model translation


Execution model translation

Extend binary translation to execution model translation.

Dynamic compilers can map threads/tasks to the HW.


Different core architectures

Can we target these from the same execution model.

What about efficiency?


Ocelot

Enables thread-aware compiler transformations.


Mapping CTAs to cores - thread fusion

Scheduler Block

Restore Registers

Spill Registers

Barrier

Original PTX Code Transformed PTX Code

Transform threads into loops over the program.

Distribute loops to handle barriers.


Mapping CTAs to cores - vectorization

Pack adjacent threads into vector instructions.

Speculate that divergence never occurs, check in case it does.


Mapping CTAs to cores - multiple instruction streams

T0 T1 T2 T3

Instructions from different threads are independent.

merge instruction streams and statically schedule on functional units.


PTX analysis


Divergence analysis


Subkernels

subkernel


Thread frontier analysis

Supporting control flow on SIMD processors requires finding divergentbranches and potential re-converge points.

if((cond1() || cond2()) && (cond3() || cond4())){ ...}

bra cond1() bra cond3()

bra cond2() bra cond4()....

entry

exit

compound conditionals

short circuit control flow

bra cond1()

bra cond3()

bra cond2()

bra cond4()

....

entry

exit

B1

B2

B3

B4

B5

Thread FrontiersBlock Id

{}

{B2 - B3}

{B3 - Exit}

{B4 - Exit}

{B5 - Exit}

T0 T1 T2 T3 T0 T1 T2 T3

thread-frontier reconvergence

of T0

thread-frontier reconvergence

of T2

post dominatorreconvergenceof T1 and T3

Push B3 on T0

Push Exit on T1

Push B5 on T2

re-convergence at thread frontiers

immediate post-dominator re-convergence

Push Exit on T4

Pop stack Exit on T4

Pop stack switch to B5 on T2

Pop stack switch to B3 on T0

post dominatorreconvergence

of T1, T2, and T3

Pop stack Exit on T1

Compiler analysis can identify immediate post donimators orthread-frontiers as re-convergence points.


Consequences of architecturedifferences


Degraded performance portability

0 1000 2000 3000 4000 5000 6000

0100

200

300

400

500

600

GF

LO

PS

N

Fermi SGEMM

AMD SGEMM

0 1000 2000 3000 4000 5000 6000

0200

400

600

800

1000

1200

1400

1600

GF

LO

PS

N

Fermi SGEMM

AMD SGEMM

Performance of two OpenCL applications, one tuned for AMD, the otherfor NVIDIA.


Memory traversal patterns

Warp(4) Cycle 1 Warp(4) Cycle 2

Warp(1) Cycle 1 Warp(1) Cycle 2

Thread loops change row major memory accesses to column majoraccesses.


Reduced memory bandwidth on CPUs

Optimized for SIMD (GPU)

Optimized for single-threaded CPU

This reduces memory bandwidth by 10x for a memory microbenchmarkrunning on a 4-core CPU.


The good news


Scaling across three decades of processors

Many existing applications still scale.

12x

480x

280GTX has 40x more peak flops than a Phenom, 480x more than anAtom.


Questions?


Databases on GPUs


Who cares about databases?


What do applications look like?

What do applications look like?


Gobs of data


Distributed systems


Lots of parallelism


What do CPU algorithms look like?

What do cpu algorithms looklike?


Btrees


Sequential algorithms

=

<

>

relation 1

relation 2

result


It doesn’t look good

Outlook not so good...


Or does it?

Where is the parallelism?


Flattened trees


Relational algebra


Inner Join

A Case Study: Inner Join


1. Recursive partitioning


2. Block streaming

Blocking into pages, shared memory buffers, and transaction sizedchunks makes memory accesses efficient.


3. Shared memory merging network

A network for join can be constructed, similar to a sorting network.


4. Data chunking

Stream compaction packs result data into chunks that can be streamedout of shared memory efficiently.


Operator fusion


Will it blend?


Yes it blends.

Operator NVIDIA C2050 Phenom 9570

inner-join 26.4-32.3 GB/s 0.11-0.63 GB/sselect 104.2 GB/s 2.55 GB/sset operators 45.8 GB/s 0.72 GB/sprojection 54.3 GB/s 2.34 GB/scross product 98.8 GB/s 2.67 GB/s


Questions?


Conclusions

Emerging heterogeneous architectures need matching execution modelabstractions.

dynamic compilation can enable portability.

When writing massively parallel codes, consider:

data structures and algorithms.

mapping onto the execution model.

transformations in the compiler/runtime.

processor micro-architecture.


Thoughts on open source software


Questions?

Questions?

Contact Me:

[email protected]

Contribute to Harmony, Ocelot, and Vanaheimr:

http://code.google.com/p/harmonyruntime/

http://code.google.com/p/gpuocelot/

http://code.google.com/p/vanaheimr/


Education

[Harvard CS264] 14 - Dynamic Compilation for Massively Parallel Processors (Gregory Diamos, Georgia Tech)