Upload
marylou-dennis
View
226
Download
0
Embed Size (px)
Citation preview
How to Compute and Prove
Lower and Upper Boundson the
Communication Costsof Your Algorithm
Part III: Graph analysis
Oded Schwartz
CS294, Lecture #10 Fall, 2011Communication-Avoiding Algorithms
www.cs.berkeley.edu/~odedsc/CS294
Based on:
G. Ballard, J. Demmel, O. Holtz, and O. Schwartz:
Graph expansion and communication costs of fast matrix multiplication.
2
Previous talk on lower bounds Communication Lower Bounds:
Approaches:
1. Reduction [Ballard, Demmel, Holtz, S. 2009]
2. Geometric Embedding[Irony,Toledo,Tiskin 04],
[Ballard, Demmel, Holtz, S. 2011a]
3. Graph Analysis [Hong & Kung 81], [Ballard, Demmel, Holtz, S. 2011b]
Proving that your algorithm/implementation is as good as it gets.
3
Previous talk on lower bounds: algorithms with “flavor” of 3 nested loops[Ballard, Demmel, Holtz, S. 2009],[Ballard, Demmel, Holtz, S. 2011a]Following [Irony,Toledo,Tiskin 04]
• BLAS, LU, Cholesky, LDLT, and QR factorizations, eigenvalues and singular values, i.e., essentially all direct methods of linear algebra.
• Dense or sparse matricesIn sparse cases: bandwidth is a function NNZ.
• Bandwidth and latency.• Sequential, hierarchical, and
parallel – distributed and shared memory models.• Compositions of linear algebra operations.• Certain graph optimization problems
[Demmel, Pearson, Poloni, Van Loan, 11]• Tensor contraction
M
M
n3
P
M
M
n3
4
Geometric Embedding (2nd approach) [Ballard, Demmel, Holtz, S. 2011a]Follows [Irony,Toledo,Tiskin 04], based on [Loomis & Whitney 49]
(1) Generalized form: (i,j) S, C(i,j) = fij( gi,j,k1 (A(i,k1), B(k1,j)),
gi,j,k2 (A(i,k2), B(k2,j)), …, k1,k2,… Sij
other arguments)
But many algorithms just don’t fit the generalized form!
For example: Strassen’s fast matrix multiplication
5
Beyond 3-nested loops
How about the communication costs of algorithmsthat have a more complex structure?
6
Communication Lower Bounds
Approaches:
1. Reduction [Ballard, Demmel, Holtz, S. 2009]
2. Geometric Embedding[Irony,Toledo,Tiskin 04],
[Ballard, Demmel, Holtz, S. 2011a]
3. Graph Analysis [Hong & Kung 81], [Ballard, Demmel, Holtz, S. 2011b]
Proving that your algorithm/implementation is as good as it gets.
7
[Strassen 69]• Compute 2 x 2 matrix multiplication
using only 7 multiplications (instead of 8).• Apply recursively (block-wise)
M1 = (A11 + A22) (B11 + B22)M2 = (A21 + A22) B11
M3 = A11 (B12 - B22)M4 = A22 (B21 - B11)M5 = (A11+ A12) B22
M6 = (A21 - A11) (B11 + B12)M7 = (A12 - A22) (B21 + B22)
C11 = M1 + M4 - M5 + M7
C12 = M3 + M5
C21 = M2 + M4
C22 = M1 - M2 + M3 + M6
Recall: Strassen’s Fast Matrix Multiplication
C21 C22
C11 C12n/2
n/2 A21 A22
A11 A12
B21 B22
B11 B12
=
T(n) = 7T(n/2) + O(n2)
T(n) = (nlog27)
8
Strassen-like algorithms
• Compute n0 x n0 matrix multiplication using only n0
0 multiplications (instead of n0
3).
• Apply recursively (block-wise)0 2.81 [Strassen 69] works fast in practice.2.79 [Pan 78]2.78 [Bini 79]2.55 [Schönhage 81]2.50 [Pan Romani,Coppersmith Winograd 84]2.48 [Strassen 87]2.38 [Coppersmith Winograd 90] 2.38 [Cohn Kleinberg Szegedy Umans 05] Group-theoretic approach
T(n) = n00 T(n/n0) + O(n2)
T(n) = (n0)
n/n0
=
9
New lower bound for Strassen’s fast matrix multiplication
[Ballard, Demmel, Holtz, S. 2011b]:The Communication bandwidth lower bound is
M
M
n7log2
M
M
n 0
M
M
n8log2
P
M
M
n7log2
P
M
M
n 0
P
M
M
n8log2
Strassen-like: Recall for cubic:For Strassen’s:
The parallel lower bounds applies to2D: M = (n2/P)2.5D: M = (c∙n2/P)
log2 7 log2 80
10
For sequential? hierarchy?Yes, existing implementation do!
For parallel 2D? parallel 2.5D?Yes: new algorithms.
11
Sequential and new 2D and 2.5D parallel Strassen-like algorithms
Sequential and Hierarchy cases: Attained by the natural recursive implementation.
Also: LU, QR,… (Black-box use of fast matrix multiplication)
[Ballard, Demmel, Holtz, S., Rom 2011]: New 2D parallel Strassen-like algorithm.
Attains the lower bound.
New 2.5D parallel Strassen-like algorithm.c 0 /2-1 parallel communication speedup over 2D implementation (c ∙ 3n2 = M∙P)
[Ballard, Demmel, Holtz, S. 2011b]:This is as good as it gets.
Implications for sequential architectural scaling
• Requirements so that “most” time is spent doing arithmetic on n x n dense matrices, n2 > M:
• Time to add two rows of largest locally storable square matrix exceeds reciprocal bandwidth
• Time to multiply 2 largest locally storable square matrices exceeds latency
Strassen-like algs do fewer flops & less communication but are more demanding on the hardware.If 2, it is all about communication.
CA Matrix multiplication algorithm
Scaling BandwidthRequirement
Scaling LatencyRequirement
Classic M1/2 M3/2
Strassen-like M0/2-1 M0/2
13
Let G = (V,E) be a d-regular graph
A is the normalized adjacency matrix, witheigenvalues 1 ≥ 2 ≥ … ≥ n
1 - max {2, | n|}
Thm: [Alon-Milman84, Dodziuk84, Alon86]12 2h
\V SS
Expansion (3rd approach) [Ballard, Demmel, Holtz, S. 2011b], in the spirit of [Hong & Kung 81]
Sd
SSEh V
SS
,min
2,
RS
WS
S
14
The Computation Directed Acyclic Graph
Expansion (3rd approach)
Communication-cost is Graph-expansion
Input / OutputIntermediate valueDependency
\V SSV
15
For a given run (Algorithm, Machine, Input)
1. Consider the computation DAG: G = (V, E)V = set of computations and inputsE = dependencies
2. Partition G into segments S of (M/2) vertices(correspond to time / location adjacency)
3. Show that every S has 3M vertices with incoming / outgoing edges perform M read/writes.
4. The total communication BW isBW = BW of one segment #segments = (M) O(n) / (M/2) = (n / M/2 -1)
MM
MM
M
S
RS
WS
V
Expansion (3rd approach)
S1
S2
S3
Read
Read
Read
Read
Read
Read
Write
Write
Write
FLOP
FLOP
FLOP
FLOP
FLOP
FLOP
Tim
e
...
16
Is it a Good Expander?
Break G into edge-disjoint graphs, corresponding to the algorithm on M1/2 M1/2 matrices.Consider the expansions of S in each part (they sum up).
S1
S2 S3
S5
S4
We need to show that M/2 expands to (M).
h(G(n)) = (M/ M/2) for n = (M1/2).
Namely, for every n, h(G(n)) = (n2/n) = ((4/7)lg n)
BW = (T(n)) h(G(M1/2))
BW = (T(n)) (G(M1/2))
Enlg n BEnlg nA
Declg nC
n2
n2
n
lg n
17
What is the CDAG of Strassen’s algorithm?
18
M1 = (A11 + A22) (B11 + B22)M2 = (A21 + A22) B11
M3 = A11 (B12 - B22)M4 = A22 (B21 - B11)M5 = (A11+ A12) B22
M6 = (A21 - A11) (B11 + B12)M7 = (A12 - A22) (B21 + B22)
C11 = M1 + M4 - M5 + M7
C12 = M3 + M5
C21 = M2 + M4
C22 = M1 - M2 + M3 + M6
The DAG of Strassen, n = 2
`
7 5 4 1 3 2 6
1,1 1,2 2,1 2,2
1,1 1,2 2,1 2,21,1 1,2 2,1 2,2
Enc1A
Dec1C
Enc1B
`
19
The DAG of Strassen, n=4
Dec1C1,1 1,2 2,1 2,2
7 5 4 1 3 2 6
One recursive level:
• Each vertex splits into four.
• Multiply blocks
Enc1 BEnc1A
Dec1C
Enc1A Enc1B
20
Enclg n BEnclg nA
Declg nC
n2
n2
n
lg n
Dec1C
The DAG of Strassen: further recursive steps
1,1 1,2 2,1 2,2
Recursive construction
Given DeciC, Construct Deci+1C:
1. Duplicate 4 times
2. Connect with a cross-layer of Dec1C
21
Enlg nBEnlg nA
Declg nC
n2
n2
n
lg n
The DAG of Strassen
1. Compute weighted sums of A’s elements.
2. Compute weighted sums of B’s elements.
3. Compute multiplications m1,m2,…,m.
4. Compute weighted sums of m1,m2,…,m to obtain C.
A B
C
22
Expansion of a Segment
Two methods to compute the expansion of the recursively constructed graph:
• Combinatorial- estimate directly the edge / vertex expansion (in the spirit of [Alon, S., Shapira, 08])
or• Spectral
- compute the edge expansion via the spectral-gap(in the spirit of the Zig-Zag analysis [Reingold, Vadhan, Wigderson 00])
23
Expansion of a Segment
Main technical challenges:
• Two types of vertices: with/without recursion.
• The graph is not regular.
`
7 5 4 1 3 2 6
1,1 1,2 2,1 2,2
1,1 1,2 2,1 2,21,1 1,2 2,1 2,2
Enc1A
Dec1C
Enc1B
24
Estimating the edge expansion- Combinatorially
SkS1
lg 1k M
S3S2 M M
• Dec1C is a consistency gadget: Mixed pays 1/12 of its edges.
• The fraction of S vertices is consistent between the 1st level and the four 2nd levels (deviations pay linearly).
In S
Not in S
Mixed
25
Communication Lower Bounds
Approaches:
1. Reduction [Ballard, Demmel, Holtz, S. 2009]
2. Geometric Embedding[Irony,Toledo,Tiskin 04],
[Ballard, Demmel, Holtz, S. 2011a]
3. Graph Analysis [Hong & Kung 81], [Ballard, Demmel, Holtz, S. 2011b]
Proving that your algorithm/implementation is as good as it gets.
26
Open Problems
Find algorithms that attain the lower bounds:• Sparse matrix algorithms• for sequential and parallel models• that auto-tune or are cache oblivious
Address complex heterogeneous hardware:• Lower bounds and algorithms
[Demmel, Volkov 08],[Ballard, Demmel, Gearhart 11]
Extend the techniques to other algorithm and algorithmic tools:• Non-uniform recursive structure
Characterize a communication lower bound for a problem rather than for an algorithm.
\V SS
?
How to Compute and Prove
Lower Boundson the
Communication Costsof Your Algorithm
Part III: Graph analysis
Oded Schwartz
CS294, Lecture #10 Fall, 2011Communication-Avoiding Algorithms
Based on:
G. Ballard, J. Demmel, O. Holtz, and O. Schwartz:
Graph expansion and communication costs of fast matrix multiplication.
Thank you!