Indirect Networks or Dynamic Networks

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

Indirect Networks orDynamic Networks

Guihai Chen

…with major presentation contribution from José Flich, UPV(and Cell BE EIB slides by Tom Ainsworth, USC)

Interconnection Networks

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

2

Questions in mind

Difference between Static/direct and Dynamic/indirect Networks

Why Multi-Stage Interconnection Networks

Large Switch and Small Switch

How to design non-blocking MINs

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

3

Outline

Network Topology

Preliminaries and Evolution

Centralized Switched (Indirect) Networks

From non-blocking crossbar to blocking MINs

From blocking MINs to non-blocking MINs

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

4

Network Topology

Preliminaries and Evolution• One switch suffices to connect a small number of devices

– Number of switch ports limited by VLSI technology, power consumption, packaging, and other such cost constraints

• A fabric of interconnected switches (i.e., switch fabric or network fabric) is needed when the number of devices is much larger

– The topology must make a path(s) available for every pair of devices—property of connectedness or full access (What paths?)

• Topology defines the connection structure across all components– Bisection bandwidth: the minimum bandwidth of all links crossing

a network split into two roughly equal halves

– Full bisection bandwidth: › Network BWBisection = Injection (or Reception) BWBisection= N/2

– Bisection bandwidth mainly affects performance• Topology is constrained primarily by local chip/board pin-outs;

secondarily, (if at all) by global bisection bandwidth

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

5

Network Topology

Preliminaries and Evolution• Several tens of topologies proposed, but less than a dozen used• 1970s and 1980s

– Topologies were proposed to reduce hop count• 1990s

– Pipelined transmission and switching techniques

– Packet latency became decoupled from hop count• 2000s

– Topology still important (especially OCNs, SANs, P2P Overlays, DCNs) when N is high

– Topology impacts performance and has a major impact on cost

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

6

Network Topology

Centralized Switched (Indirect) Networks• Crossbar network

– Crosspoint switch complexity increases quadratically with the number of crossbar input/output ports, N, i.e., grows as O(N2)

– Has the property of being non-blocking

7

6

5

4

3

2

1

0

76543210

7

6

5

4

3

2

1

0

76543210

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

7

Network Topology

From Crossbar to MINs

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

8

Network Topology

Centralized Switched (Indirect) Networks• Multistage interconnection networks (MINs)

– Crossbar split into several stages consisting of smaller crossbars

– Complexity grows as O(N × log N), where N is # of end nodes

– Inter-stage connections represented by a set of permutation functions

Omega topology, perfect-shuffle exchange

7

6

5

4

3

2

1

0

7

6

5

4

3

2

1

0

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

9

Network Topology

Appendix • Shuffle function

– N= 0…N-1

– f(i)= 2i, when I <N/2

– f(i)=(2i+1), mod N when i ≥ N/2

– Often used as a connection pattern

– unshuffle function, also often used

perfect-shuffle

7

6

5

4

3

2

1

0

unshuffle

7

6

5

4

3

2

1

0

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

10

Network Topology


16 port, 4 stage Omega network

00000001

00100011

01000101

01100111

10001001

10101011

11001101

11101111

00000001

00100011

01000101

01100111

10001001

10101011

11001101

11101111

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

11

Network Topology


16 port, 4 stage Baseline network

00000001

00100011

01000101

01100111

10001001

10101011

11001101

11101111

00000001

00100011

01000101

01100111

10001001

10101011

11001101

11101111

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

12

Network Topology


16 port, 4 stage Butterfly network

00000001

00100011

01000101

01100111

10001001

10101011

11001101

11101111

00000001

00100011

01000101

01100111

10001001

10101011

11001101

11101111

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

13

Network Topology-Correction to Butterfly


16 port, 4 stage Butterfly network

00000001

00100011

01000101

01100111

10001001

10101011

11001101

11101111

00000001

00100011

01000101

01100111

10001001

10101011

11001101

11101111

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

14

Network Topology


16 port, 4 stage Cube network

00000001

00100011

01000101

01100111

10001001

10101011

11001101

11101111

00000001

00100011

01000101

01100111

10001001

10101011

11001101

11101111

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

15

Network Topology

Centralized Switched (Indirect) Networks• Multistage interconnection networks (MINs)

– MINs interconnect N input/output ports using k x k switches

› logkN switch stages, each with N/k switches

› N/k(logkN) total number of switches

– Example: Compute the switch and link costs of interconnecting 4096 nodes using a crossbar relative to a MIN, assuming that switch cost grows quadratically with the number of input/output ports (k). Consider the following values of k:

› MIN with 2 x 2 switches› MIN with 4 x 4 switches› MIN with 16 x 16 switches

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

16

Network Topology

Centralized Switched (Indirect) Networks• Example: compute the relative switch and link costs, N = 4096

relative_cost(2 × 2)switches

= 40962 / (22 × 4096/2 × log2 4096) = 170


= 40962 / (42 × 4096/4 × log4 4096) = 170


= 40962 / (162 × 4096/16 × log16

4096) = 85

relative_cost(2 × 2)links

= 8192 / (4096 × (log2 4096 + 1)) = 2/13 = 0.1538


= 8192 / (4096 × (log4 4096 + 1)) = 2/7 = 0.2857


= 8192 / (4096 × (log16

4096 + 1)) = 2/4 = 0.5

cost(crossbar)switches

= 40962

cost(crossbar)links

= 8192

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

17

Network Topology

Relative link cost

0

0.5

1

1.5

2

2 16 128 1024 81922

32512

8192

k

N

0-0.5 0.5-1 1-1.5 1.5-2

050

100150200250300350

2 16 128 1024 81922

32512

8192

k

N

Relative switch cost

Centralized Switched (Indirect) Networks• Relative switch and link costs for various values of k and N

(crossbar relative to a MIN)

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

18

Network Topology

From blocking to non-blocking again

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

19

Network Topology

Centralized Switched (Indirect) Networks• Reduction in MIN switch cost comes at the price of performance

– Network has the property of being blocking

– Contention is more likely to occur on network links› Paths from different sources to different destinations share one or

more links

blocking topology

X

non-blocking topology

7

6

5

4

3

2

1

0

76543210

7

6

5

4

3

2

1

0

7

6

5

4

3

2

1

0

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

20

Network Topology

Centralized Switched (Indirect) Networks• How to reduce blocking in MINs? Provide alternative paths!

– Use larger switches (can equate to using more switches)› Clos network: minimally three stages (non-blocking)

» A larger switch in the middle of two other switch stages provides enough alternative paths to avoid all conflicts

– Use more switches

› Add logkN - 1 stages, mirroring the original topology

» Rearrangeably non-blocking» Allows for non-conflicting paths» Doubles network hop count (distance), d» Centralized control can rearrange established paths

› Benes topology: 2(log2N) - 1 stages (rearrangeably non-blocking)

» Recursively applies the three-stage Clos network concept to the middle-stage set of switches to reduce all switches to 2 x 2

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

21

Network Topology


16 port Crossbar network

7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

22

Network Topology


16 port, 3-stage Clos network

7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

23

Network Topology


16 port, 5-stage Clos network

7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

24

Network Topology


16 port, 7 stage Clos network = Benes topology

7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

25

Network Topology


Alternative paths from 0 to 1. 16 port, 7 stage Clos network = Benes topology

7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

26

Network Topology


7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

Alternative paths from 4 to 0. 16 port, 7 stage Clos network = Benes topology

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

27

Network Topology


7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

Contention free, paths 0 to 1 and 4 to 1. 16 port, 7 stage Clos network = Benes topology

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

28

Network Topology

• Bidirectional MINs• Increase modularity• Reduce hop count, d• Fat tree network

– Nodes at tree leaves– Switches at tree

vertices– Total link bandwidth

is constant across all tree levels, with full bisection bandwidth

– Equivalent to folded Benes topology

– Preferred topology in many SANs

Folded Clos = Folded Benes = Fat tree network


7

6

5

4

3

2

1

0

15

14

13

12

11

10

9

8

NetworkBisection

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

29

Network Topology

Myrinet-2000 Clos Network for 128 Hosts

• Backplane of the M3-E128 Switch• M3-SW16-8F fiber line card (8 ports)

http://myri.com

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

30

Network Topology

Myrinet-2000 Clos Network for 128 Hosts

• “Network in a Box” • 16 fiber line cards connected to the M3-E128 Switch backplane

http://myri.com

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

31

Network Topology

Myrinet-2000 Clos Network Extended to 512 Hosts

http://myri.com

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

12

Assignment 2-2

Chose one of the following exercises:

• Calculate how many permutations nxn Omega network could support.

• Prove that folded Clos network, folded Bens network, and Fat tree network are isomorphic to each other.

SJTU@Fall 2012Parallel Processing, Low-Diameter ArchitecturesSlide 32

Documents

Indirect Networks or Dynamic Networks