CSE 8383 - Advanced Computer Architecture

Week-12April 8, 2004

engr.smu.edu/~rewini/8383

Contents Dynamic Networks Message Passing Mechanisms Message Passing in PVM

Dynamic Network Analysis Parameters:

Cost: number of switches Delay: latency Blocking characteristics Fault tolerance

MIMD Distributed Memory Systems

Interconnection Networks

M M M M

P P P P

Interconnection Network Taxonomy

Interconnection Network

Static Dynamic

Bus-based Switch-based1-D 2-D HC

Single Multiple SS MS Crossbar

Dynamic Interconnection Networks Communication patterns are based

on program demands

Connections are established on the fly during program execution

Multistage Interconnection Network (MIN) and Crossbar

Switch Modules A x B switch module A inputs and B outputs In practice, A = B = power of 2 Each input is connected to one or

more outputs (conflicts must be avoided)

One-to-one (permutation) and one-to-many are allowed

Binary Switch

2x2Switch

Legitimate States = 4

Permutation Connections = 2

Legitimate Connections

Straight Exchange

Upper-broadcast

Lower-broadcastThe different setting of the 2X2 SE

Group WorkGeneral Case ??

Multistage Interconnection Networks

ISC1ISC1 ISC2ISC2 ISCnISCn

switches switches switches

ISC Inter-stage Connection Patterns

Perfect-Shuffle Routing Function Given x = {an, an-1, …, a2, a1} P(x) = {an-1, …, a2, a1 , an}

X = 110001P(x) = 100011

Perfect Shuffle Example000 000001 010010 100011 110100 001101 011110 101111 111

Perfect-Shuffle000001010011

100101110111

000001010011

100101110111

Exchange Routing Function Given x = {an, an-1, …, a2, a1}

Ei(x) = {an, an-1, …, ai, …, a2, a1}

X = 0000000E3(x) = 0000100

Exchange E1

000 001001 000010 011011 010100 101101 100110 111111 110

Exchange E1

000001010011

100101110111

000001010011

100101110111

Butterfly Routing Function Given x = {an, an-1, …, a2, a1} B(x) = {a1 , an-1, …, a2, an}

X = 010001P(x) = 110000

Butterfly Example000 000001 100010 010011 110100 001101 101110 011111 111

Butterfly000001010011

100101110111

000001010011

100101110111

Multi-stage network000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

MIN (cont.)1

2

3

4

5

6

7

8

9

10

11

12

001010011

100101

110111

000000001

010011

100101

110111

An 8X8 Banyan network

Min ImplementationControl (X)

Source (S) Destination (D)

X = f(S,D)

Example

X = 0 X = 1

(crossed) (straight)

A

B

C

D

A

B

C

D

Consider this MINS1S2

S3

S4

S5

S6

S7

S8

D1D2

D3D4

D5

D6

D7

D8

stage 1 stage 2 stage 3

Example (Cont.) Let control variable be X1, X2, X3 Find the values of X1, X2, X3 to connect:

S1 D6 S7 D5 S4 D1

The 3 connectionsS1S2

S3

S4

S5

S6

S7

S8

D1D2

D3D4

D5

D6

D7

D8

stage 1 stage 2 stage 3

Boolean Functions X = x1, x2, x3 S = s2, s2, s3 D = d1, d2, d3

Find X = f(S,D)

Crossbar Switch M1 M2 M3 M4 M5 M6 M7 M8

P1

P2

P3

P4

P5

P6

P7

P8

Analysis and performance metricsdynamic networks Performance comparison of dynamic networks

Networks Delay Cost Blocking

Degree of FT

Bus O(N) O(1) Yes 0Multiple-bus O(mN) O(m) Yes (m-1)

MIN O(logN) O(NlogN) Yes 0Crossbar O(1) O(N2) No 0

Message Passing Mechanisms Message Format

Message arbitrary number of fixed length packets

Packet basic unit containing destination address. Sequence number is needed

A packet can further be divided into flits (flow control digits)

Routing and sequence occupy header flit

Message, Packets, Flits

Message

Packet

Data flit

Destination

Sequence

Store and Forward Routing Packets are the basic units of

information flow Each node uses a packet buffer A packet is transferred from S to D

through a sequence of intermediate nodes

Channel and buffer must be available

Wormhole Routing Flits are the basic units of information

flow Each node uses a flit buffer Flits are transferred from S to D through a

sequence of intermediate routers in order (Pipeline)

Can be visualized as a railroad train Flits from different packets cannot be

mixed up

Latency Analysis L packet length (in bits) W Channel bandwidth (bits/sec) D Distance (number of hops) F flit length (in bits) TSF = D * L/W TWH = L/W + D* F/W L/W if L>>F

(independent of D)

Communication Patterns Point to Point 1 - 1 Multicast 1 - n Broadcast 1 - all Conference n - n

Routing Efficiency Two Parameters

Channel Traffic (number of channels used to deliver the message involved)

Communication Latency (distance)

Multicast on a mesh (5 unicasts)

Traffic ?

Latency ?

Multicast on a mesh (multicast pattern 1

Traffic ?

Latency ?

Multicast on a mesh (multicast pattern 2)

Traffic ?

Latency ?

Broadcast (tree structure)3 2 3 4

2 1 2 3

1 1 2

Message Passing in PVM

User application

Library

Daemon

1

2 3

4

User application

Library

Daemon

5

6 7

8

Sending Task Receiving Task

Standard PVM asynchronous communication A sending task issues a send command

(point 1) The message is transferred to the

daemon (point 2) Control is returned to the user

application (points 3 & 4) The daemon will transmit the message

on the physical wire sometime after returning control to the user application (point 3)

Standard PVM asynchronous communication (cont.) The receiving task issues a receive

command (point 5) at some other time In the case of a blocking receive, the

receiving task blocks on the daemon waiting for a message (point 6). After the message arrives, control is returned to the user application (points 7 & 8)

In the case of a non-blocking receive, control is returned to the user application immediately (points 7 & 8)

Send (3 steps)1. A send buffer must be initialized2. The message is packed into the

buffer3. The completed message is sent

to its destination(s)

Receive (2 steps)1. The message is received2. The received items are unpacked

Message Buffers Buffer Creation (before packing)

Bufid = pvm_initsend(encoding_option)

Bufid = pvm_mkbuf(encoding_option)

Encoding option Meaning0 XDR1 No encoding2 Leave data in place

Message Buffers (cont.) Data Packing

pvm_pk*() pvm_pkstr() – one argument

pvm_pkstr(“This is my data”); Others – three arguments

1. Pointer to the first item2. Number of items to be packed3. Stride

pvm_pkint(my_array, n, 1); Packing functions can be called multiple

times to pack data into a single message

Sending a message Point to point (one receiver)

info = pvm_send(tid, tag)

broadcast (multiple receivers)info = pvm_mcast(tids, n, tag)info = pvm_bcast(group_name, tag)

Pack and Send (one step)info = pvm_psend(tid, tag, my_array, length, data type)

Receiving a message Blocking

bufid = pvm_recv(tid, tag)-1 wild card in either tid or tag

Nonblocking bufid = pvm_nrecv(tid, tag) bufid = 0 (no message was received)

Timeout bufid = pvm_trecv(tid, tag, timeout)bufid = 0 (no message was received)

Different Receive in PVM

Pvm_recv()

wait

Time

Funcitonis called

Time is expired

Message arrival

Blocking

Pvm_nrecv()

Continue execution

Non-blocking

Pvm_trecv()

wait

Timeout

Resume execution

Resume execution

Data unpackingpvm_upk*() pvm_upkstr() – one argument

pvm_upkstr(string);

Others – three arguments1. Pointer to the first item2. Number of items to be unpacked3. Stride

pvm_upkint(my_array, n, 1);