Efficient broadcasting in wormhole-routed multicomputers: a network-partitioning approach

44 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 10, NO. 1, JANUARY 1999

Efficient Broadcasting inWormhole-Routed Multicomputers:A Network-Partitioning ApproachYu-Chee Tseng, Member, IEEE Computer Society, San-Yuan Wang,

and Chin-Wen Ho, Member, IEEE Computer Society

Abstract—In this paper, a network-partitioning approach for one-to-all broadcasting on wormhole-routed networks is proposed. Tobroadcast a message, the scheme works in three phases. First, a number of data-distributing networks (DDNs), which can workindependently, are constructed. Then the message is evenly divided into submessages, each being sent to a representative node inone DDN. Second, the submessages are broadcast on the DDNs concurrently. Finally, a number of data-collecting networks(DCNs), which can work independently too, are constructed. Then, concurrently on each DCN, the submessages are collected andcombined into the original message. Our approach, especially designed for wormhole-routed networks, is conceptually similar butfundamentally very different from the traditional approach (e.g., [4], [13], [18], [31]) of using multiple edge-disjoint spanning trees inparallel for broadcasting in store-and-forward networks. One interesting issue is on the definition of independent DDNs and DCNs,in the sense of wormhole routing. We show how to apply this approach to tori, meshes, and hypercubes. Thorough analyses andcomparisons based on different system parameters and configurations are conducted. The results do confirm the advantage of ourscheme, under various system parameters and conditions, over other existing broadcasting algorithms.

Index Terms—Collective communication, hypercube, interconnection network, mesh, one-to-all broadcast, parallel processing,torus, wormhole routing.

——————————��F��——————————

1 INTRODUCTION

N a multicomputer network, processors often need tocommunicate with each other for various reasons, such

as data exchange or event synchronization. Efficient com-munication has been recognized to be critical for high-performance computing. One essential communication op-erator is the one-to-all broadcast, where a source node needsto send a message to every other node in the network.Broadcast has many applications, such as linear algebraalgorithms [12], barrier synchronization [34], parallel graphalgorithms, parallel matrix algorithms, distributed table-lookup, fast Fourier transformation, and cache coherence.The one-to-all broadcast, together with other operators suchas all-to-all broadcast, personalized broadcast, and data reduc-tion, are termed as collective communication and have re-ceived intensive attention recently [1], [2], [15], [17], [28],[29], [30].

We consider the communication network using wormholerouting switching technology [7], [19], which is character-ized with low communication latency and is quite insensi-tive to routing distance in the absence of link contention.Such technology has been adopted by many new-generation parallel machines, such as the Intel TouchstoneDELTA [11], Intel Paragon, MIT J-machine [20], CaltechMOSAIC, nCUBE 3 [8], and Cray T3D and T3E [14], [6].

In this paper, a network-partitioning approach for one-to-all broadcasting on wormhole-routed networks is pro-posed. To broadcast a message, the scheme works in threephases, as follows: First, a number of data-distributing net-works (DDNs), which can work independently, are con-structed. Then, the message is evenly divided into submes-sages, each being sent to a representative node in one DDN.Second, the submessages are broadcast on the DDNs con-currently. Finally, a number of data-collecting networks(DCNs), which can work independently too, are con-structed. Then, concurrently on each DCN, the submes-sages are collected and combined into the original message.One interesting issue in this approach is on how to definetwo subnetworks to be independent in the sense of worm-hole routing—independent networks should be able towork independently without interferences. Formal defini-tions can be seen in Section 2.

One typical approach to the broadcast problem is toutilize multiple spanning trees in parallel for transmission.For instance, Johnson and Ho [13] show how to use n edge-disjoint spanning trees in an n-cube for various versions ofbroadcast problems. While Bermond et al. [4] show how toconstruct two edge-disjoint spanning trees in a 2D torus,Michallon and Trystram [18] further use four disjoint treesto facilitate broadcasting. On the side of star graphs, Tsengand Sheu [31] have used n - 1 congestion-2 edge-disjointspanning trees for broadcasting in an n-star. Note that in allthe above work [4], [13], [18], [31], the number of spanningtrees is a fixed number, given a fixed network of a fixed di-mension. Furthermore, the spanning trees are “real” graphs in

1045-9219/99/$10.00 © 1999 IEEE

²²²²²²²²²²²²²²²²

•� The authors are with the Department of Computer Science and InformationEngineering, National Central University, Chung-Li, 32054, Taiwan.�E-mail: {yctseng, sywang, hocw}@csie.ncu.edu.tw.

Manuscript received 21 Oct. 1996; revised 8 Jan. 1998.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log Number 100329.

I

TSENG ET AL.: EFFICIENT BROADCASTING IN WORMHOLE-ROUTED MULTICOMPUTERS: A NETWORK-PARTITIONING APPROACH 45

the sense that, for any edge of a tree, its two endpoints mustexist in the tree. Such spanning trees are suitable for store-and-forward types of networks, but may not be suitable forwormhole-routed ones.

Specially designed for wormhole-routed networks, ourapproach is conceptually similar but fundamentally verydifferent from the approach used in [4], [13], [18], [31]. First,the DDNs and DCNs used in our scheme may not be span-ning trees. Second, the numbers of DDNs and DCNs areadjustable parameters, which can be used to optimize theperformance. Third, a subnetwork may not be a “graph” instandard graph-theoretic terminology—an edge may existin a subnetwork without both of its endpoints existing. Thisin fact carries special meaning in a wormhole-routed net-work, in which a message may pass the router of a nodewithout interfering the computation in the node. Thus, inour work, a network may contain a path of edges with onlytwo nodes existing at the start and end of the path (we callsuch a network a dilated network). Due to the distance-insensitive characteristic of wormhole routing, the trans-mission along the path is expected to be pretty fast.

We show how to apply our scheme to 2D tori, 2Dmeshes, and hypercubes. Thorough analyses and compari-sons based on different system parameters and configura-tions are conducted. The results do confirm the advantageof our scheme, under various system parameters and con-ditions, over other existing broadcasting algorithms (e.g.,the U-torus scheme for one-port tori by Robinson et al. [22],the U-mesh scheme for one-port meshes by McKinley et al.[16], the Scatter-Collect and the Edge-Disjoint-Spanning-Fences schemes for one-port tori/meshes by Barnett et al.[3], the Postal model when applied to one-port tori/meshes[5], the dominating-set approaches for all-port meshes/toriby Tsai and McKinley [24], [25], [26], [27], and the schemesfor all-port hypercubes by Ho and Kao [10] and Wang andKu [32]). In particular, in our schemes, because mesh/torusare partitioned into DDNs that are dilated mesh/torus,broadcasting on these DDNs can be easily done by directlyapplying existing broadcasting algorithms [3], [16], [22],[26], [27] for ordinary mesh/torus. Following most stan-dard terminology in the literature [19], we divide the com-munication latency into two parts: startup cost (for initial-izing a communication on a communication link) and

transmission cost (for sending data on the links). The latteris further divided into the cost for the header flit (whichneeds the router to make a routing decision) and that forthe follow-up flits (which do not). For a quick overview andcomparison, based on these three factors, of existing andour algorithms, refer to Table 1, Table 2, Table 4, Table 5,Table 6, Table 7, Fig. 8, Fig. 14, Fig. 15, Fig. 16, and Fig. 18.As can be observed, under one-port model, the U-torus,U-mesh, and Postal schemes perform better with fairly smallmessages; the Scatter-Collect and Edge-Disjoint-Spanning-Fences schemes are better when broadcast message is im-practically large; our network-partitioning schemes are use-ful when the broadcast message is of a reasonable size. Un-der all-port model, the dominating-set approaches (formeshes/tori) and Ho-Kao scheme (for hypercubes) performbetter than ours at short messages, but are worse than oursat larger messages. In addition, we also use an nCUBE/2 toemulate a torus and conduct some experiments. More de-tails are in Section 3.1.5 and the obtained results do con-form to our analysis.

The rest of this paper is organized as follows. Prelimi-naries are given in Section 2. Sections 3 and 4 present ourrouting algorithms for 2D tori and meshes, respectively,under various system configurations. Our algorithms forhypercubes are presented in Section 5. Finally, conclusionsare drawn in Section 6.

2 PRELIMINARIES

2.1 System ModelIn a wormhole-routed network, each node contains a sepa-rate router to handle the communication tasks. The archi-tecture of a generic node is shown in Fig. 1 [19]. Each routersupports some numbers of internal channels (connecting tothe local processor/memory) and external channels (con-necting to other routers). Each of these channels consists ofa pair of input and output channels. From the connectivitybetween routers, we can define the topology of a worm-hole-routed network as G = (V, C), where V is the node setand C specifies the channel connectivity. Throughout thispaper, we assume that the channel connectivity is bi-directional, in the sense that if there is a connection fromrouter x to router y, then the reverse connection also exists.

Fig. 1. A 4 � 4 torus and the generic node architecture for wormhole routing. The gray nodes and dotted links constitute subnetwork G1, whileblack nodes and solid links constitute subnetwork G2.


Both the one-port and all-port models will be consideredin this paper, where under the former model a node canonly send and simultaneously receive one message at atime, while under the latter a node can simultaneously sendand receive messages along all channels. Reflecting on thearchitecture in Fig. 1, the router will have one (four) internalchannels under the one-port (all-port) model. We note thatthese models, however, do not limit the routing capabilityof routers—a router can still concurrently send and receiveflits along all external channels.

In a wormhole-routed network, a message is partitionedinto flits. The header flit governs the routing, and the re-maining flits follow the header in a pipelined fashion. Inthe contention-free case, the communication latency forsending a message of L bytes from a source node to a desti-nation node with distance d can be formulated as

T ds

L

BLB

f+ + , where Ts is the startup time, Lf is the length ofeach flit, and B is the bandwidth of the channel. In this pa-per, as a convention, the broadcast latency of an algorithm

will be written in three factors as aTs + bTf + g LTc, where

Tf

L

Bf= can be regraded as the cost to transmit a header flit,

and Tc = 1/B as that for a nonheader flit. We will comparethe a, b, and g values of our algorithm against others.

Throughout the paper, we will consider the network Gas mesh, torus, or hypercube. Routing on these networksare assumed be the commonly used dimensional-orderedrouting [7], [19]. For example, the unicast path shown inFig. 1 follows such routing.

2.2 Independent SubnetworksGiven a wormhole-routed network G = (V, C), a subnet-work G� = (V�, C�) of G is one such that V� µ V and C� µ C.However, a subnetwork is not necessary a “graph” in stan-dard graph-theoretic terms. Specifically, suppose channel(x, y) ¶ C�. Then the vertices x and y incident by c are notnecessarily in the vertex set V�. This carries special meanings tous: In subnetwork G�, the local processors of x and y are notallowed to send and receive messages in G�, but the routers inthem can help propagating flits in G�. That is, worms can passthrough x and y, but should not be initiated from or destinedto x and y. This leads to the following definitions:

DEFINITION 1. Two subnetworks G1 = (V1, C1) and G2 = (V2, C2)of G are said to be independent under the one-portmodel if V1 > V2 = ® and C1 > C2 = ®.

DEFINITION 2. Two subnetworks G1 = (V1, C1) and G2 = (V2, C2)of G are said to be independent under the all-portmodel if C1 > C2 = ®.

There is some subtlety in the above definitions. For in-stance, Fig. 1 contains two subnetworks G1 and G2, whichare independent under both one-port and all-port models.However, the two subnetworks G1 and G2 of network Gshown in Fig. 2 are only independent under the all-portmodel, but not one-port model, because they share com-mon nodes. Hence, one can simultaneously use G1 and G2to perform communication safely under the all-port model,but not so under the one-port model.

2.3 A General Broadcasting SchemeGiven any network G, here we propose a general broad-casting scheme. From G we first construct two kinds ofsubnetworks: data-distributing networks (DDNs) and data-collecting networks (DCNs). Suppose we have h DDNs,DDN0, DDN1, ¤ , DDNh-1, and k DCNs, DCN0, DCN1, ¤,DCNk-1. We require the following properties in our model:

P1. DDN0, DDN1, ¤, DDNh-1 are mutually independent(under the given port model).

P2. DCN0, DCN1, ¤, DCNk-1 are mutually independent(under the given port model) and they together con-tain all nodes of G.

P3. DDNi and DCNj intersect in at least one node, for all 0 �i < h and 0 � j < k.

With the above properties, our broadcast scheme worksin three phases as follows. Here, the source node is x andthe broadcast message is M.

PHASE 1. Node x evenly partitions M into h submessagesM0, M1, ¤, Mh-1 and distributes each Mi to one repre-sentative node ri of network DDNi, i = 0..h - 1.

PHASE 2. Concurrently in each DDNi, node ri broadcasts thesubmessage Mi to the rest of the nodes in DDNi.

PHASE 3. Concurrently in each network DCNi, i = 0..k - 1,each node collects submessages M0, M1, ¤, Mh-1 fromthose that have received submessages in Phase 2 andcombines them into the original M.

Clearly, P1 and P2 ensure the concurrent execution inPhases 2 and 3, respectively. The correctness of broadcast isguaranteed by P2 and P3. The following properties are nota necessity, but would offer regularity in designing Phases 2and 3.

(a) (b) (c)

Fig. 2. (a) 4 � 4 mesh network G, (b) a subnetwork G1 of G, and (c) a subnetwork G2 of G.


P4. DDN0, DDN1, ¤, DDNh-1 are isomorphic.

P5. DCN0, DCN1, ¤, DCNk-1 are isomorphic.

In the following three sections, we discuss how to definethe DDNs and DCNs in tori, meshes, and hypercubes.

2.4 Independent Subnetworks in Tori

A 2D torus Ts�t consists of s � t nodes each denoted as px,y,where 0 � x < s and 0 � y < t. A node px y0 0, is connected to

px y1 1, if and only if either x1 =(x0 ± 1) mod s and y1 = y0, or

y1 = (y0 ± 1) mod t and x1 = x0. (Hereafter, we will omitsaying “mod” whenever the context is clear.) The torus isarranged on a plane as shown in Fig. 3 and we will use“row” and “column” as in standard algebra to refer to a setof components in the network.

DEFINITION 3. Given a torus Ts�t and any integer h that divides boths and t, the data distribution network DDNk = (Vk, Ck),where 0 � k < h, is defined as follows:

Vk = {px,y|x = ah + k, y = bh + k, for all a = 0..(s/h) - 1and b = 0..(t/h) - 1}

Ck = {all channels at rows ah + k and at columns bh + k}.

Intuitively, each DDN is a “dilated-h” torus of size (s/h) �(t/h), in the sense that each edge is dilated by a path of hedges. An example is shown in Fig. 3a with four dilated-44 � 4 tori embedded in a 16 � 16 torus. One can easily verify

that the above defined h DDNs are mutual independentunder both one-port and all-port models.

DEFINITION 4. Given a torus Ts�t and an integer h which dividesboth s and t, the data collecting network DCNa,b = (Va,b,Ca,b), 0 � a < s/h and 0 � b < t/h, consists of vertex set Va,b= {px,y |x = a � h + i, y = b � h + j for all i, j = 0..h - 1 andchannel set Ca,b which is defined to be the set of edges in-duced by Va,b in Ts�t.

Intuitively, these DDNs are obtained by evenly slicingthe torus into st/h2 blocks, each being a square h � h mesh(see the example in Fig. 3a). We can also show that eachDDNk must intersect with each DCNa,b by exactly one node.So these DDNs and DCNs satisfy all properties P1–P5.

With brief observation on Fig. 3a, one can easily deviseeight new (directed) DDNs by partitioning each (undi-rected) DDNs further into two directed subnetworks, oneusing only positive channels (i.e., in positive-x and positive-y directions), and the other negative channels only (i.e., innegative-x and negative-y directions). Now, each directedDDN is a directed torus. However, these eight DDNs areonly mutually independent under the all-port model sincesome DDNs share common nodes. The following definitionshows how to conquer this problem:

DEFINITION 5. Given a torus Ts�t and any integer h that dividesboth s and t, the positive data distribution network

DDN V Ck k k+ + += ,4 9, where 0 � k < h, is defined as follows:

(a) (b)

Fig. 3. Independent subnetworks in a 16 � 16 torus (under both one- and all-port models): (a) four dilated-4 undirected 4 � 4 tori, and (b) eightdilated-4 directed 4 � 4 tori.


V p x ah k y bh k a s h

b t h

k x y+ = = + = + = −

= −

, , , . .

. .

for all

and

0 1

0 1

2 7J2 7 B

C ag k bh kk+ = + +all positive channels at row and columns< A,

and the negative data distribution network

DDN V Ck k k− − −= ,4 9

is defined as follows:

V p x ah k y bh k a s h

b t h

k x y− = = + + = + = −

= −

, , , . .

. .

1 0 1

0 1

for all

and

2 7J2 7 B

C ag k

bh kk− = +

+ +

all negative channels at row

and columns

<A1 .

Intuitively, DDNk+ is the same as DDNk defined earlier

except that it only uses the positive channels, and DDNk− is

obtained by cyclically shifting DDNk to the right by oneposition and it only uses negative channels. Each DDN isnow a dilated-h directed (s/h) � (t/h) torus. An example isshown in Fig. 3b with eight dilated-4 directed 4 � 4 tori em-bedded in a 16 � 16 torus (for visual clarity, only typicalchannels are shown). Clearly, these positive and negativeDDNs are isomorphic and mutually independent underboth one-port and all-port models.

2.5 Independent Subnetworks in MeshesA 2D s � t mesh, denoted as Ms�t, is defined similar to Ts�texcept that there are no wrapped-around links. The samenotations used for tori will be used here, too.

One fundamental difference between meshes and tori isthat meshes are not node-symmetric. Thus, it is easy to“shift” a DDN in a torus to any desired location. However,this is less obvious for the case of meshes. The followingdefinition provides a way to find a set of DDNs with respectto a given mesh node.

DEFINITION 6. Given a mesh Ms�t, a node pi,j in the mesh, andany integer h that divides both s and t, the data-distribution network DDNk = (Vk, Ck) with respect topi,j, 0 � k < h, is defined as follows:

V p x ah k i s y bh k j t

a s h b t h

k x y= = + + = + +

= − = −

, mod , mod ,

. . , . .

1 6 1 6J2 7 2 7 Bfor all 0 1 0 1

C ah k j s

bh k i t

k = + +

+ +

all channels at rows and at columns1 6=1 6 A

mod

mod .

Intuitively, these DDNs are obtained from the DDNs of atorus (in Definition 3) by shifting them horizontally andvertically by i and j positions, respectively. Each DDN is adilated-h mesh of size (s/h) � (t/h). For instance, the DDNsin Fig. 4 are defined with respect to node p3,1. These DDNsare isomorphic and mutually independent. It is alsostraightforward to extend Definition 5 to obtain 2h inde-pendent directed DDNs; we leave this part to the reader.

The DCNs are defined exactly the same as in tori (Defini-tion 4). Each DDN still intersects with any DCN at one node.So properties P1–P5 still hold true under our definitions.

2.6 Independent Subnetworks in Hypercubes

A binary n-cube is an undirected graph with 2n nodes each

labeled with a distinct binary string b1 b2 ¤ bn. Node b1 ¤ bi

¤ bn and node b b bi n1K K are joined by an edge along di-

mension i, where bi is the one’s complement of bi. It is well-known that an n-cube has a recursive structure that can bepartitioned into many subcubes. A subcube of dimension d � n

can be denoted by a ternary string x1 x2 ¤ xn, where xi ¶{0, 1, *}, with exactly d *s, where * means a “don’t-care.”

This subcube consists of all nodes obtained from x1 x2 ¤ xn

by arbitrarily replacing each *-symbol with 0 or 1. For in-stance, *01*0 is a 2-cube in a 5-cube consisting of nodes{00100, 00110, 10100, 10110}.

DEFINITION 7. Given an n-cube and an integer d, 1 � d � n, wedefine 2d DDNs each being an (n - d)-cube of the form b1b2L bd*

n-d (*n-d denotes a sequence of n - d *s). Also,we define 2n-d DCNs, each being a d-cube of the for-mat *dad+1ad+2 L an.

LEMMA 1. For any pair of DDN and DCN defined in Definition 7,they intersect in exactly one node.

So properties P1–P5 all hold true.

3 ONE-TO-ALL BROADCAST IN A TORUS

In this section, we apply the DDNs and DCNs of a torusdefined in Section 2.4 to our general broadcast scheme. We

Fig. 4. Four dilated-4 independent 4 � 4 mesh in a 16 � 16 mesh de-fined with respect to node p3,1.


consider both one-port and all-port communication models.Although our scheme can be applied to nonsquare tori, be-low we develop the scheme based on a torus T n n2 2×

. This is

solely for the purpose of simplifying the analysis and com-parison part. So the possible value of h (which will define

the numbers of DDNs and DCNs) is h = 2d, d = 1..n. Without

loss of generality, we let the source node be p0,0. The mes-sage to be broadcast is M of length L bytes.

3.1 One-Port ModelBelow, we first develop our schemes based on the (undi-rected) DDNs defined in Definition 3. Then we will discussthe alternative if we use the (directed) DDNS in Definition 5.

3.1.1 Phase 1: Scattering to Representative Nodes ofUndirected DDNs

There are h undirected DDNs: DDN0, DDN1, ¤, DDNh-1.We can let ri = pi,i be the representative node of DDNi, i =1..h - 1. So the source node p0,0 should evenly partition Minto submessages M0, M2, ¤, Mh-1 and then distribute eachMi to ri. This can be done efficiently by a simple recursivedoubling as follows. Let r0 = p0,0. Node r0 first sends sub-messages Mh/2, Mh/2+1, ¤, Mh-1 to rh/2 using any shortestpath. Then, r0 and rh/2 each plays as the source of M0, M1,¤, Mh/2-1 and Mh/2, Mh/2+1, ¤, Mh-1, respectively, and re-cursively distributes these submessages to nodes r0, r1, ¤,rh/2-1 and nodes rh/2, rh/2+1, ¤, rh-1. Fig. 5a illustrates thisprocess in the upper-left 4 � 4 submesh of the torus whenh = 4. The execution time of this phase is

T T TL

T

dT T LT

sd i

f i ci

d

sd

f d c

11

2 22

2 2 1 11

21

= + +��

��

= + − + −��

��

−

=∑ 4 9

4 9 . ( )

3.1.2 Phase 2: Broadcasting in Undirected DDNsIn Phase 2, every representative node ri, i = 0..h - 1, shouldbroadcast submessage Mi on DDNi in parallel. Recall thateach of our DDNs is in fact a dilated-h tori. So, any existingbroadcasting algorithm for tori should be able to apply toour DDNs. In the literature, there are three such schemes:recursive doubling (RD) [22], scatter-collect (SC) [3], andedge-disjoint spanning fences (EDSF) [3]. Below, we will

introduce these alternatives and analyze the costs whenapplying them to our dilated DDNs. Our analyses do takethe dilation of our DDNs into consideration, though it onlycauses a little penalty due to the distance-insensitive char-acteristic of wormhole routing.

Our first alternative, the RD scheme [22], was originallydesigned for multicasting on a torus, but can be used forbroadcasting here. Recursive doubling is in fact a com-monly used technique in parallel processing and, thus, weomit the details. In an ordinary (undilated) 2n � 2n torus, theRD scheme takes time

T T T LT nT T nLTRDt

sn i

f c sn

f ci

n

= + + = + − +− +

=∑2 2 2 2 2 21

14 9 4 9 .

Applied to our dilated-h DDNs (of size 2n-d � 2n-d), itshould be rewritten as

T n d T Tn d

d LTRDs

n df c2

1 12 2 22

2= − + − +−+ +1 6 4 9 1 6

. (2)

Our second alternative is the SC scheme [3]. In a 2n � 2n

torus, the scheme works in four stages as follows:

1)�Column Scattering: The source node slices the broadcastmessage evenly into 2n submessages and, then, scattersthem across the column where the source resides;

2)�Row Scattering: Each node receiving a submessage instep 1 further slices the submessage into 2n smallersubmessages and scatters them across the row whereit resides;

3)�Row Collecting: Each row independently forms a logi-cal ring and every node circulates the submessage itreceived in step 2 around the ring; and

4)�Column Collecting: Each column independently formsa logical ring and every node circulates the submes-sages it received in step 3 around the ring.

In an ordinary 2n � 2n torus, the SC algorithm takes time [3]:

T T T LT T T LT

T T LT T T LT

n T T LT

SCt

sn i

f i c sn i

f i n ci

n

i

n

s f n c s f n cii

ns

nf n c

nn

= + +��

�� + + +

��

��

+ + +��

�� + + +

��

��

= + − + − + −��

��

− −+

==

=

−

=

−

+ +

∑∑

∑∑

21

22

1

2

1

2

1

2

2 2 2 2 4 2 11

2

11

21

2 1

1

2 1

1 224 9 4 9 .

(a) (b) (c)

Fig. 5. Broadcasting in a one-port torus: (a) Phase 1 happening in the upper-left 4 � 4 submesh of the torus, (b) row-broadcasting of Phase 3, and(c) column-collecting of Phase 3.


In our dilated-h DDNs (of size 2n-d � 2n-d), the cost is

T n d T T

LT

SC n ds

n df

d n d c

22 2

2

2 2 1 2 2

21

2

1

2

= + − − + −

+ −��

��

− + +

−

4 9 4 9

.

Our third alternative is the EDSF scheme [3]. First, twoedge-disjoint (but not node-disjoint) spanning fences areconstructed from the torus.1 The source node then parti-tions the broadcast message into k1 submessages and alter-nately injects them into the two fences. The way submes-sages are injected will ensure that no node needs to propa-gate submessages for both fences at the same step (and,thus, the one-port assumption is not violated).

The execution time of the EDSF scheme on an ordinaryT n n2 2×

is

T k T TLk TEDSF

t n ns f c= + + + +��

��1

12 24 9 ,

where k1 should be determined for best performance asfollows:

k

LT T T

L i LT T T L

LT T T

nc s f

nc s f

nc s f

1

1

1

1

1 2 1

2

2

=

+ <

+ >

+

%

&KK

'KK

+

+

+

if

f

otherwise

4 94 9

4 9. (3)

In our dilated DDNs, the TEDSFt should be translated into

T k T TL

kTEDSF n d n d

sd

f d c2 22

2 2 22

= + + + +��

��

− −4 9 ,

where k2, the best number of sub-messages, isk

LT T T

L LT T T L

LT T T

n dc s

df

d n dc s

df

d

n dc s

df

2

2 1

2 1

2 1

1 2 2 1

2 2 2 2

2 2

4

=

+ <

+ <

+

%

&KK

'KK

− +

− +

− +

if

if

otherwise

4 94 9

4 9. ( )

3.1.3 Phase 3: Data Collecting in DCNsAfter Phase 2, in each DCN (which is an h � h mesh), thediagonal nodes have each received one of M0, M1, ¤, Mh-1.These submessages should be distributed to every node ofthe DCN. This is implemented in two stages: row broadcast-ing followed by column collecting.

In the row broadcasting stage, each node holding a sub-message broadcasts along its own row in a recursive dou-bling manner. An example is shown in Fig. 5b in a 4 � 4DCN. This takes d communication phases and incurs cost

T T T LTts

d if

dc

i

d

3 11

1

2 2_ .− + +− −

=∑ 4 9 (5)

In the column collecting stage, each node collects thesubmessages from the other nodes located on the samecolumn. We first embed a logical (directed) ring on each

1. Thus, in our terminology, these spanning fences are independent un-der the all-port model, but not so under the one-port model.

column of the DCN. This is done by first visiting evennodes downward the column and then odd nodes up-ward the column. An example of such embedding isshown in Fig. 6. Clearly, this gives a dilation-2 embed-ding. With this embedding, every node then pipelines itssubmessage following the direction of the ring h - 1 steps,after which the broadcasting is done. An example isshown in Fig. 5c in a 4 � 4 DCN. The column collectingstage runs in time

T h T T LTts f

dc3 2

1 1 2 2_ ,= − + + −1 64 9 (6)

where the constant 2 comes from the embedding dilation.Summing the above, the total cost of this phase is

T T T

d T Td

LT

t t

ds

df d c

3 3 11

3 21

2 1 3 2 1 11

27

= +

= + − + − + +−�

��

_ _

. ( )4 9 4 9

3.1.4 Performance Analysis and ComparisonThe total costs of our algorithm, based on the SC, EDSF, andRD schemes in Phase 2 are, respectively as follows (“NP”stands for network partitioning):

T T T T

n T T

n dLT

NP RDt RD

ds

n d df

d c

−+

= + +

= + − + + × + −

+ +− −�

��

1 2 3

1 22 2 1 2 3 2 2 5

22 2

2

4 9 4 9

T T T T

n T T

dLT

NP SCt SC

n d ds

n df

d n d c

−− + +

−

= + +

= + + − + × − −

+ + −��

��

1 2 3

1 1

2

2 2 2 3 3 2 2 5

22

2

2

4 9 4 9

T T T T

k d T

k T

dk LT

NP EDSFt EDSF

n d ds

d nf

d

n d

c

−− +

+

− +

= + +

= + + + −

+ + + −

+ +−

+��

��

1 2 3

21

21

2 1

2

2 2 2 1

5 2 2 5

21

2

2

4 92 74 9

.

For ease of comparison, we summarize the costs of RD,SC, and EDSF schemes (for a same-size torus) and ours inTable 1. Note that the cost of each algorithm should be aTs+ bTf + g LTc. There are several parameters interacting inTable 1. To understand how much performance gain can beobtained by using our schemes (by applying SC, EDSF, orRD in Phase 2) against the original SC, EDSF, and RD al-gorithms, we draw Fig. 7, which shows the broadcast la-tency versus various message lengths in a 25 � 25 torus withcommunication parameters Ts = 150 msec, Tf = 2 msec, and

Fig. 6. A dilation-2 embedding of a logical ring into a linear path.


Tc = 0.5 msec (note that the relative values of these parame-ters, instead of their absolute values, are what really matter;this should hold true throughout the analysis of this paper).From Fig. 7a, we see that NP-RD is much better than RD inmost range of message sizes. Larger messages usually re-sult in more performance gain. Also, the larger the value ofd is, the more the performance gains. This is due to thelarger number of DDNs performing broadcast at the sametime. This can be verified from Table 1, where RD hassmaller a value but much larger g value than NP-RD.

From Fig. 7b, we observe that SC outperforms NP-SConly when d � 3 and L is fairly large (� 20K bytes). Whend = 4 or 5, NP-SC outperforms SC in all ranges of L. Thiscan be verified from Table 1, which indicates that SC haslarger a but slightly smaller g value than NP-SC. A similarphenomenon can be observed in Fig. 7c, and it does showin Table 1 that EDSF has larger a but slightly smaller g thanNP-EDSF.

To give an overall comparison, Fig. 8 shows the per-formance of RD, SC, EDSF, NP-RD, NP-SC, and NP-EDSF

TABLE 1COMPARISON OF BROADCAST COSTS IN A 2

n � 2

n TORUS UNDER THE ONE-PORT MODEL

(a) (b)

(c)

Fig. 7. Broadcast latency vs. message size in a 25 � 2

5 torus with Ts = 150 msec, Tf = 2 msec, and Tc = 0.5 msec. (a) RD vs. NP-RD, (b) SC vs.

NP-SC, (c) EDSF vs. NP-EDSF.


in a 25 � 25 torus with Ts/Tc set to 20 and 300. We used d = 3in our NP-based schemes. One referee also suggested us toinclude the Postal model2 into our comparison. As can be seen,RD performs the best with fairly small messages (L � 32bytes when Ts/Tc = 20, and L � 256 bytes when Ts/Tc = 300).Under Ts/Tc = 20, SC is the best when L is 3K , 5K, andunder Ts/Tc = 300, SC is the best when L � 40K. EDSF isonly useful when Ts/Tc = 20 and L > 5K (from the figure weconjecture that when Ts/Tc = 300, EDSF may become thebest when L is impractically large). In all other cases, it isbeneficial to use our NP-based schemes. In other words,our NP-based schemes are useful when the broadcast mes-sage is of a reasonable size, and this justifies the practicalvalue of our result.

It is also worth comparing Figs. 8a and 8b to observe theeffect of Ts/Tc on latency (in current parallel machines, thisratio ranges largely between 10 to 1000). We observe that alower ratio Ts/Tc tends to give our algorithms more ad-vantage over RD, while a higher ratio Ts/Tc tends to giveour algorithms more advantage over SC and EDSF. Thetrend should remain the same as the ratio Ts/Tc changes.

3.1.5 An Emulation Experiment on HypercubeThe above comparisons are based on mathematical analy-ses. In order to justify that our analyses are correct, we haveconducted an emulation experiment on a four-dimensionalnCUBE/2 with 16 nodes. (The reason is that we do not haveaccess to mesh/torus machines at the time of writing.) It iswell-known that an n-cube has a subgraph which is a 2i � 2j

torus such that i + j � n. So we can use the 4-cube to emulatea 4 � 4 torus.

2. The Postal model [5] is designed for one-to-all broadcast in a completegraph. Based on factors such as network size, message size, and the ma-chine’s communication parameters, the model will determine an appropri-ate spanning tree to propagate the broadcast message. Although there aremany factors to determine a good tree, as proposed in [5], it is possible tosimplify the construction of the tree by using a divide-and-conquer tech-nique. In case of tori/meshes (which are not complete graphs), it is alsopossible to schedule congestion-free communication. The performanceresults presented in Fig. 8 are obtained by varying the dividing factor (re-ferred to as a in [5]) and choosing the one giving the least latency.

Fig. 9a shows the scenario of our emulation of a 4 � 4 torususing a 4-cube to perform the RD scheme. Fig. 9b shows thescenario to perform NP-RD when the emulated 4 � 4 torus ispartitioned into two DDNs which are dilation-2 2 � 2 torus(i.e., d = 1). Because nCUBE/2 also uses wormhole and di-mensional-ordered routing, the cube can precisely emulatethe message-passing paths that we expect in an ordinary4 � 4 torus. So the emulation experiment should be quitereliable. In the same manner, we also ran SC and NP-SCschemes on this emulated torus.

The obtained emulation results are shown in Fig. 10a. Asreported in [23], the communication parameters of nCUBE/2are: Ts = 150 msec, Tf = 2 msec, and Tc = 0.46 msec. We applythese parameters to the predicted communication latencyformulae developed in Table 1 and draw Fig. 10b. Com-paring these two figures, we see that the predicted per-formance for RD, NP-RD, and NP-SC does conform closelyto our emulation results. However, the emulated SC schemeis worse than our prediction, probably because the broad-cast message is partitioned into too many pieces. Due to thesize limitation of communication buffers in nCUBE/2, wecould only simulate messages as large as 32K bytes. RDperforms the best with fairly small messages (L � 256bytes). When 256 � L � 4K, NP-RD outperforms others.With L > 4K, NP-SC is the best. SC is the worst in oursimulation, but we expect that it becomes the best when L ismuch larger.

3.1.6 NP-RD2h: Applying the NP Approach Based onDirected DDNs and the RD Scheme

Below, we briefly discuss how to apply our network-partitioning approach based on the directed DDNs in Defi-nition 5. We use the RD scheme in Phase 2. As before, we

set h = 2d, where 1 � d � n and, thus, there are 2h DDNs:

DDNk+ and DDNk

− , k = 0..h - 1.

In Phase 1, we let r pk k k+ = , (resp., r pk k k

−+= 1, ) be the rep-

resentative node of DDNk+ (resp., DDNk

− ), k = 0..h - 1. Wefirst perform the same Phase 1 for undirected DDNs (refer

(a) (b)

Fig. 8. The effect of Ts /Tc ratioin a 25 � 2

5 torus with h = 2

3, Tf = 2 msec and Tc = 0.5 msec. (a) Ts = 10 msec and Ts /Tc = 20, (b) Ts = 150 msec

and Ts /Tc = 300.


to Section 3.1.1). At the end, each rk+ sends half of its sub-

message to rk− (which is on rk

+ ’s right-hand side).In Phase 2, we apply the RD scheme [22] on each di-

rected DDN. This causes no problem because the RDscheme only uses a directed torus. However, only a sub-message of half length is broadcast, as compared to the NP-RD scheme.

After Phase 2, the diagonal nodes on each DCN haveeach received a submessage. Further, by Defintion 5, nodesof the right-hand side of these diagonal nodes have eachalso received a submessage. We first let the former nodessend their submessages to the latter on their left-hand sideand, then, execute the same Phase 3 as in Section 3.1.3.

Let’s call this scheme NP-RD2h. NP-RD2h only incurstwo extra submessage transmissions (of size L

h2 , one-hopaway) in Phases 1 and 3 as opposed to NP-RD, but offers thebenefit of broadcasting a smaller submessage (of L

h2 bytes) in

Phase 2. It is not hard to derive the exact cost of NP-RD2h.See Fig. 8, which also contains a draw for NP-RD2h (withd = 3). Generally speaking, NP-RD2h performs similarly toNP-RD. In some range of L, it is indeed worthwhile to usethis scheme.

As a final note, the SC and EDSF schemes also use onlyone direction of the torus. So we can similarly develop aNP-SC2h and a NP-EDSF2h based on the 2h directedDDNs. However, our simulation tests showed no benefit inusing them in most ranges of L. So the details, which areeasy to develop, are omitted here.

3.2 All-Port ModelNow we consider broadcasting under the all-port model. Forease of making comparison, we use a square torus T n n2 2×

. We

will only use the undirected DDNs in Definition 3, but we donot use the directed DDNs in Definition 5, as we know ofno scheme for broadcasting on a directed all-port torus (to

(a) (b)

Fig. 9. Emulation of a 4 � 4 torus using a 4-cube: (a) the scenario of running the RD scheme and (b) the scenario of running the NP-RD scheme.

(a) (b)

Fig. 10. Comparison of RD, NP-RD, SC, and NP-SC in a 4 � 4 torus: (a) emulation results using a four-dimensional nCUBE/2 and (b) predictedlatency from analysis using Ts = 150 msec, Tf = 2 msec and Tc = 0.46 msec.


be used in Phase 2). The possible number of DDNs will be

h = 2d, d = 1..n.

3.2.1 Phase 1: Scattering to Representative Nodes

Without loss of generality, let pm,m, m h= 2 be the source node.

Also, let ri = pi,i be the representative node of DDNi, 0 � i � h.

Observe that these pi,is form a diagonal in DCN0,0. The prob-

lem now becomes distributing submessages M0, M1, ¤, Mh-1

from the center of an h � h mesh to the diagonal nodes.The problem can be solved efficiently in two stages. Let

D(px,y, k) denote the sequence of k nodes [px,y, px+1,y+1, px+2,y+2,¤, px+k-1,y+k-1]. (For instance, D(p0,0, h) is the main diagonalof DCN0,0.) As shown below, in the first stage, each Mi willbe sent to one node in the ith row of the h � h mesh, i =0..h - 1. In the second stage, Mi will be sent to pi,i.

STAGE 1. We divide h evenly into five integers ti, i = 0..4, such

that h tii=

=∑ 0

4. This can be done by letting ti

h i= −5

(see [9, chapter 3]). Also, let s ti kk

i=

=∑ 0. We define

five node sequences Q0, Q1, Q2, Q3, Q4 as follows:

Q D p ts0 0 01= , ,4 9 , Q D p ts s1 10 0

= , ,4 9 , Q D p ts s2 21 1= , ,4 9 ,

Q D p ts s3 32 2= , ,4 9 , and Q D p ts s4 41 3

= , ,4 9. (See Fig. 11a

for an illustration.) Let the center nodes of Q0, Q1, Q2,

Q3, Q4 be: C0 = intersection node of column m and Q0,

C1 = center node of Q1, C2 = pm,m, C3 = center node of

Q3, and C4 = intersection node of column m and Q4.

First, node pm,m concurrently sends ti submessages to

nodes Ci, i = 0, 1, 3, 4, following the dimension-

ordered routing. Then, Ci acts as the source node to

scatter the ti submessages it received to rows covered

by Qi. In Figs. 11a and 11b, we illustrate the routing inthe first and second steps, respectively. The about re-cursion is repeated until the length of each of se-quences Q0, Q1, Q2, Q3, and Q4 reduces to one. Afterthe recursion completes, one node in each row ofDCN0,0, will have received a submessage.

STAGE 2. Every node that received a submessage in stage 1concurrently sends the submessage to ri = pi,i, i = 0..h - 1.

With an arbitrary value of h, the execution time will be

T Th

TLT

ThT L

h T

h T hT h LT

As i f

ci

i

h

sf

c

s f c

11

5

25 5 2

1 1 4 1

5

= +�

"## +

�

"##

��

�� + + +

�

"##

��

��

≤ + + + +=∑

log

log .3 8 2 7Plugging in h = 2d, the cost is upper-bounded by

T d T T LTAts

df

dc1 5 2 1 2 1 4 1 2≤ + + + +log .3 8 4 9

3.2.2 Phase 2: Broadcasting in DDNsNext, we need to perform broadcasting in each DDN. Wewill use the extended dominating node (EDN) scheme proposedby [26], [27] designed for broadcasting in an all-port torus.The idea is illustrated in Fig. 12. In one step, the source nodeS sends the broadcast message to three nodes A, B, C. In an-other step with a T-pattern, 16 nodes will have received thebroadcast message and these 16 nodes already form a regularpattern. Clearly, we can now partition the network into 16submeshes and recursively perform the above steps. This

works for tori of size 4k � 4k. It is also possible to extend the

scheme for tori of size (2 � 4k) � (2 � 4k) [26], [27].On an ordinary T n n2 2×

torus, the EDN scheme incurs a

cost of

T nTn

T nLTEDN s

n

f c= +− +

++2 2 2 2

3

1 mod ,.

2 74 9

(a) (b)

Fig. 11. Communication patterns in stage 1 of Phase 1 in an all-port torus: (a) the first step and (b) the second step.


Translated into our dilated DDNs, the cost is

T n d Tn d

T

n dLT

EDNs

d n d

f

d c

2

1 12 2 2 2

2

= − +− + −

+−

+ − +

1 6 2 74 9mod ,

.

3

3.2.3 Phase 3: Data Collecting in DCNsSimilar to the one-port case (Section 3.1.3), the collecting isimplemented in two stages: row broadcasting and columncollecting.

The row broadcasting stage can be done by applying arecursive tripling technique as follows. We consider a list of

consecutive h nodes xi, i h h h= + +− −2 2 21 2, , ,K , located in a

same row of the torus such that x0 is the node which re-ceived a submessage in Phase 2. Note that the h consecutivenodes may be located in two consecutive DCNs. Then wepartition the list evenly into three sublists, the first of length

h3 , the second h−1

3 , and the third h−23 . Also, let x� and

x�� be the middles of the first and third sublists, respectively.In one phase, x0 concurrently sends its submessage to x�and x��. Then x�, x0, x�� recursively act as the sources of thefirst, second, and third sublists, respectively. The idea isillustrated in Fig. 13. So, the number of nodes receiving thesubmessages will be tripled after each step. This gives acost of

T Th

TLh T

h Th

h T hLh T

As i f c

i

h

s f c

3 11

3 3 3

3

2

3

_

log

log log log .

= +�

"## +

�

"##

��

��

≤ +�

"## +

��

�� +

�

"##

=∑

The column collecting stage can be done by invoking thesame steps as in Section 3.1.3. The cost is the same Tt

3 21_ in

(6). So the cost of this phase is upper-bounded by

T T T

h h T h hh

T

h hLh T

A At t

s f

c

3 3 1 3 21

3 3

3

1 2 2 2

1 8

= +

= + − + + − +�

"##

��

��

+ + −�

"##

_ _

log log

log . ( )

3 8

3 8Letting h = 2d, the cost of this phase is bounded by

T d T d T

dL

T

At ds

df

dd c

3 3 3

3

2 2 1 252 2 2

2 2 12

= + − + + −��

��

+ + −�

"##

log log

log .

4 9

4 9

3.2.4 Performance Analysis and ComparisonSumming all three costs, the broadcast latency of our algo-rithm (termed as NP-EDN) in a T n n2 2×

is

T T T T

n d d d T

n dT

n d dLT

NP EDNAt EDN At

ds

nd

d

f

d c

−

+−

+

= + +

≤ + − +

+ + +−�

��

+ +− +�

��

1 2 3

3 5

21

1

3

2 2 2 2

23

133 2

2 23

54

2

2

log log

,

log.

4 92 7mod

We summarize the broadcast costs of the EDN and ourNP-EDN in Table 2. Our a and b factors are larger thanthose of the EDN scheme; the amount of difference dependson the parameter d. However, our g factor is close to a con-stant, while that of EDN is linear to n. This is because nomessage partitioning is used in the EDN scheme. So weshould expect much performance improvement over EDNwhen the message is large.

To understand the interaction among the a, b, g factors, inFig. 14 we depict the costs of EDN and NP-EDN in a T

2 25 5×

with Ts = (10 msec or 150 msec), Tf = 2 msec, and Tc = 0.5 msecfor various message sizes. Only with fairly small messages (L �64 bytes when Ts/Tc = 20 and L � 512 bytes when Ts/Tc = 300),will EDN perform better. As the message sizes increase, sig-nificant gain can be obtained by using NP-EDN. Although a

larger ratio Ts/Tc gives EDN more advantage, by comparingFigs. 14a and 14b, the effect seems to be very limited. Also,to get the best performance of NP-EDN, a median value ofd (= 2 or 3) would be appropriate.

4 ONE-TO-ALL BROADCAST IN A MESH

Next, we apply our network-partitioning approach to a 2n � 2n

mesh. Again, the possible numbers of DDNs are still h = 2d

for d = 1..n. Let the source node be px,y. We use the DDNsdefined in Definition 6 with respect to the source px,y. Also,let px,y reside in DCNa,b for some a and b. We choose the in-tersection of DDNi and DCNa,b as the representative node riof DDNi (there is only one such node).

4.1 Phases 1, 2, and 3The schemes for meshes are similar to those for tori. However,there do exist some differences, mainly due to nonsymmetricstructure of meshes. We summarize our schemes, as well as

Fig. 12. Two broadcasting steps of the EDN scheme in a 16 � 16 torus.

Fig. 13. Row broadcasting of Phase 3 in an all-port torus.


the associated costs, for meshes in Table 3, under both one-portand all-port models. As before, in phase 2, we have choices ofusing the RD, SC, and EDSF schemes under the one-port case,and using the dominating (termed as D-node) scheme underthe all-port case. So our NP-based schemes are named ac-cordingly as NP-RDM, NP-SCM, NP-EDSFM, and NP-D. Notethat when applying the SC scheme [3], we have modified itsscatter phases as follows. We embed on each column and rowof our DDNs (which are meshes) a dilated-2 directed ring, asillustrated in Fig. 6. On these columns/rows the column/row-scattering is then performed in a recursive-doubling manner.On the contrary, [3] suggests to traverse the whole col-umn/row of the mesh along the positive direction and thenconnect the two boundary nodes by the negative channels.The long path connecting the boundary nodes will incur largerb value as opposed to our dilated-2 embedding.

4.2 Performance Analysis and ComparisonComparisons on the broadcast latency, under one-port andall-port models, are summarized in Table 4 and Table 5,respectively. Note that the best values of k3 and k4 for EDSFand NP-EDSFM in Table 4 are, respectively,

k

LT T T

L LT T T L

LT T T

nc s

nf

nc s

nf

nc s

nf

3

1

1

1

1 2 2 1

2 2

2 2

9

=

+ <

+ >

+

%

&KK

'KK

+

+

+

if

if

otherwise

4 9 4 94 9 4 9

4 9 4 9 .

( )

k

n dc s

n df

d n dc s

n df

d

n dc s

n df

LT T T

L LT T T L

LT T T

4

1 2 2 2 1

2 2 2 2 2

2 2 2

2 1

2 1

2 1

10

=− +

− +

− +

+ − <

+ − >

+ −

%

&KKK

'KKK

if

if

otherwise

4 94 94 94 9

4 94 9 .

( )

Fig. 15 draws the latency of all schemes in a 25 � 25 meshunder various values of L. The general trend is similar tothat in tori—that RD works the best under fairly small mes-sages, ours work the best under median-size messages, SCoutperforms ours at larger messages (� 15K when Ts/Tc =20 and � 40K when Ts/Tc = 300), and EDSF is only usefulwhen L is extremely large.

In Table 5, it shows that our NP-D has higher a and bfactors. But the g = O(n) of the D-node scheme is reduced byour NP-D to almost a constant if d is set to O(log n). So NP-D should be much faster D-node as L is large. Fig. 16 showsthe comparison under various situations. As can be seen,only under small messages (� 64 bytes when Ts/Tc = 20 and� 1K when Ts/Tc = 300) will the D-node scheme work bet-ter. As the message becomes larger the benefit of using ourNP-D algorithm enlarges. Also, by comparing Figs. 16a and16b, it indicates that a smaller ratio of Ts/Tc tends to giveour NP-D more advantage.

5 ONE-TO-ALL BROADCAST IN AN ALL-PORTHYPERCUBE

In this section, we apply our network-partitioning ap-proach to an all-port n-cube. Without loss of generality, let


n � 2

n TORUS UNDER THE ALL-PORT MODEL

(a) (b)

Fig. 14. Performance comparison of EDN and NP-EDN schemes in a 25 � 2

5 torus with Tf = 2 msec and Tc = 0.5 msec. (a) Ts = 10 msec and Ts /Tc = 20,

(b) Ts = 150 msec and Ts /Tc = 300.


the source node be 00 L 0. The DDNs and DCNs are asdefined in Definition 7. The possible numbers of DDNs and

DCNs are 2d and 2n-d, respectively, d = 1..n. The message tobe broadcast is M of size L bytes. The representative node of

DDN of the form b1 b2 L bd*n-d is b1 b2 L bd0

n-d.

5.1 Phases 1, 2, and 3Note that all representative nodes together form a subcube*d0n-d. In Phase 1, we construct the well-known binomialspanning tree [13] in subcube *d0n-d rooted at 00 L 0. Thenthe root scatters the submessage through the tree. The costcan be easily found to be

TABLE 3SUMMARY OF HOW OUR NP-BASED SCHEMES WORK IN A 2

n � 2

n MESH


n � 2

n MESH UNDER THE ONE-PORT MODEL


n � 2

n MESH UNDER THE ALL-PORT MODEL


T dT dT LTcs f d c1 1

1

2= + + −

��

�� .

In Phase 2, all DDNs simultaneously perform the broad-cast scheme by Ho and Kao [10] for an all-port hypercube.This scheme (termed as HK) utilizes a near-optimal numberof start-ups, but may incur high transmission cost (as willbe seen later, the latter cost can be reduced by applying ournetwork-partitioning approach). In an n-cube, the HKscheme takes time THK(n) = anTs + bnTf +gn LTc, where theparameters an, bn, gn can be derived recursively as follows:

α γ α

β β

α α γβ β β β

n n n n

n n n

i i

n

n n

i

= = + ≥

= + ≥

= = = ≤ ≤= = = =

− +

− +

1 5

5

1 2 2 4

1 2 4 5

2

2

1

1

1

1 2 3 4

log

log

,

,

, ,

, , , .

As our DDNs are (n - d)-cubes, the cost becomes

T T T LThkn d s n d f

n dd c2 2

= + +− −−α β

γ.

In Phase 3, in each DCN, each node has a submessage tobe broadcast. This is in fact an all-to-all broadcast and canbe done using the standard exchange algorithm in [13]. Thecost can be easily derived to be

T dT dT LTcs f d c3 1

1

2= + + +

��

�� .

Fig. 17 shows an example how broadcasting from node0000 is done in a 4-cube with d set to two.

5.2 Performance Analysis and ComparisonTable 6 compares the HK and our NP-HK schemes. Oura value is larger than the HK’s (in fact, the HK’s avalue is very close to optimum). On the contrary, our gvalue is much less than the HK’s. For ease of comparison,we list the a, b, and g values of HK and NP-HK when thenetwork is a 5-cube, 10-cube, or 15-cube in Table 7.

(a) (b)

Fig. 15. Broadcast latency at various message sizes in a one-port 25 � 2

5 mesh with h = 2

3, Tf = 2 msec, and Tc = 0.5 msec. (a) Ts = 10 msec and

Ts /Tc = 20, (b) Ts = 150 msec and Ts /Tc = 300.

(a) (b)

Fig. 16. Broadcast latency in an all-port 25 � 2

5 mesh with Tf = 2 msec and Tc = 0.5 msec. (a) Ts = 10 msec and Ts /Tc = 20, (b) Ts = 150 msec and

Ts /Tc = 20.


(a) (b)

(c) (d)

Fig. 17. An example of broadcasting in an all-port 4-cube: (a) Phase 1, (b) Phase 2, (c) first step of Phase 3, (d) second step of Phase 3.

TABLE 6COMPARISON OF BROADCAST COSTS IN AN ALL-PORT n-CUBE

TABLE 7COMPARISON OF a, b, g VALUES IN A 5-CUBE, 10-CUBE, AND 15-CUBE AT VARIOUS d VALUES

(a) (b)

Fig. 18. Performance comparison of HK and NP-HK in a 10-cube with Tf = 2 msec and Tc = 0.5 msec. (a) Ts = 10 msec and Ts /Tc = 20, (b) Ts = 150 msec

and Ts /Tc = 20.


To understand the interaction among the a, b, g fac-tors, in Fig. 18, we depict the costs of HK and NP-HK ina 10-cube with Ts = (20 msec or 150 msec), Tf = 2 msec, andTc = 0.5 msec for various message sizes. The results indi-cate that HK is better when L is small (� 64 bytes whenTs/Tc = 20, and � 0.75K when Ts/Tc = 300). The largerratio of Ts/Tc does give HK more advantage, due to thefact that our a value is larger. But as L increases, signifi-cant gain can be obtained by using NP-HK.

As a final comment, we note that the recently proposedscheme for broadcasting on all-port hypercubes by Wangand Ku [32] uses a more-near-to-optimal number of start-ups than that of the HK scheme. We can plug this schemeinto Phase 2 easily. However, only very limited numbers ofphases (typically one or two for reasonable ns) will besaved and the routing is not dimensional-ordered. Our testsalso showed a similar scenario as discussed above, so thedetails are omitted here.

6 CONCLUSIONS

In this paper, we have presented a network-partitioningapproach for one-to-all broadcast in wormhole networks.The approach works based on constructing multiple in-dependent subnetworks which can work concurrently toincrease the parallelism in communication. The network-partitioning-based approach distinguishes itself from thetraditional edge-disjoint-spanning-trees-based approach (e.g.,[4], [13], [18], [31], [5]) in many fundamental aspects.First, instead of trees, the subnetworks could be of anytopology. Second, instead of a fixed constant, the numberof independent subnetworks could be an adjustable pa-rameter. Last, instead of a standard graph, a subnetworkcould be a “dilated” graph and, thus, the special dis-tance-insensitive characteristic of wormhole routing canbe better utilized. As to future research, we feel thatthese fundamental issues may be used as a guideline indesigning other collective communication patterns inwormhole networks.

We have also shown how to apply this network-partitioning approach to tori, meshes, and hypercubes. Oneinteresting phenomenon is that many existing algorithmsdesigned for these networks can be easily plugged into ourschemes and be used by our dilated subnetworks. Exten-sive analyses, comparisons, and simulations have beenperformed when plugging in these alternatives and the re-sults do confirm the advantage of using our network-partitioning approach in certain, usually reasonable, situa-tions and configurations. All these have strongly justifiedthe practical value of this approach.

Finally, the analyses in this work are based on a synchro-nous model, i.e., the communication steps are performed oneafter without interference. So the depth-contention problem[16] can be further studied when the communication stepsare performed without synchronization.

ACKNOWLEDGMENTS

This research was supported in part by the National ScienceCouncil of the Republic of China under Grant no. NSC87-

2213-E-008-012 and Grant no. NSC87-2213-E-008-016. Apreliminary version of this paper appeared in the Proceed-ings of the Symposium on Parallel and Distributed Processing,1996 [33].

REFERENCES

[1]� V. Bala, J. Bruck, R. Cypher, P. Elustondo, A. Ho, C.T. Ho, S. Kip-mis, and M. Snir, “CCL: A Portable and Tunable Collective Com-munication Library for Scalable Parallel Computer,” Proc. Int’lParallel Processing Symp., pp. 835-843, Cancun, Mexico, Apr. 1994.

[2]� M. Barnett, S. Gupta, D.G. Payne, L. Shuler, R. van de Geijn, and J.Watts, “Interprocessor Collective Communication Library (Inter-Com),” Proc. Scalable High Performance Computing Conf., pp. 357-364, 1994.

[3]� M. Barnett, D.G. Payne, R. van de Geijn, and J. Watts, “Broad-casting on Meshes with Worm-Hole Routing,” J. Parallel and Dis-tributed Computing, vol. 35, pp. 111-121, 1996.

[4]� J.-C. Bermond, P. Michallon, and D. Trystram, “Broadcasting inWraparound Meshes with Parallel Monodirectional Link,” ParallelComputing, vol. 18, pp. 639-648, 1992.

[5]� J. Bruck, L.D. Coster, N.Dewulf, C.-T. Ho, and R. Lauwereins, “Onthe Design and Implementation of Broadcast and Global CombineOperations Using Postal Model,” IEEE Trans. Parallel and Distrib-uted Systems, vol. 7, no. 3, pp. 256-265, Mar. 1996.

[6]� Cray T3E Scalable Parallel Processing System. Cray Research Inc.,1995.

[7]� W. Dally and C. Seitz, “The Torus Routing Chip,” J. DistributedComputing, vol. 1, no. 3, pp. 187-196, 1986.

[8]� B. Duzett and R. Buck, “An Overview of the nCUBE3Supercomputer,” Proc. Symp. Frontiers of Massively Parallel Compu-tation, pp. 458-464, 1992.

[9]� R.L. Graham, D.E. Knuth, and O. Patashnik, Concrete Mathematics.Addison-Wesley, 1994.

[10]� C.-T. Ho and M.-Y. Kao, “Optimal Broadcasting in All-PortWormhole-Routed Hypercubes,” IEEE Trans. Parallel and Distrib-uted Systems, vol. 6, no. 2, pp. 200-204, Feb. 1995.

[11]� A Touchstone DELTA System Description. Intel Corp., 1990.[12]� S.L. Johnson, “Communication Efficient Basic Linear Algebra

Computations on Hypercubes Architectures,” J. Parallel and Dis-tributed Computing, vol. 4, no. 2, pp. 133-172, 1991.

[13]� S.L. Johnson and C.-T. Ho, “Optimum Broadcasting and Person-alized Communication in Hypercubes,” IEEE Trans. Computers,vol. 39, no. 9, pp. 1,249-1,268, Sept. 1989.

[14]� R.E. Kessler and J.L. Schwazmeier, “CRAY T3D: A New Dimen-sion for Cray Research,” Proc. COMPCON ’93, pp. 176-182, 1993.

[15]� P.K. McKinley, Y.-J. Tsai, and D.F. Robinson, ”Collective Commu-nication in Wormhole-Routed Massively Parallel Computers,”Computer, vol. 28, no. 12, pp. 39-50, Dec. 1995.

[16]� P.K. McKinley, H. Xu, A.-H. Esfahanian, and L.M. Ni, “Unicast-Based Multicast Communication in Wormhole-Routed Networks,”IEEE Trans. Parallel and Distributed Systems, vol. 5, no. 12, pp. 1,252-1,265, Dec. 1994.

[17]� ”Document for Standard Message-Passing Interface,” MessagePassing Interface Forum, Nov. 1993.

[18]� P. Michallon and D. Trystram, “Minimum Depth Arcs-DisjointSpanning Trees for Broadcasting on Wrap-Around Meshes,” Proc.Int’l Conf. Parallel Processing, vol. 1, pp. 80-83, 1995.

[19]� L.M. Ni and P.K. McKinley, “A Survey of Wormhole Routing Tech-niques in Directed Network,” Computer, vol. 26, no. 2, pp. 62-76,Feb. 1993.

[20]� P.R. Nuth and W.J. Dally, “The J-Machine Network,” Proc. IEEE Int’lConf. Computer Design: VLSI in Computer and Processors, pp. 420-423,1992.

[21]� J. Park, H.G. Kim, S. Hwang, J. Kim, I. Jang, H. Yoon, and J.W. Cho,“An Efficient Unicast-Based Multicast Algorithm in Two-PortWormhole-Routed 2D Mesh Networks,” Proc. IEEE Int’l Conf. Algo-rithms and Architecture for Parallel Processing, pp. 326-331, 1996.

[22]� D.F. Robinson, P.K. McKinley, and B.H.C. Cheng, “Optimal Multi-cast Communication in Wormhole-Routed Torus Networks,” IEEETrans. Parallel and Distributed Systems, vol. 6, no. 10, pp. 1,029-1,042,Oct. 1995.

[23]� m. Schmidt-Voigt, “Efficient Parallel Communication with thenCUBE 2S Processor,” Parallel Computing, vol. 20, pp. 509-530, 1994.


[24]� Y.-J. Tsai and P.K. McKinley, “A Dominating Set Model for Broad-casting in All-Port Wormhole-Routed 2D Mesh Networks,” Proc.ACM Int’l Conf. Supercomputing, pp. 126-135, 1994.

[25]� Y.-J. Tsai and P.K. McKinley, “An Extended Dominating Node Ap-proach to Collective Communication in All-Port Wormhole-Routed2D Meshes,” Proc. Scalable High Performance Conf., pp. 199-206,Knoxville, Tenn., May 1994.

[26]� Y.-J. Tsai and P.K. McKinley, “A Broadcasting Algorithm for All-PortWormhole-Routed Torus Networks,” Proc. Symp. Frontiers of Mas-sively Parallel Computation, pp. 529-536, 1995.

[27]� Y.-J. Tsai and P.K. McKinley, “A Broadcasting Algorithm for All-PortWormhole-Routed Torus Networks,” IEEE Trans. Parallel and Dis-tributed Systems, vol. 7, no. 8, pp. 876-885, Aug. 1996.

[28]� Y.-C. Tseng and S.K.S. Gupta, “All-To-All Personalized Communi-cation in a Wormhole-Routed Torus,” IEEE Trans. Parallel and Dis-tributed Systems, vol. 7, no. 5, pp. 498-505, May 1996.

[29]� Y.-C. Tseng, T.-H. Lin, S.K.S. Gupta, and D.K. Panda, “Bandwidth-Optimal Complete Exchange on Wormhole-Routed 2D/3D TorusNetworks: A Diagonal-Propagation Approach,” IEEE Trans. Paralleland Distributed Systems. vol. 8, no. 4, pp. 380-396, Apr. 1997.

[30]� Y.-C. Tseng, D.K. Panda, and T.-H. Lai, “A Trip-Based MulticastingModel in Wormhole-Routed Networks with Virtual Channels,”IEEE Trans. Parallel and Distributed Systems, vol. 7, no. 2, pp. 138-150,Feb. 1996.

[31]� Y.-C. Tseng and J.-P. Sheu, “Toward Optimal Broadcast in StarGraph Using Multiple Spanning Trees,” IEEE Trans. Computers,vol. 46, no. 5, pp. 593-599, May 1997.

[32]� C.-M. Wang and C.-Y. Ku, “A Near-Optimal Broadcasting Algo-rithm in All-Port Wormhole-Routed Hypercubes,” Proc. ACM Int’lConf. Supercomputing, pp. 147-153, 1995.

[33]� S.-Y. Wang, Y.-C. Tseng, and C.-W. Ho, “Efficient Single-NodeBroadcast in Wormhole-Routed Multicomputers: A Network-Partitioning Approach,” Proc. Symp. Parallel and Distributed Proc-essing, 1996.

[34]� H. Xu, P.K. McKinley, and L.M. Ni, “Efficient Implementation of BarrierSynchronization in Wormhole-Routed Hypercubes Multicomputers,” J.Parallel and Distributed Computing, vol. 16, pp. 172-184, 1992.

Yu-Chee Tseng received his BS and MS de-grees in computer science from the NationalTaiwan University and the National Tsing-HuaUniversity in 1985 and 1987, respectively. From1989 to 1990, he worked for the WANG Labora-tory and D-LINK Inc. as a software engineer. Heobtained his PhD in computer and informationscience from Ohio State University in January of1994. From February 1994 to July 1996, he waswith the Department of Computer Science,Chung-Hua University, Taiwan. Since August

1996, he has been an associate professor in the Department of Com-puter Science and Information Engineering, National Central Univer-sity, Chung-Li, Taiwan. Dr. Tseng has served on the program commit-tee of the International Conference of Parallel and Distributed Sys-tems, 1996, and on the program committee of the International Con-ference on Parallel Processing, 1998. His research interests includeparallel and distributed computing, fault-tolerant computing, parallelcomputer architecture, wireless network, and mobile computing. Dr.Tseng is a member of the IEEE Computer Society and the ACM.

San-Yuan Wang received his BS and PhD de-grees from the Department of Computer Scienceand Information Engineering of the Tamkang Uni-versity and the National Central University in 1991and 1998, respectively. His research interestsinclude parallel and distributed computing, parallelcomputer architecture, and mobile computing.

Chin-Wen Ho received the BS in mathematicsfrom National Taiwan University in 1979, and MSand PhD degrees in computer sciences fromNational Tsing Hua University, Hsinchu, Taiwan,in 1984 and 1988, respectively. He is an associ-ate processor in the Department Of ComputerSciences and Information Engineering at Na-tional Central University, Chung-Li, Taiwan. He isa member of the IEEE Computer Society. Hisresearch interests include algorithm design andanalysis, graph theory, and parallel processing.

Documents

Efficient broadcasting in wormhole-routed multicomputers: a network-partitioning approach