A Heterogeneous Multiple Network-On-Chip Design: An Application-Aware Approach

A Heterogeneous Multiple Network-On-Chip Design:

An Application-Aware Approach

Asit K. Mishra Chita R. DasOnur Mutlu

2

Executive summary• Problem: Current day NoC designs are agnostic to application requirements and

are provisioned for the general case or worst case. Applications have widely differing demands from the network

• Our goal: To design a NoC that can satisfy the diverse dynamic performance requirements of applications

• Observation: Applications can be divided into two general classes in terms of their requirements from the network: bandwidth-sensitive and latency-sensitive

- Not all applications are equally sensitive to bandwidth and latency

• Key idea: Design two NoC - Each sub-network customized for either BW or LAT sensitive applications - Propose metrics to classify applications as BW or LAT sensitive - Prioritize applications’ packets within the sub-networks based on their sensitivity

• Network design: BW optimized network has wider link width but operates at a lower frequency and LAT optimized network has narrow link width but operates at a higher frequency

• Results: Our proposal is significantly better when compared to an iso-resource monolithic network (5%/3% weighted/instruction throughput improvement and 31% energy reduction)

3

• Channel bandwidth affects network latency, throughput and energy/power

• Increase in channel BW leads to- Reduction in packet serialization- Increase in router power

Resource requirements of various applications - I

Impact of channel bandwidth on application performance

4



Simulation settings:

• 8x8 multi-hop packet based mesh network

• Each node in the network has an OoO processor (2GHz), private L1 cache and a router (2GHz) • Shared 1MB per core shared L2

• 6VC/PC, 2 stage router

5



appl

u

wrf art

deal

sjen

g

barn

es

grm

cs

nam

d

h264 gc

c

pvra

y

tont

o

libq

gobm

k

asta

r

milc

hmm

er

swim

sjbb sa

p

xala

n

sphn

x

bzip

lbm

sjas

sopl

x

cact

s

omne

t

gem

s

mcf

01234567

64b links 128b links 256b links 512b links

IT (

norm

. to

64b

links

)

6



appl

u

wrf art

deal

sjen

g

barn

es

grm

cs

nam

d

h264 gc

c

pvra

y

tont

o

libq

gobm

k

asta

r

milc

hmm

er

swim

sjbb sa

p

xala

n

sphn

x

bzip

lbm

sjas

sopl

x

cact

s

omne

t

gem

s

mcf

01234567


IT (

norm

. to

64b

links

)

1. 18/30 (21/36 total) applications’ performance is agnostic to channel BW (8x BW inc. → less than 2x performance inc.)

7



appl

u

wrf art

deal

sjen

g

barn

es

grm

cs

nam

d

h264 gc

c

pvra

y

tont

o

libq

gobm

k

asta

r

milc

hmm

er

swim

sjbb sa

p

xala

n

sphn

x

bzip

lbm

sjas

sopl

x

cact

s

omne

t

gem

s

mcf

01234567


IT (

norm

. to

64b

links

)

1. 18/30 (21/36 total) applications’ performance is agnostic to channel BW (8x BW inc. → less than 2x performance inc.)

2. 12/30 (15/36 total) applications’ performance scale with increase in channel BW (8x BW inc. → at least 2x performance inc.)

8

• Reduction in router latency (by increasing frequency)- Reduction in packet latency- Increase in router power consumption

Impact of network latency on application performance

Resource requirements of various applications - II

9

Resource requirements of various applications - II

Simulation settings:

• … same as last experiment

• 128b links

• Added dummy stages (2-cycle and 4-cycle ) to each router


10

Resource requirements of various applications - IIap

plu

wrf art

deal

sjen

g

barn

es

grm

cs

nam

d

h264 gc

c

pvra

y

tont

o

libq g

asta

r

milc h

swim

sjbb sa

p

xala

n

sphn

x

bzip

lbm

sjas

sopl

x

cact

s

omne

t

gem

s

mcf

0.5

0.6

0.7

0.8

0.9

1.0

1.12-cycle router 4-cycle router 6-cycle router

IT (n

orm

. to

2-cy

cle ro

uter

)


11


plu

wrf art

deal

sjen

g

barn

es

grm

cs

nam

d

h264 gc

c

pvra

y

tont

o

libq g

asta

r

milc h

swim

sjbb sa

p

xala

n

sphn

x

bzip

lbm

sjas

sopl

x

cact

s

omne

t

gem

s

mcf

0.5

0.6

0.7

0.8

0.9

1.0


IT (n

orm

. to

2-cy

cle ro

uter

)

1. 18/30 (21/36 total) applications’ performance is sensitive to network latency (3x latency reduction → at least 25% performance improvement)


12


plu

wrf art

deal

sjen

g

barn

es

grm

cs

nam

d

h264 gc

c

pvra

y

tont

o

libq g

asta

r

milc h

swim

sjbb sa

p

xala

n

sphn

x

bzip

lbm

sjas

sopl

x

cact

s

omne

t

gem

s

mcf

0.5

0.6

0.7

0.8

0.9

1.0


IT (n

orm

. to

2-cy

cle ro

uter

)

2. 12/30 (15/36 total) applications’ performance is marginally sensitive to network latency (3x latency increase → less than 15% performance improvement)

1. 18/30 (21/36 total) applications’ performance is sensitive to network latency (3x latency reduction → at least 25% performance improvement)


13

a

wrf art

deal s ba g

h264 gc

c p t

libq g

asta

r

milc h

sjbb sa

p x s

bzip

lbm

sjas

sopl

x

cact

s o

mcf

0.5

0.7

0.9


IT (n

orm

. to

2-cy

cle

rout

er)

Application-aware approach to designing multiple NoCs ap

plu

wrf art

deal

sjen

g ba g

h264 gc

c

pvra

y

tont

o

libq g

asta

r

milc h

swim

sjbb sa

p

xala

n s

bzip

lbm

sjas

sopl

x

cact

s o

mcf

01234567


IT (

norm

. to

64b

links

)

14

a

wrf art

deal s ba g

h264 gc

c p t

libq g

asta

r

milc h

sjbb sa

p x s

bzip

lbm

sjas

sopl

x

cact

s o

mcf

0.5

0.7

0.9


IT (n

orm

. to

2-cy

cle

rout

er)

Application-aware approach to designing multiple NoCs

Based on the observations:

1. Applications can be classified into distinct classes: typically LAT/BW sensitive2. LAT sensitive applications can benefit from low network latency3. BW sensitive applications can benefit from high network bandwidth4. Not all applications are equally sensitive to either LAT or BW5. Monolithic network cannot optimize both classes simultaneously

appl

u

wrf art

deal

sjen

g ba g

h264 gc

c

pvra

y

tont

o

libq g

asta

r

milc h

swim

sjbb sa

p

xala

n s

bzip

lbm

sjas

sopl

x

cact

s o

mcf

01234567


IT (

norm

. to

64b

links

)

15

a

wrf art

deal s ba g

h264 gc

c p t

libq g

asta

r

milc h

sjbb sa

p x s

bzip

lbm

sjas

sopl

x

cact

s o

mcf

0.5

0.7

0.9


IT (n

orm

. to

2-cy

cle

rout

er)

Application-aware approach to designing multiple NoCs

Solution

Two NoCs where each (sub)network is optimized for either LAT or BW sensitive applications

appl

u

wrf art

deal

sjen

g ba g

h264 gc

c

pvra

y

tont

o

libq g

asta

r

milc h

swim

sjbb sa

p

xala

n s

bzip

lbm

sjas

sopl

x

cact

s o

mcf

01234567


IT (

norm

. to

64b

links

)

16

Design methodology

Processors/L1$ Network L2$ and Mem. Controllers

Logical view of a multicore processor

17

Design methodology



1

Identify LAT/BW sensitive applications- Proposes a novel dynamic application classification scheme

1

18

Design methodology



1


1

2 Design sub-networks based on applications’ demand- This network architecture is better than a monolithic iso-resource

network

2

19

Design methodology



1


1

2 Design sub-networks based on applications’ demand- This network architecture is better than a monolithic iso-resource

network

2

DE

MU

X

20

Design: Dynamic classification of applications

Network episode Compute episode

time

Application life cycleO

utst

andi

ng n

etw

ork

pack

ets

21



time

Application life cycle

• App. has at least one outstanding packet• Processor is likely stalling → low IPC

Out

stan

ding

net

wor

k pa

cket

s

22



time



• App. has no outstanding packet• High IPC

Out

stan

ding

net

wor

k pa

cket

s

23



Epi

sode

hei

ght

Episode length time




Episode length = Number of consecutive cycles there are net. packets

Episode height = Avg. number of L1 packets injected during an episode

Out

stan

ding

net

wor

k pa

cket

s

24



time




Out

stan

ding

net

wor

k pa

cket

s

Short episode ht.: Low MLP, each request is critical (LAT sensitive)Tall episode ht.: High MLP (BW sensitive)

Short episode len.: Packets are very critical (LAT sensitive)Long episode len.: Latency tolerant (could be de-prioritized)

Episode length Epi

sode

hei

ght

25

Classification and rankingClassification Length

Long Medium ShortTall gems, mcf sphinx, lbm, cactus, xalan sjeng, tonto

Height Medium omnetpp, apsiocean, sjbb, sap, bzip,

sjas, soplex, tpc

applu, perl, barnes, gromacs, namd, calculix,

gcc, povray, h264, gobmk, hmmer, astar

Short leslie art, libq, milc, swim wrf, deal

Classification: LAT/BW

26

Classification and ranking

Classification: LAT/BW

Ranking Length Long Medium Short

High Rank-4 Rank-2 Rank-1Height Medium Rank-3 Rank-2 Rank-2

Short Rank-4 Rank-3 Rank-1

Ranking: Sensitivity to LAT/BW

Classification Length Long Medium Short

Tall gems, mcf sphinx, lbm, cactus, xalan sjeng, tonto

Height Medium omnetpp, apsiocean, sjbb, sap, bzip,

sjas, soplex, tpc

applu, perl, barnes, gromacs, namd, calculix,

gcc, povray, h264, gobmk, hmmer, astar

Short leslie art, libq, milc, swim wrf, deal

27

Network design

1N-128

28

Network design

1N-128 2N-64x256-ST(Steering)

29

Network design


2N-64x256-ST-RK(Steering+Ranking)

30

Network design



2N-64x256-ST-RK(FS)(Steering+Ranking and

Frequency Scaling)

31

Network design

1N-128

1N-256

2N-64x256-ST(Steering)


2N-64x256-ST-RK(FS)(Steering+Ranking and

Frequency Scaling)

1N-512 (High BW)2N-128X128

1N-320(iso-BW)

1N-320(FS)(iso-resource)

32

Analysis

1N-1

28

1N-2

56

2N-1

28x1

28

1N-5

12

2N-6

4x256

-ST

2N-6

4x256

-ST+R

K(no F

S)

2N-6

4x256

-ST+R

K(FS)

1N-3

20(n

o FS)

1N-3

20 (F

S)0

10

20

30

40

50

60

Wei

ghte

d sp

eedu

p

Performance (25 WL with 50% BW and 50% LAT)

33

1N-1

28

1N-2

56

2N-1

28x1

28

1N-5

12

2N-6

4x256

-ST

2N-6

4x256

-ST+R

K(no F

S)

2N-6

4x256

-ST+R

K(FS)

1N-3

20(n

o FS)

1N-3

20 (F

S)0

10

20

30

40

50

60

Wei

ghte

d sp

eedu

p

Analysis


34

1N-1

28

1N-2

56

2N-1

28x1

28

1N-5

12

2N-6

4x256

-ST

2N-6

4x256

-ST+R

K(no F

S)

2N-6

4x256

-ST+R

K(FS)

1N-3

20(n

o FS)

1N-3

20 (F

S)0

10

20

30

40

50

60

Wei

ghte

d sp

eedu

p

Analysis

+18%


35

1N-1

28

1N-2

56

2N-1

28x1

28

1N-5

12

2N-6

4x256

-ST

2N-6

4x256

-ST+R

K(no F

S)

2N-6

4x256

-ST+R

K(FS)

1N-3

20(n

o FS)

1N-3

20 (F

S)0

10

20

30

40

50

60

Wei

ghte

d sp

eedu

p

Analysis


+7%+18%

36

1N-1

28

1N-2

56

2N-1

28x1

28

1N-5

12

2N-6

4x256

-ST

2N-6

4x256

-ST+R

K(no F

S)

2N-6

4x256

-ST+R

K(FS)

1N-3

20(n

o FS)

1N-3

20 (F

S)0

10

20

30

40

50

60

Wei

ghte

d sp

eedu

p

Analysis


+5%+7%+18%

37

1N-1

28

1N-2

56

2N-1

28x1

28

1N-5

12

2N-6

4x256

-ST

2N-6

4x256

-ST+R

K(no F

S)

2N-6

4x256

-ST+R

K(FS)

1N-3

20(n

o FS)

1N-3

20 (F

S)0

10

20

30

40

50

60

Wei

ghte

d sp

eedu

p

Analysis


5%+5%

+7%+18%

38

1N-1

28

1N-2

56

2N-1

28x1

28

1N-5

12

2N-6

4x256

-ST

2N-6

4x256

-ST+R

K(no F

S)

2N-6

4x256

-ST+R

K(FS)

1N-3

20(n

o FS)

1N-3

20 (F

S)0

10

20

30

40

50

60

Wei

ghte

d sp

eedu

p

Analysis


w. 2%5%

+5%+7%+18%

39

1N-1

28

1N-2

56

2N-1

28x1

28

1N-5

12

2N-6

4x256

-ST

2N-6

4x256

-ST+R

K(no F

S)

2N-6

4x256

-ST+R

K(FS)

1N-3

20(n

o FS)

1N-3

20 (F

S)0

10

20

30

40

50

60

Wei

ghte

d sp

eedu

p

Analysis


w. 2% w. 2%5%

+5%+7%+18%

40

1N-1

28

1N-2

56

2N-1

28x1

28

1N-5

12

2N-6

4x256

-ST

2N-6

4x256

-ST+R

K(no F

S)

2N-6

4x256

-ST+R

K(FS)

1N-3

20(n

o FS)

1N-3

20 (F

S)0

10

20

30

40

50

60

Wei

ghte

d sp

eedu

p

Analysis


w. 2% w. 2%5%

+5%+7%+18%

1N-128

1N-256

2N-128x128

1N-512

2N-64x256-ST

2N-64x256-ST+

RK(no FS)

2N-64x256-ST+RK(FS)

1N-320(no FS)

1N-320 (FS)

0

0.4

0.8

1.2

1.6

2

Nor

mal

ized

ene

rgy

Energy (25 WL with 50% BW and 50% LAT)

- 47%- 59%

41

1N-1

28

1N-2

56

2N-1

28x1

28

1N-5

12

2N-6

4x256

-ST

2N-6

4x256

-ST+R

K(no F

S)

2N-6

4x256

-ST+R

K(FS)

1N-3

20(n

o FS)

1N-3

20 (F

S)0

10

20

30

40

50

60

Wei

ghte

d sp

eedu

p

Analysis


w. 2% w. 2%5%

+5%+7%+18%

1N-128

1N-256

2N-128x128

1N-512

2N-64x256-ST

2N-64x256-ST+

RK(no FS)

2N-64x256-ST+RK(FS)

1N-320(no FS)

1N-320 (FS)

0

0.4

0.8

1.2

1.6

2

Nor

mal

ized

ene

rgy

Energy (25 WL with 50% BW and 50% LAT)

- 47%- 59%

Best EDP across all designs

42

Conclusions• Problem: Current day NoC designs are agnostic to application requirements and

are provisioned for the general case or worst case. Applications have widely differing demands from the network

• Our goal: To design a NoC that can satisfy the diverse dynamic performance requirements of applications

• Observation: Applications can be divided into two general classes in terms of their requirements from the network: bandwidth-sensitive and latency-sensitive

- Not all applications are equally sensitive to bandwidth and latency

• Key idea: Design two NoC - Each sub-network customized for either BW or LAT sensitive applications - Propose metrics to classify applications as BW or LAT sensitive - Prioritize applications’ packets within the sub-networks based on their sensitivity

• Network design: BW optimized network has wider link width but operates at a lower frequency and LAT optimized network has narrow link width but operates at a higher frequency

• Results: Our proposal is significantly better when compared to an iso-resource monolithic network (5%/3% weighted/instruction throughput improvement and 31% energy reduction)

43

Thank youQ?

Asit [email protected]

44

Backup Slides . . .

45

Other metrics considered for application classification

0

20

40

60

80

100

120

0

200

400

600

800

1000

1200

1400

1600

1800

2000

L1MPKI L2MPKI Slack

L1/L

2 M

PK

I

Sla

ck (i

n cy

cles

)

46

Analysis of network episode length and height

appl

u

wrf art

deal

sjeng

barn

es

grm

cs

nam

d

h264 gc

c

pvra

y

tont

o

libq

gobm

k

asta

r

milc h

swim

sjbb

sap

xala

n

sphn

x

bzip

lbm

sjas

sopl

x

cact

s

omne

t

gem

s

mcf

02468

101214

Avg.

epi

sode

hei

ght

(net

wor

k pa

cket

s)

appl

u

wrf art

deal

sjeng

barn

es

grm

cs

nam

d

h264 gc

c

pvra

y

tont

o

libq

gobm

k

asta

r

milc

hmm

er

swim

sjbb

sap

xala

n

sphn

x

bzip

lbm

sjas

sopl

x

cact

s

omne

t

gem

s

mcf

0100020003000400050006000

Avg.

epi

sode

leng

th

(in cy

cles)

Short length/heightMedium length/heightLong length/High height

0.3M10K

0.4M18K

47

Analysis of network episode length and height

appl

u

wrf art

deal

sjeng

barn

es

grm

cs

nam

d

h264 gc

c

pvra

y

tont

o

libq

gobm

k

asta

r

milc h

swim

sjbb

sap

xala

n

sphn

x

bzip

lbm

sjas

sopl

x

cact

s

omne

t

gem

s

mcf

02468

101214

Avg.

epi

sode

hei

ght

(net

wor

k pa

cket

s)

appl

u

wrf art

deal

sjeng

barn

es

grm

cs

nam

d

h264 gc

c

pvra

y

tont

o

libq

gobm

k

asta

r

milc

hmm

er

swim

sjbb

sap

xala

n

sphn

x

bzip

lbm

sjas

sopl

x

cact

s

omne

t

gem

s

mcf

0100020003000400050006000

Avg.

epi

sode

leng

th

(in cy

cles)

Short length/heightMedium length/heightLong length/High height

Based on performance scaling sensitivity to bandwidth and frequency

0.3M10K

0.4M18K

48

Empirical results to support the classification SPECjbb (sjbb) as cut-off for BW/LAT sensitive applications

49


50


51


52


Why 9 clusters?

0 5 10 15 20 25 30 350

255075

100125150175200225

Number of clusters

With

in g

roup

sum

of

squa

res

53


Why 9 clusters?

0 5 10 15 20 25 30 350

255075

100125150175200225

Number of clusters

With

in g

roup

sum

of

squa

res 13x

54


Why 9 clusters?

0 5 10 15 20 25 30 350

255075

100125150175200225

Number of clusters

With

in g

roup

sum

of

squa

res

55

Analysis with varying workload combinations

0% BANDWIDTH 100% LATENCY

25% BAND-WIDTH 75% LATENCY




0.7

0.9

1.1

1.3

1.5WS IT

WS

and

IT (n

orm

. to

1N-1

28 n

et.)

56

Comparison to prior works

1N-1

28-S

TC

1N-1

28-S

T+R

K

2N-1

28x1

28-L

D-B

AL

2N-6

4x25

6-W

-LD

-BA

L

2N-6

4x25

6-S

T+R

K...

0.8

1.0

1.2

1.4WS IT

WS

and

IT (n

orm

. to

1N-1

28 n

et.)

57

Dynamic steering of packets

appl

u art

barn

es

nam

d

gcc

libq

asta

r

hmm

er

lesl

ie

calc

ulix

sjen

g

sap

sphn

x

lbm

sopl

x

omne

t

mcf

ocea

n

0%

20%

40%

60%

80%

100%

Latency-optimized network

% p

acke

ts in

sub

-net

wor

k

58

Design: Putting it all together



MU

X

59




MU

X

Classify applications based on sensitivity to network BW/LAT

60


Episode LEN/HT



MU

X


Use network episode length/height to dynamically

identify apps

61


Episode LEN/HT

Design LAT/BW optimized networks



MU

X



identify apps

62



Episode LEN/HT

Design LAT/BW optimized networks



MU

X


identify appsPrioritization within

networks

63

Summary

• A NoC paradigm based on top-down approach (application demand/requirement analysis)

• An efficient design paradigm for future heterogeneous multicores

64

Summary



Small core

GPGPUs

Accelerators/ ASIC

Latency critical

Throughput (BW) critical


Latency critical (real-

time constraints)

Big core

65

Summary



Small core

GPGPUs

Accelerators/ ASIC

Latency critical

Big core

Providing all these guarantees in one network is hard

Multiple networks: each customized for one metric



Latency critical (real-

time constraints)

66

Summary


• An efficient design paradigm for future heterogeneous multicore m/c

Latency

Throughput

Local communication

Long haul comm.

Power

1 cycle/ bufferless/Faster

routers

1 cycle/ high bandwidth

Power efficient links/DVFS router

Hybrid/ fewer connectivity

network

Butterfly/express channels

MU

XShare 2D space or 3D layers

Episode LEN/HT/??

Documents

A Heterogeneous Multiple Network-On-Chip Design: An Application-Aware Approach