Routers with a Single Stage of Buffering

Routers with a Single Stage of Buffering

Sundar Iyer, Rui Zhang, Nick McKeownHigh Performance Networking Group,Stanford University, http://yuba.stanford.edu

ACM SIGCOMMFriday, Aug 23rd 2002

2

Contents

1. Background & motivation

2. An abstraction of a router with the pigeon-hole principle

3. Analysis of a FIFO work conserving router

4. Analysis of a PIFO work conserving router

5. Comparison and conclusion

3

How do we design routers?We pick a capacity target & architecture

Then & only then do we simulate

Load

Delay

0 100

We hope customers will put up with unpredictable performance

We can’t guarantee, that the router is work conserving

4

How can we re-design routers?

Then, we see how we can build a router

We require that a router be work conserving

We pick a capacity target & architecture

We take the departure time of a packet as a requirement

5

Motivation (A different way of thinking about routers)

This talk is about routers which are designed from the ground up.

We start with the departure times of the packets & take them as a requirement.

Then, we figure out the number of resources that we need to ensure that packets depart at their departure time.

6

A look at single buffered routersA router with 3 ports with memories which run at the line rate

R

R R

R

C1

A1B1C2

C3C2C3

B1

A1

C1C-4

FIFO Departure Time = 4

Destined to output C

C4

Arriving packetsPackets in buffersEmpty buffersFinal arriving packet

7

Contents

1. Background & motivation

2. An abstraction of a router with the pigeon-hole principle

3. Analysis of a FIFO work conserving router

4. Analysis of a PIFO work conserving router

5. Comparison and conclusion

8

An abstract model There are P pigeon holes which can contain an

infinite number of pigeons.

Assume that time is slotted, and in any one time slot at most N pigeons can arrive and at most N can depart. at most one pigeon can enter or leave a specific pigeon hole.

When a pigeon arrives, we know the exact time slot at which it will depart.

For any router: What is the minimum P, such that all N pigeons can be

immediately placed in a pigeon hole when they arrive, and can depart at the right time?

9

Solving the abstract model When a pigeon arrives in a time slot it must

satisfy the following constraints

No more than N – 1 other pigeons arrive at that timeslot. No more than N other pigeons depart at that timeslot. No more than N - 1 other pigeons can depart at the same

time as this pigeon departs (perhaps in future).

By the pigeon-hole principle, 3N –1 pigeon holes are sufficient for each pigeon to arrive

and leave the pigeon holes without conflict.

10

A take away ..

If you want to build a work conserving router …

Evaluate the uniquely assigned departure time for every packet.

Define the constraints on every packet.

Apply the pigeon-hole principle.

Find the minimum resources required so that every packet departs at the right time.

11

Contents

1. Background & motivation

2. An abstraction of a router with the pigeon-hole principle

3. Analysis of a FIFO work conserving router

4. Analysis of a PIFO work conserving router

5. Comparison and conclusion

12

E.g. 1: Parallel Shared Memory Router

R

A router with 3 ports and k=7 memories running at rate R

R

R R

R

B

A

C

k = 7 memories at rate R is not sufficient .... but 8 memories suffice

R

13

R

A router with 3 ports and k=8 memories running at rate R

R

R R

R

B

A

C

By the pigeon-hole principle, k 3N –1 memories at rate R, suffices.

R

How many memories for FIFO? (Parallel Shared Memory)

14

Switch Fabric

LineCard

LineCard

LineCard

Buffer Buffer Buffer

E.g. 2: DSM Router (Distributed Shared Memory)

The central memories have been moved to distributed line cards.

15

Why is the DSM router interesting? The bandwidth in & out of each memory

is at the line rate.

The memory is shared across inputs and outputs.

Memory and line cards can be added incrementally.

16

A router with 3 ports and k=9 memories running at rate R

By the pigeon-hole principle, k 3N -1 memories at rate R, suffices.

LineCard

LineCard

LineCard

Switch Fabric

How many memories for FIFO? (Distributed Shared Memory)

17

Previous Work (which use the counting/pigeon-hole principle)

Combined Input Output Queued (CIOQ) Router Prabhakar and McKeown, -1997 Chuang, Goel, McKeown & Prabhakar -1998 Krishna, Patel, Charny, Simcoe –1998 Charny - 1998

Parallel Packet Switch (PPS) Iyer, Awadallah & McKeown – 2000 Iyer & McKeown - 2001

18

Contents

1. Background & motivation

2. An abstraction of a router with the pigeon-hole principle

3. Analysis of a FIFO work conserving router

4. Analysis of a PIFO work conserving router

5. Comparison and conclusion

19

What is the Problem with non-FIFO? …1

First Problem: The departure time of a cell is not fixed in policies such

as strict priority, weighted fair queueing etc.

The counting technique depends on being able to predict the departure time and schedule it.

We will use an model called PIFO which captures a number of these scheduling policies for analysis

20

A Push in First Out (PIFO) Queue

Arriving packets are “pushed-in” to an arbitrary location in the departure queue.

Packets depart from the head of line.

The relative ordering between packets in the queue does not change.

21

Enabling QoS with PIFO queues

C1B2 B1 A3 A2 A1C1

WFQ with 3 per-flow queues, destined to same output, Weights 3:2:1

C2B1B2A1A2A3

C2

B2 B1 A3 A2 A1C1B3 B3

A single PIFO queue for an output, Weights 3:2:1

C2

22

How many memories for PIFO? (Parallel Shared Memory)

R

A router with 3 ports and k=9 memories running at rate R

R

2.5R R

R

B

A

C

R

2 41 3

k = 9 memories at rate R is not sufficient.. but 10 memories suffice

2.4

23

How many memories are needed? (Distributed Shared Memory)

A cell which arrives at time t, destined to output port j must not be written to memories which

1. Are used to write the other N-1 arriving cells at t.

2. Are used to read the N departing cells at t.

3. Will be used to read the N-1 cells of output j before t.

4. Will be used to read the N-1 cells of output j after t.

There are four constraint sets By the pigeon-hole principle, 4N-2 memories at rate R, or

a memory bandwidth of 4NR is sufficient

24

Contents

1. Background & motivation

2. An abstraction of a router with the pigeon-hole principle

3. Analysis of a FIFO work conserving router

4. Analysis of a PIFO work conserving router

5. Comparison and conclusion

25

Comparison of FIFO & PIFO (Distributed Shared Memory)

ArbiterBW of fabric

BW of Mem.

BW of fabric

BW. Of Mem.

Hard4NR4NR3NRDSM Router

5NR

Easy4NR6NR3NR 8NR

Easy6NR4NR4NR 6NR

FIFO PIFO

26

Comparison with the CIOQ Router

CIOQDSM

2NR5NRTotal Fabric BW6NR4NRTotal Memory BW

The DSM has a lower memory bandwidth than the CIOQ, which is the main bottleneck in routers.

The DSM arbiter is simpler than the CIOQ arbiter.

27

Conclusion There was a class of routers which were not

analyzed earlier

Distributed/Parallel shared memory

Existing techniques were not applicable

We came up with a counting technique based on the pigeon-hole principle

Single buffered routers Combined Input-Output Queued router