IEE, October 2001Nick McKeown1 High Performance Routers Slides originally by Nick McKeown Professor of Electrical Engineering and Computer Science, Stanford

IEE, October 2001Nick McKeown 1

High PerformanceSwitching and RoutingTelecom Center Workshop: Sept 4, 1997.

High Performance Routers

Slides originally by Nick McKeownProfessor of Electrical Engineering and Computer Science, Stanford University

[email protected]/~nickm

Stanford High Performance Networking group: http://klamath.stanford.edu

2IEE, October 2001Nick McKeown

Outline

Background What is a router? Why do we need faster routers? Why are they hard to build?

Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.

The Future


What is Routing Forwarding?

R3

A

B

C

R1

R2

R4 D

E

FR5

R5F

R3E

R3D

Next HopDestination

DD


What is Routing?

R3

A

B

C

R1

R2

R4 D

E

FR5

R5F

R3E

R3D

Next HopDestination

D

DDD

16 3241

Data

Options (if any)

Destination Address

Source Address

Header ChecksumProtocolTTL

Fragment OffsetFlagsFragment ID

Total Packet LengthT.ServiceHLenVer

20

byte

s


What is Routing?

A

B

C

R1

R2

R3

R4 D

E

FR5


Points of Presence (POPs)

A

B

C

POP1

POP3POP2

POP4 D

E

F

POP5

POP6 POP7POP8


Where High Performance Routers are Used

R10 R11

R4

R13

R9

R5

R2R1 R6

R3 R7

R12

R16R15

R14

R8

(2.5 Gb/s)

(2.5 Gb/s)(2.5 Gb/s)

(2.5 Gb/s)


What a Router Looks Like

Cisco GSR 12416 Juniper M160

6ft

19”

2ft

Capacity: 160Gb/sPower: 4.2kW

3ft

2.5ft

19”

Capacity: 80Gb/sPower: 2.6kW


Generic Router Architecture

LookupIP Address

UpdateHeader

Header ProcessingData Hdr Data Hdr

1M prefixesOff-chip DRAM

AddressTable

AddressTable

IP Address Next Hop

QueuePacket

BufferMemoryBuffer

Memory1M packetsOff-chip DRAM



LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

Data Hdr

Data Hdr

Data Hdr

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

Data Hdr

Data Hdr

Data Hdr


Why do we Need Faster Routers?

1. To prevent routers becoming the bottleneck in the Internet.

2. To increase POP capacity, and to reduce cost, size and power.


0,1

1

10

100

1000

10000

1985 1990 1995 2000

Spec

95In

t CPU

resu

ltsWhy we Need Faster Routers

1: To prevent routers from being the bottleneck

0,1

1

10

100

1000

10000

1985 1990 1995 2000

Fib

er

Ca

pa

cit

y (

Gb

it/s

)

TDM DWDM

Packet processing Power Link Speed

2x / 18 months 2x / 7 months

Source: SPEC95Int & David Miller, Stanford.


POP with smaller routers

Why we Need Faster Routers

2: To reduce cost, power & complexity of POPs

POP with large routers

Ports: Price >$100k, Power > 400W. It is common for 50-60% of ports to be for interconnection.


Why are Fast Routers Difficult to Make?

1. It’s hard to keep up with Moore’s Law:

The bottleneck is memory speed. Memory speed is not keeping up

with Moore’s Law.



Speed of Commercial DRAM



with Moore’s Law.

0.001

0.01

0.1

1

10

100

1000

1980 1983 1986 1989 1992 1995 1998 2001

Acc

ess

Tim

e (n

s)

Moore’s Law2x / 18 months

1.1x / 18 months





with Moore’s Law. 2. Moore’s Law is too slow:

Routers need to improve faster than Moore’s Law.


Router Performance Exceeds Moore’s Law

Growth in capacity of commercial routers: Capacity 1992 ~ 2Gb/s Capacity 1995 ~ 10Gb/s Capacity 1998 ~ 40Gb/s Capacity 2001 ~ 160Gb/s Capacity 2003 ~ 640Gb/s

Average growth rate: 2.2x / 18 months.


Outline



The Future


RouteTableCPU Buffer

Memory

LineInterface

MAC

LineInterface

MAC

LineInterface

MAC

Typically <0.5Gb/s aggregate capacity

First Generation Routers

Shared Backplane

Line Interface

CPU

Memory


Second Generation RoutersRouteTableCPU

LineCard

BufferMemory

LineCard

MAC

BufferMemory

LineCard

MAC

BufferMemory

FwdingCache

FwdingCache

FwdingCache

MAC

BufferMemory

Typically <5Gb/s aggregate capacity


Third Generation Routers

LineCard

MAC

LocalBuffer

Memory

CPUCard

LineCard

MAC

LocalBuffer

Memory

Switched Backplane

Line Interface

CPUMem

ory FwdingTable

RoutingTable

FwdingTable

Typically <50Gb/s aggregate capacity


Fourth Generation Routers/Switches

Optics inside a router for the first time

Switch Core Linecards

Optical links

100sof metres

0.3 - 10Tb/s routers in development


Outline



The Future



LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

LookupIP Address

AddressTable

AddressTable

LookupIP Address

AddressTable

AddressTable

LookupIP Address

AddressTable

AddressTable


IP Address Lookup

Why it’s thought to be hard:1. It’s not an exact match: it’s a

longest prefix match. 2. The table is large: about 120,000

entries today, and growing. 3. The lookup must be fast: about 30ns

for a 10Gb/s line.


IP Lookups find Longest Prefixes

128.9.16.0/21 128.9.172.0/21

128.9.176.0/24

0 232-1

128.9.0.0/16142.12.0.0/1965.0.0.0/8

128.9.16.14

Routing lookup: Find the longest matching prefix (aka the most specific route) among all prefixes that match the destination address.


IP Address Lookup




for a 10Gb/s line.


Address Tables are Large

0

20000

40000

60000

80000

100000

120000

140000

Aug-87 May-90 J an-93 Oct-95 J ul-98 Apr-01 J an-04

Num

ber

of

prefi

xes

Source: Geoff Huston, Oct 2001


IP Address Lookup




for a 10Gb/s line.


Lookups Must be Fast

12540Gb/s2003

31.2510Gb/s2001

7.812.5Gb/s1999

1.94622Mb/s1997

40B packets (Mpkt/s)

LineYear


IP Address LookupBinary tries

Example Prefixes:a) 00001b) 00010c) 00011d) 001e) 0101f) 011g) 100h) 1010i) 1100j) 11110000

e

f g

h i

j

0 1

a b c

d


Prefix Length Distribution

99.5% prefixes are 24-bits or shorter

0

10000

20000

30000

40000

50000

60000

70000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Prefix Length

Num

ber

of P

refi

xes

Source: Geoff Huston, Oct 2001


24-8 Direct Lookup Trie

0000……0000 1111……1111

0 224-1

24 bits

8 bits

0 28-1

When pipelined, allows one lookup per memory access. Although inefficient use of memory, total memory cost < $20.


IP Address LookupSummary

Lookup limited by memory bandwidth.

Lookup uses high-degree trie.

State of the art: 10Gb/s line rate.Scales to: 40Gb/s line rate.

By 2008, entire IPv4 address space will fit on one $20 DRAM!


Outline



The Future



LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory


Fast Packet Buffers

Example: 40Gb/s packet bufferSize = RTT*BW = 10Gb; 40 byte packets

Write Rate, R

1 packetevery 8 ns

Read Rate, R

1 packetevery 8 ns

BufferManager

BufferMemory

Use SRAM?+ fast enough random access time, but

- too low density to store 10Gb of data.

Use SRAM?+ fast enough random access time, but

- too low density to store 10Gb of data.

Use DRAM?+ high density means we can store data, but- too slow (50ns random access time).

Use DRAM?+ high density means we can store data, but- too slow (50ns random access time).


DRAM Buffer Memory

Packet Caches

Buffer Manager

SRAM

ArrivingPackets

DepartingPackets12

Q

21234

345

123456

Small ingress SRAM cache of FIFO headscache of FIFO tails

5556

9697

8788

57585960

899091

1

Q

2

Small ingress SRAM

1

57 6810 9

79 81011

1214 1315

5052 515354

8688 878991 90

8284 838586

9294 9395 68 7911 10

1

Q

2

DRAM Buffer Memory

b>>1 packets at a time


Packet BuffersSummary

Packet buffers limited by memory bandwidth.

Packet buffer caches use hybrid SRAM+DRAM.

State of the art: 10Gb/s line rate. Scales to: 40Gb/s line rate.


Outline



The Future



LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

Data Hdr

Data Hdr

Data Hdr

1

2

N

1

2

N

N times line rate

N times line rate



LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

Data Hdr

Data Hdr

Data Hdr

1

2

N

1

2

N

Data Hdr

Data Hdr

Data Hdr

Scheduler


0% 20% 40% 60% 80% 100%Load

Del

ayOutput Buffered Switches

The best that any queueing system can

achieve.


0% 20% 40% 60% 80% 100%Load

Del

ayInput Buffered Switches

Head of Line Blocking


achieve.

2 2 58%


Head of Line Blocking


Virtual Output Queues


0% 20% 40% 60% 80% 100%Load

Del

ayA Router with Virtual Output

Queues


achieve.


Maximum Weight Matching

A1(n)

N N

LNN(n)

A1N(n)

A11(n)

L11(n)

1 1

AN(n)

ANN(n)

AN1(n)

D1(n)

DN(n)

L11(n)

LN1(n)

“Request” Graph Bipartite Match

S*(n)

MaximumWeight Match

*

( )( ) arg max( ( ) ( ))T

S nS n L n S n


The Evolution of Switching

Theory:

Practice:

InputQueueing

(IQ)

InputQueueing

(IQ)

InputQueueing

(IQ)

InputQueueing

(IQ)

58% [Karol, 1987]

IQ + VOQ,Maximum weight matching

IQ + VOQ,Maximum weight matching

IQ + VOQ,Sub-maximal size matching

e.g. PIM, iSLIP.

IQ + VOQ,Sub-maximal size matching

e.g. PIM, iSLIP.

100% [M et al., 1995]

Different weight functions,incomplete information, pipelining.

Different weight functions,incomplete information, pipelining.

Randomized algorithmsRandomized algorithms

100% [Tassiulas, 1998]

100% [Various]

Various heuristics, distributed algorithms,

and amounts of speedup

Various heuristics, distributed algorithms,

and amounts of speedup

IQ + VOQ,Maximal size matching,

Speedup of two.

IQ + VOQ,Maximal size matching,

Speedup of two.

100% [Dai & Prabhakar, 2000]


SwitchingSummary

Routers use virtual output queues, and a centralized scheduler.

State of the art: 2.5Tb/s. Scales to: ~10Tb/s.

100% throughput will be standard soon.


Current Internet Router Technology

Summary

There are three potential bottlenecks: Address lookup, Packet buffering, and Switching.

Techniques exist today for: 10+Tb/s Internet routers, with 40Gb/s linecards.

But what comes next…?


Outline



The Future More parallelism. Eliminating schedulers. Introducing optics into routers. Natural evolution to circuit switching?


External Parallelism: Multiple Parallel Routers

What we’d like:R

R R

R

The building blocks we’d like to use:

R

RR

R

NxN

IP Router capacity 100s of Tb/s


Multiple parallel routers Load Balancing

R R

R

1

2

…

…

k

R

R

R

R/k R/k

R

R

R


Intelligent Packet Load-balancing

Parallel Packet Switching

1

2

k

1

N

rate, R

rate, R

rate, R

rate, R

1

N

Router

Bufferless

R/k R/k


Parallel Packet Switching

AdvantagesSingle-stage of bufferingNo excess link capacitykpower per subsystem kmemory bandwidth klookup rate


Parallel Packet SwitchTheorem

If S > 2k/(k+2) 2 then a parallel packet switch can precisely emulate a single big router.


Outline





Eliminating schedulersTwo-Stage Switch [Chang et al., 2001]

1

N

1

N

1

N

External Outputs

Internal Inputs

External Inputs

First Round-Robin Second Round-Robin

Load Balancing


Outline





Do optics belong in routers?

They are already there.Connecting linecards to switches.

Optical processing doesn’t belong on the linecard.You can’t buffer light.Minimal processing capability.

Optical switching can reduce power.


Optics in routers

Switch Core Linecards

Optical links


Complex linecards

PhysicalLayer

Framing&

Maintenance

PacketProcessing

Buffer Mgmt&

Scheduling

Buffer Mgmt&

Scheduling

Buffer & StateMemory


Typical IP Router Linecard

10Gb/s linecard: Number of gates: 30M Amount of memory: 2Gbits Cost: >$20k Power: 300W

LookupTables

SwitchFabric

Arbitration

Optics


Replacing the switch fabric with optics

SwitchFabric

Arbitration

PhysicalLayer

Framing&

Maintenance

PacketProcessing

Buffer Mgmt&

Scheduling

Buffer Mgmt&

Scheduling




LookupTables

Optics

PhysicalLayer

Framing&

Maintenance

PacketProcessing

Buffer Mgmt&

Scheduling

Buffer Mgmt&

Scheduling




LookupTables

Opticselectrical

SwitchFabric

Arbitration

PhysicalLayer

Framing&

Maintenance

PacketProcessing

Buffer Mgmt&

Scheduling

Buffer Mgmt&

Scheduling




LookupTables

Optics

PhysicalLayer

Framing&

Maintenance

PacketProcessing

Buffer Mgmt&

Scheduling

Buffer Mgmt&

Scheduling




LookupTables

Optics

optical

Req/Grant Req/Grant

Candidate technologies 1. MEMs.

2. Fast tunable lasers + passive optical couplers.

3. Diffraction waveguides.4. Electroholographic materials.


160Gb/s

40Gb/s

40Gb/s

40Gb/s

40Gb/s

Optical2-stageSwitch

• Line termination

• IP packet processing

• Packet buffering

• Line termination

• IP packet processing

• Packet buffering

160-320Gb/s

160-320Gb/s

Linecard #1 Linecard #625

100 Tb/s IP Router, 625 linecards, each operating at 160Gb/s.

The Stanford Phicticious Optical Router


Outline





Evolution to circuit switching

• Optics enables simple, low-power, very high capacity circuit switches.

• The Internet was packet switched for two reasons:Expensive links: statistical

multiplexing.Resilience: soft-state routing.

• Neither reason holds today.


Fast Links, Slow Routers

0,1

1

10

100

1000

10000

1985 1990 1995 2000

Fib

er

Cap

acit

y (G

bit

/s)

Fiber optics DWDM0.1

1

10

100

1000

10000

1985 1990 1995 2000

Spe

c95I

nt C

PU

res

ults

Processing Power Link Speed (Fiber)

2x / 2 years 2x / 7 months

Source: SPEC95Int; Prof. Miller, Stanford Univ.


Fewer Instructions

1

10

100

1000

1996 1997 1998 1999 2000 2001

(log

scal

e)

Instructions per packet since 1996


Outline





References

General1. J. S. Turner “Design of a Broadcast packet switching

network”, IEEE Trans Comm, June 1988, pp. 734-743.2. C. Partridge et al. “A Fifty Gigabit per second IP Router”,

IEEE Trans Networking, 1998.3. N. McKeown, M. Izzard, A. Mekkittikul, W. Ellersick, M.

Horowitz, “The Tiny Tera: A Packet Switch Core”, IEEE Micro Magazine, Jan-Feb 1997.

Fast Packet Buffers1. Sundar Iyer, Ramana Rao, Nick McKeown “Design of a fast

packet buffer”, IEEE HPSR 2001, Dallas.


ReferencesIP Lookups1. A. Brodnik, S. Carlsson, M. Degermark, S. Pink. “Small

Forwarding Tables for Fast Routing Lookups”, Sigcomm 1997, pp 3-14.

2. B. Lampson, V. Srinivasan, G. Varghese. “ IP lookups using multiway and multicolumn search”, Infocom 1998, pp 1248-56, vol. 3.

3. M. Waldvogel, G. Varghese, J. Turner, B. Plattner. “Scalable high speed IP routing lookups”, Sigcomm 1997, pp 25-36.

4. P. Gupta, S. Lin, N. McKeown. “Routing lookups in hardware at memory access speeds”, Infocom 1998, pp 1241-1248, vol. 3.

5. S. Nilsson, G. Karlsson. “Fast address lookup for Internet routers”, IFIP Intl Conf on Broadband Communications, Stuttgart, Germany, April 1-3, 1998.

6. V. Srinivasan, G.Varghese. “Fast IP lookups using controlled prefix expansion”, Sigmetrics, June 1998.


ReferencesSwitching• N. McKeown, A. Mekkittikul, V. Anantharam, and J. Walrand. Achieving

100% Throughput in an Input-Queued Switch. IEEE Transactions on Communications, 47(8), Aug 1999.

• A. Mekkittikul and N. W. McKeown, "A practical algorithm to achieve 100% throughput in input-queued switches," in Proceedings of IEEE INFOCOM '98, March 1998.

• L. Tassiulas, “Linear complexity algorithms for maximum throughput in radio networks and input queued switchs,” in Proc. IEEE INFOCOM ‘98, San Francisco CA, April 1998.

• D. Shah, P. Giaccone and B. Prabhakar, “An efficient randomized algorithm for input-queued switch scheduling,” in Proc. Hot Interconnects 2001.

• J. Dai and B. Prabhakar, "The throughput of data switches with and without speedup," in Proceedings of IEEE INFOCOM '00, Tel Aviv, Israel, March 2000, pp. 556 -- 564.

• C.-S. Chang, D.-S. Lee, Y.-S. Jou, “Load balanced Birkhoff-von Neumann

switches,” Proceedings of IEEE HPSR ‘01, May 2001, Dallas, Texas.


ReferencesFuture• C.-S. Chang, D.-S. Lee, Y.-S. Jou, “Load balanced

Birkhoff-von Neumann switches,” Proceedings of IEEE HPSR ‘01, May 2001, Dallas, Texas.

• Pablo Molinero-Fernndez, Nick McKeown "TCP Switching: Exposing circuits to IP" Hot Interconnects IX, Stanford University, August 2001

• S. Iyer, N. McKeown, "Making parallel packet switches practical," in Proc. IEEE INFOCOM `01, April 2001, Alaska.

Documents

IEE, October 2001Nick McKeown1 High Performance Routers Slides originally by Nick McKeown Professor of Electrical Engineering and Computer Science, Stanford