048866: Packet Switch Architectures

Dr. Isaac KeslassyElectrical Engineering, Technion

isaac@ee.technion.ac.ilhttp://comnet.technion.ac.il/~isaac/

Introduction

Spring 2006 048866 – Packet Switch Architectures 2

Course Content

Most of the results in the course are less than 10 years old!

Material is new and still changing…No reference book available yet…

Uses slides by Profs. Nick McKeown and Balaji Prabahakar (Stanford)

Spring 2006 048866 – Packet Switch Architectures 3

Outline

BackgroundWhat is a router?Why do we need faster routers?Why are they hard to build?

Architectures and techniquesThe evolution of router architecture.IP address lookup.Packet buffering.Switching.

Spring 2006 048866 – Packet Switch Architectures 4

What is Routing?

R3

A

B

C

R1

R2

R4 D

E

FR5

R5FR3ER3DNext HopDestination

D

Spring 2006 048866 – Packet Switch Architectures 5

What is Routing?

R3

A

B

C

R1

R2

R4 D

E

FR5

R5FR3ER3DNext HopDestination

D

16 3241

Data

Options (if any)

Destination Address

Source Address

Header ChecksumProtocolTTL

Fragment OffsetFlagsFragment ID

Total Packet LengthT.ServiceHLenVer

20 b

ytes

Spring 2006 048866 – Packet Switch Architectures 6

What is Routing?

A

B

C

R1

R2

R3

R4 D

E

FR5

Spring 2006 048866 – Packet Switch Architectures 7

Points of Presence (POPs)

A

B

C

POP1

POP3POP2

POP4 D

E

F

POP5

POP6 POP7POP8

Spring 2006 048866 – Packet Switch Architectures 8

Where High Performance Routers are Used

R10 R11

R4

R13

R9

R5R2R1 R6

R3 R7

R12

R16R15

R14R8

(10 Gb/s)

(10 Gb/s)(10 Gb/s)

(10 Gb/s)

Spring 2006 048866 – Packet Switch Architectures 9

What a Router Looks Like

Cisco GSR 12416 Juniper M160

6ft

19”

2ft

Capacity: 160Gb/sPower: 4.2kWFull rack

3ft

2.5ft

19”

Capacity: 80Gb/sPower: 2.6kWHalf-a-rack

Spring 2006 048866 – Packet Switch Architectures 10

Core Router Market

Estimates cover different parametersDell’Oro (Feb. 17, 2005):

$1.2 billion in 2004 (up 66%)Includes high-end (10Gbps) router market

Dell’Oro (Feb. 21, 2006):$4.4 billion in 2005Includes “worldwide service provider router market”

Infonetics (Feb. 21, 2006): $5.2 billion in 2005 (up 31%)Includes core/edge routers

Core router marketRelatively small (in comparison to enterprise market), but brings expertise to companyVendors: Cisco, Juniper, Avici, Nortel, Lucent, Alcatel, Huawei, etc.

Spring 2006 048866 – Packet Switch Architectures 11

Router Market

Source: Infonetics (Nov. 2004)

Spring 2006 048866 – Packet Switch Architectures 12

Basic Architectural Componentsof an IP Router

Control Plane

Datapathper-packet processing

SwitchingForwardingTable

RoutingTable

Routing Protocols

Spring 2006 048866 – Packet Switch Architectures 13

Per-packet processing in an IP Router

1. Accept packet arriving on an incoming link.

2. Lookup: lookup packet destination address in the forwarding table, to identify outgoing port(s).

3. Header Processing: Manipulate packet header: e.g., decrement TTL, update header checksum.

4. Switching: Send packet to the outgoing port(s).5. Buffering: Buffer packet in the queue.

6. Transmit packet onto outgoing link.

Spring 2006 048866 – Packet Switch Architectures 14

Generic Router Architecture

LookupIP Address

UpdateHeader

Header ProcessingData Hdr Data Hdr

~1M prefixesOff-chip DRAM

AddressTable

AddressTable

IP Address Next Hop

QueuePacket

BufferMemoryBufferMemory

~1M packetsOff-chip DRAM

Spring 2006 048866 – Packet Switch Architectures 15

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

Generic Router Architecture

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

Data Hdr

Data Hdr

Data Hdr

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

Data Hdr

Data Hdr

Data Hdr

Spring 2006 048866 – Packet Switch Architectures 16

Outline

BackgroundWhat is a router?Why do we need faster routers?Why are they hard to build?

Architectures and techniquesThe evolution of router architecture.IP address lookup.Packet buffering.Switching.

Spring 2006 048866 – Packet Switch Architectures 17

Why Do We Need Faster Routers?

1. To prevent routers from becoming the bottleneck in the Internet.

2. To increase POP capacity, and to reduce cost, size and power.

Spring 2006 048866 – Packet Switch Architectures 18

0,1

1

10

100

1000

10000

1985 1990 1995 2000

Spec

95In

t CPU

resu

ltsWhy We Need Faster Routers

1: To prevent routers from being the bottleneck

Packet Processing Power

2x / 18 months

Source: SPEC95Int & Coffman and Odlyzko.

Single Fiber Capacity(commercial)

≥ 2x / year

Spring 2006 048866 – Packet Switch Architectures 19

POP with smaller routers

Why we Need Faster Routers 2: To reduce cost, power & complexity of POPs

POP with large routers

Ports: Price >$50k, Power > 400W. It is common for 50-60% of ports to be for interconnection.

Spring 2006 048866 – Packet Switch Architectures 20

Why are Fast Routers Difficult to Make?

1. It’s hard to keep up with Moore’s Law:The bottleneck is memory speed.Memory speed is not keeping up with Moore’s Law.

Spring 2006 048866 – Packet Switch Architectures 21

Why are Fast Routers Difficult to Make?Speed of Commercial DRAM

1. It’s hard to keep up with Moore’s Law:The bottleneck is memory speed.Memory speed is not keeping up with Moore’s Law.

0.01

0.1

1

10

100

10001980 1983 1986 1989 1992 1995 1998 2001

Acc

ess

Tim

e (n

s)

Moore’s Law2x / 18 months

1.1x / 18 months

Spring 2006 048866 – Packet Switch Architectures 22

Why are Fast Routers Difficult to Make?

1. It’s hard to keep up with Moore’s Law:The bottleneck is memory speed.Memory speed is not keeping up with Moore’s Law.

2. Moore’s Law is too slow:Routers need to improve faster than Moore’s Law.

Spring 2006 048866 – Packet Switch Architectures 23

Router Performance Exceeds Moore’s Law

Growth in capacity of commercial routers:Capacity 1992 ~ 2Gb/sCapacity 1995 ~ 10Gb/sCapacity 1998 ~ 40Gb/sCapacity 2001 ~ 160Gb/sCapacity 2003 ~ 640Gb/s

Average growth rate: 2.2x / 18 months.

Spring 2006 048866 – Packet Switch Architectures 24

Outline

BackgroundWhat is a router?Why do we need faster routers?Why are they hard to build?

Architectures and techniquesThe evolution of router architecture.IP address lookup.Packet buffering.Switching.

Spring 2006 048866 – Packet Switch Architectures 25

RouteTableCPU Buffer

Memory

LineInterface

MAC

LineInterface

MAC

LineInterface

MAC

Typically <0.5Gb/s aggregate capacity

Shared Backplane

Line Interface

CPU

Memory

First-Generation Routers

Spring 2006 048866 – Packet Switch Architectures 26

RouteTableCPU

LineCard

BufferMemory

LineCard

MAC

BufferMemory

LineCard

MAC

BufferMemory

FwdingCache

FwdingCache

FwdingCache

MAC

BufferMemory

Typically <5Gb/s aggregate capacity

Second-Generation Routers

Spring 2006 048866 – Packet Switch Architectures 27

LineCard

MAC

LocalBufferMemory

CPUCard

LineCard

MAC

LocalBufferMemory

Switched Backplane

Line Interface

CPUMemory Fwding

Table

RoutingTable

FwdingTable

Typically <50Gb/s aggregate capacity

Third-Generation Routers

Spring 2006 048866 – Packet Switch Architectures 28

Switch Core Linecards

Optical links

100sof metres

0.3 - 10Tb/s routers

Fourth-Generation RoutersMulti-Racks, Optical Links

Spring 2006 048866 – Packet Switch Architectures 29

Optical Switch Core Linecards

Optical links

100sof metres

10-100Tb/s routers, in project

(Future) Fifth-Generation RoutersOptical Switch Core

Spring 2006 048866 – Packet Switch Architectures 30

Optical Switch Core Optical Linecards

Optical links

100sof metres

100-1000Tb/s routers, in the far future

(Future) Sixth-Generation RoutersAll-Optical Routers

Spring 2006 048866 – Packet Switch Architectures 31

Outline

BackgroundWhat is a router?Why do we need faster routers?Why are they hard to build?

Architectures and techniquesThe evolution of router architecture.IP address lookup.Packet buffering.Switching.

Spring 2006 048866 – Packet Switch Architectures 32

Generic Router Architecture

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

LookupIP Address

AddressTable

AddressTable

LookupIP Address

AddressTable

AddressTable

LookupIP Address

AddressTable

AddressTable

Spring 2006 048866 – Packet Switch Architectures 33

IP Address Lookup

Why it’s thought to be hard:It’s not an exact match: it’s a longest prefix match. The table is large: about 150,000 entries today, and growing. The lookup must be fast: about 30ns for a 10Gb/s line.

Spring 2006 048866 – Packet Switch Architectures 34

IP Lookups find Longest Prefixes

128.9.16.0/21 128.9.172.0/21

128.9.176.0/24

0 232-1

128.9.0.0/16142.12.0.0/1965.0.0.0/8

128.9.16.14

Routing lookup: Find the longest matching prefix (aka the most specific route) among all prefixes that match the destination address.

Spring 2006 048866 – Packet Switch Architectures 37

IP Address Lookup

Why it’s thought to be hard:It’s not an exact match: it’s a longest prefix match.The table is large: about 150,000 entries today, and growing. The lookup must be fast: about 30ns for a 10Gb/s line.

Spring 2006 048866 – Packet Switch Architectures 38

Address Tables are Large

Source: http://www.cidr-report.org/

Spring 2006 048866 – Packet Switch Architectures 39

IP Address Lookup

Why it’s thought to be hard:It’s not an exact match: it’s a longest prefix match.The table is large: about 150,000 entries today, and growing.The lookup must be fast: about 30ns for a 10Gb/s line.

Spring 2006 048866 – Packet Switch Architectures 40

Lookups Must be Fast

12540Gb/s200331.2510Gb/s20017.812.5Gb/s19991.94622Mb/s1997

40B packets (Mpkt/s)

LineYear

500160Gb/s2005?

Spring 2006 048866 – Packet Switch Architectures 41

Outline

BackgroundWhat is a router?Why do we need faster routers?Why are they hard to build?

Architectures and techniquesThe evolution of router architecture.IP address lookup.Packet buffering.Switching.

Spring 2006 048866 – Packet Switch Architectures 42

Generic Router Architecture

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

BufferManager

BufferMemory

BufferMemory

Spring 2006 048866 – Packet Switch Architectures 43

Fast Packet Buffers

Example: 40Gb/s packet bufferSize = RTT*BW = 10Gb; 40 byte packets

Write Rate, R

1 packetevery 8 ns

Read Rate, R

1 packetevery 8 ns

BufferManager

BufferMemory

Use SRAM?+ fast enough random access time, but- too low density to store 10Gb of data.

Use SRAM?+ fast enough random access time, but- too low density to store 10Gb of data.

Use DRAM?+ high density means we can store data, but- too slow (50ns random access time).

Use DRAM?+ high density means we can store data, but- too slow (50ns random access time).

Spring 2006 048866 – Packet Switch Architectures 44

Outline

BackgroundWhat is a router?Why do we need faster routers?Why are they hard to build?

Architectures and techniquesThe evolution of router architecture.IP address lookup.Packet buffering.Switching.

Spring 2006 048866 – Packet Switch Architectures 45

Generic Router Architecture

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

Data Hdr

Data Hdr

Data Hdr

1

2

N

1

2

N

N times line rate

N times line rate

Spring 2006 048866 – Packet Switch Architectures 46

Generic Router Architecture

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

AddressTable

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

QueuePacket

BufferMemory

BufferMemory

Data Hdr

Data Hdr

Data Hdr

1

2

N

1

2

N

Data Hdr

Data Hdr

Data Hdr

Scheduler

Spring 2006 048866 – Packet Switch Architectures 47

Current Internet Router TechnologySummary

There are three potential bottlenecks:Address lookup,Packet buffering, andSwitching.

Most difficult – and focus of this course!