Winter 2006EE384x Handout 11 EE384x: Packet Switch Architectures Handout 1: Logistics and Introduction Professor Balaji Prabhakar

Winter 2006 EE384x Handout 1 1

EE384x: Packet Switch Architectures

Handout 1: Logistics and Introduction

Professor Balaji [email protected]

Professor Nick [email protected]


OutlineThis two course sequence is about the theory and practice of designing

packet switches and Internet routers.

1. Introduction: What is a packet switch? The evolution of Internet routers, their basic architectural components, and some example architectures.

2. Part I: Output Queued Switches (Emphasis on Deterministic Analysis) OQ as the simplest and ideal architecture.Output queueing and shared-memory switches. Packet arrival processes: ()-constrained arrivals, leaky buckets, Bernoulli arrivals, bursty arrivals, adversaries.Providing bandwidth and delay guarantees, scheduling, fairness, Fair-Queueing, Generalized Processor Sharing and Deficit Round Robin.Practical difficulties: When output queued switches are impractical.Memory bandwidth and capacity scaling. Some approaches: Emulating output queued switches. Parallel packet buffers as standalone shared memory, with design examples. Routers with a single stage of buffering and constraint sets, Parallel Shared Memory Routers, Distributed Shared Memory Routers, and Parallel Packet Switches. Output link scheduling in a Distributed Shared Memory router. Combined input and output queued (CIOQ) switches, stable marriage matchings.


Outline3. Part II: Input Queued Switches (Emphasis on

Probabilistic Analysis). What is an input-queued (IQ) switch?Definition of IQ switch with single FCFS queue. Switching fabrics, crossbars. Head of line blocking. The balls and bins model. Proof of Karol's 2-sqrt(2) (58%) result. Virtual output queues and crossbar schedulers. Bipartite Matchings: Maximum Sized Matchings, Maximum Weight Matchings, maximal matchings. Definitions of 100% throughput. When traffic is uniform: simple RR and random matchings. When traffic matrix is known: Birkhoff- von Neuman decomposition. When traffic is not known: heuristics. PIM, iSLIP, WFA.

4. Fundamentals (Review Sessions): Introduction to probability, Poisson process, Discrete and Continuous-time Markov chains. Basic queueing theory: M/M/1, M/G/1, Little’s result, PASTA.


Outline EE384y

1. Part II: Input Queued Switches (Continued).Intro to Lyapunov functions, proof that max weight matching gives 100% throughput.Some case studies: The Tiny Tera architecture. The Cisco GSR 12000.

2. Part III: Other Switch ArchitecturesBuffered crossbars. Scaling crossbars and parallelism. Multistage switches: Clos networks, 2-stage switches (random and deterministic).

3. Part IV: Other Switch FunctionsAddress Lookup: Exact matches, longest prefix matches, performance metrics, hardware and software solutions. Packet Classification: For firewalls, QoS, and policy-based routing; graphical description and examples of 2-D classification, examples of classifiers, theoretical and practical considerations.

4. Special topics.5. Project presentations.

5 EE384x Handout 1 Winter 2006

Some logisticsWeb page: http://www.stanford.edu/class/ee384x

Course assistant:Denise Murphy – [email protected] 267; Tel: (650) 723-4731

TAs:Mohsen Bayati [email protected] Dukkipati [email protected]

Grades:You need to sign up with “eeclass” on the EE384x web page.

mailto:[email protected]

mailto:[email protected]


More LogisticsPrerequisite

EE284/CS244A and familiarity with probability.Useful

Stats 116 (or EE178, EE278) and CS161Papers

URLs to all the papers are on the eeclass web page.Grading

(40%) 5 Problem sets (10%) Several surprise quizzes (20%) In-class midterm exam (February 21) (30%) Final exam (Thursday March 23, 3:30 - 6:30 PM)

SITN Students Same schedule as in-class students Fax your assignment to us: (650) 618-1938

All deadlines are hard!


An IntroductionThe class starts here!

Background What is a router? Why do we need faster routers? Why are they hard to build?

Architectures and techniques The evolution of router architecture. IP address lookup. Packet buffering. Switching.


What is Routing?R3

A

B

C

R1

R2

R4 D

E

FR5

R5FR3ER3DNext HopDestination

D


What is Routing?R3

A

B

C

R1

R2

R4 D

E

FR5

R5FR3ER3DNext HopDestination

D

16 3241

DataOptions (if any)

Destination AddressSource Address

Header ChecksumProtocolTTLFragment OffsetFlagsFragment ID

Total Packet LengthT.ServiceHLenVer

20 b

ytes


What is Routing?

A

B

C

R1

R2

R3

R4 D

E

FR5


Points of Presence (POPs)

A

B

C

POP1

POP3POP2

POP4 D

E

F

POP5

POP6 POP7 POP8


Where High Performance Routers are Used

R10 R11

R4

R13

R9

R5R2R1 R6

R3 R7

R12

R16R15

R14R8

(2.5 Gb/s)

(2.5 Gb/s) (2.5 Gb/s)

(2.5 Gb/s)


What a Router Looks LikeCisco GSR 12416 Juniper M160

6ft

19”

2ft

Capacity: 160Gb/sPower: 4.2kW

3ft

2.5ft

19”

Capacity: 80Gb/sPower: 2.6kW


Basic Architectural Components

of an IP Router

Control Plane

Datapathper-packet processing

SwitchingForwardingTable

Routing Table

Routing Protocols


Per-packet processing in an IP Router

1. Accept packet arriving on an incoming link.2. Lookup packet destination address in the

forwarding table, to identify outgoing port(s).

3. Manipulate packet header: e.g., decrement TTL, update header checksum.

4. Send packet to the outgoing port(s).5. Buffer packet in the queue.6. Transmit packet onto outgoing link.


Generic Router Architecture

LookupIP Address

UpdateHeader

Header ProcessingData Hdr Data Hdr

~1M prefixesOff-chip DRAM

AddressTable

IP Address Next Hop

QueuePacket

BufferMemory

~1M packetsOff-chip DRAM


Generic Router ArchitectureLookup

IP AddressUpdateHeader

Header Processing

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

Data Hdr

Data Hdr

Data Hdr

BufferManager

BufferMemory

BufferManager

BufferMemory

BufferManager

BufferMemory

Data Hdr

Data Hdr

Data Hdr


Why do we Need Faster Routers?

1. To prevent routers becoming the bottleneck in the Internet.

2. To increase POP capacity, and to reduce cost, size and power.


0,1

1

10

100

1000

10000

1985 1990 1995 2000

Spec

95In

t CPU

resu

ltsWhy we Need Faster Routers

1: To prevent routers from being the bottleneck

0,1

1

10

100

1000

10000

1985 1990 1995 2000

Fibe

r Cap

acity

(Gbi

t/s)

TDM DWDM

Packet processing Power Link Speed

2x / 18 months 2x / 7 months

Source: SPEC95Int & David Miller, Stanford.


POP with smaller routers

Why we Need Faster Routers 2: To reduce cost, power & complexity of

POPsPOP with large routers

Ports: Price >$100k, Power > 400W. It is common for 50-60% of ports to be for interconnection.


Why are Fast Routers Difficult to Make?

1. It’s hard to keep up with Moore’s Law: The bottleneck is memory speed. Memory speed is not keeping up with Moore’s Law.



Speed of Commercial DRAM


0.001

0.01

0.1

1

10

100

10001980 1983 1986 1989 1992 1995 1998 2001

Acce

ss T

ime

(ns)

Moore’s Law2x / 18 months

1.1x / 18 months




2. Moore’s Law is too slow: Routers need to improve faster than Moore’s Law.


Router Performance Exceeds Moore’s Law

Growth in capacity of commercial routers: Capacity 1992 ~ 2Gb/s Capacity 1995 ~ 10Gb/s Capacity 1998 ~ 40Gb/s Capacity 2001 ~ 160Gb/s Capacity 2003 ~ 640Gb/s

Average growth rate: 2x / 18 months.


OutlineBackground

What is a router? Why do we need faster routers? Why are they hard to build?



RouteTableCPU Buffer

Memory

LineInterface

MAC

LineInterface

MAC

LineInterface

MAC

Typically <0.5Gb/s aggregate capacity

First Generation RoutersShared Backplane

Line Interface

CPUMemory


Second Generation RoutersRouteTableCPU

LineCard

BufferMemory

LineCard

MAC

BufferMemory

LineCard

MAC

BufferMemory

FwdingCache

FwdingCache

FwdingCache

MAC

BufferMemory

Typically <5Gb/s aggregate capacity


Third Generation Routers

LineCard

MAC

LocalBuffer

Memory

CPUCard

LineCard

MAC

LocalBuffer

Memory

Switched Backplane

Line Interface

CPUMemory Fwding

Table

RoutingTable

FwdingTable

Typically <50Gb/s aggregate capacity


Fourth Generation Routers/Switches

Optics inside a router for the first time

Switch Core Linecards

Optical links

100sof metres

0.3 - 10Tb/s routers in development


OutlineBackground






Header Processing

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

BufferManager

BufferMemory

BufferManager

BufferMemory

BufferManager

BufferMemory

LookupIP Address

AddressTable

LookupIP Address

AddressTable

LookupIP Address

AddressTable


IP Address LookupWhy it’s thought to be hard:

1. It’s not an exact match: it’s a longest prefix match. 2. The table is large: about 150,000 entries today,

and growing. 3. The lookup must be fast: about 30ns for a 10Gb/s

line.


IP Lookups find Longest Prefixes

128.9.16.0/21128.9.172.0/21

128.9.172.0/24

0 232-1

128.9.0.0/16142.12.0.0/1965.0.0.0/8

128.9.16.14

Routing lookup: Find the longest matching prefix (aka the most specific route) among all prefixes that match the destination address.





line.


Address Tables are Large

Source: http://www.cidr-report.org/





line.


Lookups Must be Fast

12540Gb/s200331.2510Gb/s20017.812.5Gb/s19991.94622Mb/s1997

40B packets (Mpkt/s)

LineYear


OutlineBackground






Header Processing

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

QueuePacket

BufferMemory

QueuePacket

BufferMemory

QueuePacket

BufferMemory

BufferManager

BufferMemory

BufferManager

BufferMemory

BufferManager

BufferMemory


Fast Packet BuffersExample: 40Gb/s packet buffer

Size = RTT*BW = 10Gb; 40 byte packets

Write Rate, R1 packet

every 8 ns

Read Rate, R

1 packetevery 8 ns

BufferManager

BufferMemory

Use SRAM?+ fast enough random access time, but- too low density to store 10Gb of data.

Use DRAM?+ high density means we can store data, but- too slow (50ns random access time).


OutlineBackground






Header Processing

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

QueuePacket

BufferMemory

QueuePacket

BufferMemory

QueuePacket

BufferMemory

Data Hdr

Data Hdr

Data Hdr

1

2

N

1

2

N

N times line rate

N times line rate




Header Processing

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

LookupIP Address

UpdateHeader

Header Processing

AddressTable

QueuePacket

BufferMemory

QueuePacket

BufferMemory

QueuePacket

BufferMemory

Data Hdr

Data Hdr

Data Hdr

1

2

N

1

2

N

Data Hdr

Data Hdr

Data Hdr

Scheduler

Winter 2006 EE384x Handout 1 440% 20% 40% 60% 80% 100%

Load

Dela

yA Router with Input Queues

The best that any queueing system can

achieve.

Winter 2006 EE384x Handout 1 450% 20% 40% 60% 80% 100%Load

Dela

yA Router with Input Queues

Head of Line Blocking


achieve.

2 2 58%


Head of Line Blocking


Virtual Output Queues

Winter 2006 EE384x Handout 1 480% 20% 40% 60% 80% 100%

Load

Dela

yA Router with Virtual Output

Queues


achieve.


Maximum Weight MatchingA1(n)

N N

LNN(n)

A1N(n)

A11(n)L11(n)

1 1

AN(n)

ANN(n)

AN1(n)

D1(n)

DN(n)

L11(n)

LN1(n)

“Request” Graph Bipartite Match

S*(n)

MaximumWeight Match

*

( )( ) argmax( ( ) ( ))T

S nS n L n S n


Outline of Proof

( 1) ( 1 .

( )

) ( ) (

[ ( )

) ( )

]

( )

We can show that:

Hence, if is large enough, there is an expected single-step downward drif t in occupancy. and 100

T TE L n L n L n L n |

L n

E L n

L n c L n

%throughput is achieved.


There are now many ways to achieve 100% throughput…

52

Scheduling algorithms to achieve 100% throughput

1. Basic switch model.2. When traffi c is unif orm (Many algorithms…)3. When traffi c is non-unif orm, but traffi c matrix is known.

• Technique: Birkhoff -von Neumann decomposition.4. When matrix is not known.

• Technique: Lyapunov function.5. When algorithm is pipelined, or inf ormation is incomplete.

• Technique: Lyapunov function.6. When algorithm does not complete.

• Technique: Randomized algorithm.7. When there is speedup.

• Technique: Fluid model.8. When there is no algorithm.

• Technique: 2-stage load-balancing switch.• Technique: Parallel Packet Switch.


The Evolution of SwitchingTheory:

Practice:

InputQueueing

(IQ)

InputQueueing

(IQ)

58% [Karol, 1987]

IQ + VOQ,Maximum weight matching

IQ + VOQ,Sub-maximal size matching

e.g. PIM, iSLIP.

100% [M et al., 1995]

Different weight functions,incomplete information, pipelining.

Randomized algorithms

100% [Tassiulas, 1998]

100% [Various]

Various heuristics, distributed algorithms,

and amounts of speedup

IQ + VOQ,Maximal size matching,

Speedup of two.100% [Dai & Prabhakar, 2000]


Current Internet Router TechnologySummary

There are three potential bottlenecks: Address lookup, Packet buffering, and Switching.

Techniques exist today for: 10+Tb/s Internet routers, with 40Gb/s linecards.

Documents

Winter 2006EE384x Handout 11 EE384x: Packet Switch Architectures Handout 1: Logistics and Introduction Professor Balaji Prabhakar