Vakgroep Informatietechnologie – Onderzoeksgroep IBCN Label scalability of Carrier Ethernet Benchmarking Carrier Ethernet Technologies Workshop Session

Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

Label scalability of Carrier Ethernet

Benchmarking Carrier Ethernet Technologies Workshop Session AI.2 - Scientific and Technical ResultsEuroNGI 2008, Krakow, Poland April 30, 2008

Wouter Tavernier, Koen Casier (Ghent University – IBBT)Luis Caro (University of Girona)Dimitri Papadimitriou (Alcatel-Lucent Bell NV)

Label scalability & efficiency of Carrier Ethernet – Wouter TavernierVakgroep Informatietechnologie – Onderzoeksgroep IBCN

Agenda

PROBLEM STATEMENT

Labels in Carrier Ethernet Label scalability Label optimization

Results

Short word on label lookups

Conclusion


Problem statement

Evolution of Ethernet being a LAN technology towards a Carrier Technology Ethernet = low cost Ethernet = ubiquitous Ethernet = plug & play

LAN environments: traffic streams between tens of end-users Metro/Access environments: traffic streams between thousands of end-users

PROBLEM STATEMENT: Is Carrier Ethernet able to cope with increasing number of traffic streams

IP/ MPLS

ISP1

ISPn

Video

WWW

LargeCO

Level 2remotes

RegionalPOP

RU/MTU

Small CO /Level 1remotes

AggregationNetwork

[ring / star]

< 5,5 km< 1,5km 2,1 - 3,0 km

ADSL/ADSL2+

VDSL

FE/GigEor PON

ADSL

RAM /DLC

FE/GigE/SONET

CLE

BRAS

CustomerPremises

RegionalNetwork

BroadbandServices

Aggregator

AccessNetwork

[ring / star]

BNG

EthernetAggregation

(optional)

< 50 km

@ 50% penetration@ 10% penetration

# of lines

# of customers 50-50010-100

100-1K

100-2K20-400

200-4K

5K-10K

1K-2K

10K-20K

25K-50K

5K-10K

50K-100K

SDSL (business) FE/GigE/SONET/CDWM

OLT /CO DSLAM

Metro Core Core

Metro Aggregation

Metro Access

First Mile

Ethernet

Around 5 to 10 nodes

IP/ MPLS

ISP1

ISPn

Video

WWW

LargeCO

Level 2remotes

RegionalPOP

RU/MTU


AggregationNetwork

[ring / star]

< 5,5 km< 1,5km 2,1 - 3,0 km

ADSL/ADSL2+

VDSL

FE/GigEor PON

ADSL

RAM /DLC

FE/GigE/SONET

CLE

BRAS

CustomerPremises

RegionalNetwork

BroadbandServices

Aggregator

AccessNetwork

[ring / star]

BNG

EthernetAggregation

(optional)

< 50 km


# of lines


100-1K

100-2K20-400

200-4K

5K-10K

1K-2K

10K-20K

25K-50K

5K-10K

50K-100K


OLT /CO DSLAM

Metro Core Core

Metro Aggregation

Metro Access

First Mile

Ethernet


LAN Metro networkL


No labels in bridged Ethernet

A

B

C

X

Y

Z

ZC

12

3

512

3

4

Dest Out Port

X 1

Z 3

A 4

Dest Out Port

B 2

X 5

Z 5

C 3

Forwarding in native Ethernet bridging : stateless (CL) 48 bit MAC-address based

SA PayloadDA

7 6 6

Sync SD

1 46 - 1500

EthType

2

ZC

ZC

ZC

ZC


Labels in Carrier Ethernet

A

B C

D

E

The concept of a connection (LSP) allows for (CO): Traffic Engineering Advanced recovery/protection techniques BW guarantees

Forwarding is based on a label linked to the connection state

Two connections (LSPs) from A to ETwo connections (LSPs) from A to E: maintains state for the green and red connection and forwards based on a

label

Labell

Labell


Domain-wide PBB-TE label

A

B C

D

E

<100, E>

<200, E>

<100, A>

Distinct routes for dest E, by diff B-VID.

Ingress A has 2 LSPs to E, for e.g.: Protection switching Load balancing

B-VID 100 reused on same link

C-DA C-SA C-TAG

C-DA C-SA S-TAG Type DATA

Type DATA

C-TAG

B-DA B-SA B-TAG 802.1ad FramePrio I-SID

FCS

FCS

FCS

6 octets 6 octets 4 octets 2 octets 2 octets

6 octets 6 octets 4 octets 4 octets 2 octets 2 octets

6 octets 6 octets 4 octets 1 octet 3 octets 2 octets

Customer Frame

802.1ad S-tagged Frame

802.1ah I-tagged Frame

Label remains constant along connection (no SWAP)!


E,2

00

E,200

E,200


C-DA C-SA C-TAG


Type DATA

C-TAG


FCS

FCS

FCS




Customer Frame



Link local ELS label

A

B C

D

E

Distinct routes for dest E, by diff S-VID.

Ingress A has 2 LSPs to E, for e.g.: Protection switching Load balancing


100

Label can be swapped in intermediate hops along connection!

200

B-VID 100 reused on same link

100

S-VID 100 reused on other link


Label usage optimization

Label balancing (online routing optimized ELS) Shortest path routing takes into account labels used on a link as

cost Merging (ELS)

1 incoming label per interface + 1 outgoing label GAIN: x-1 labels on outgoing local interface

Shared forwarding (PBB-TE) 1 label needed GAIN: x-1 global labels

.

.

.x incoming


Label scalability

Given the label:

How do different label schemes react under changing conditions?

What is the influence of the topology? What is the influence the traffic matrix? What is the influence of the traffic matrix on typical BW

usage?

C-DA C-SA C-TAG


Type DATA

C-TAG


FCS

FCS

FCS




Customer Frame



ELS

PBB-TE


Study assumptions

Study the impact on label usage of the following dimensions: Topology (CONNECTEDNESS) Traffic matrix (SIZE, UNIFORMITY) BW usage

NORMALIZATION: Tests have run 100 times Only one dimension has been changed in a time

Shortest Path Routing (Dijkstra hop count) is taken as base routing algorithm

Base topology of 100 nodes (connectedness +-2) & 1000 demands

AB

C

D

E

1 23

4-9

10i

ii

L

iii

x

a

b

L


Connectedness of a network

Connectedness of a topology: what is the average node degree of a node in a topology: ring vs. full mesh

Example: Single (un-)connected 4-connected Full mesh


Connectedness of the topology

0

50

100

150

200

250

300

350

400

450

500

0 10 20 30 40 50 60 70 80 90

Average node degree

lab

el

sp

ac

e r

eq

uir

ed

Link Local Link Local (bal.) Link Local (M)Node Local Node Local (bal.) Node Local (M)Global Global (SF)

S


Connectedness vs BW (Luis)

The topology generator considered for generating the topologies in this section is IGEN

For all the topologies the link capacity is set to 10Gb/s and bandwidth request of 100Mb, 200Mb and 300Mb are generated until the network is overloaded.

The implemented algorithm is the SPF

Overload network with links of 10G capacity A


Maximum number of labels

0

500

1000

1500

2000

2500

3000

3500

2 10 20 30 40 50 60 70 80 90 100

node degree

Labels per link

Labels per link with agg

Labels per link withagg+merging

Labels per destination

Labels per destinationwith agg

Labels per destinationwith agg_ VLAn reut

Labels per destinationwith agg+inv trees

Connectedness vs. BW (Luis)


Maximum Number of Labels

0

10

20

30

40

50

60

2 10 20 30 40 50 60 70 80 90 100

node degree


Labels perdestination with agg_VLAn reut

Labels perdestination withagg+inv trees

Connectedness vs. BW (Luis)



020406080

100120140160180

number of nodes

Labels per link

Labels per link withagg






Topology size vs. BW (Luis)


Topology size vs. BW (Luis)


020406080

100120140160180

20 40 60 80 100

120

140

160

180

200

number of nodes

Labels per link

Labels per link withagg







Uniformity of the traffic matrix

UNIFORMITY: X demands directed to 1 vs. x destinations?

Example: 4 demands 2 destinations vs 4 destinations

4 paths / 1 dest 4 paths / 2 dest 4 paths / 4 dest


Uniformity of the traffic matrix

0

50

100

150

200

250

300

350

400

450

500

0 10 20 30 40 50 60 70 80 90 100

Number of nodes used as destination

lab

el

sp

ac

e r

eq

uir

ed


S


Size of the traffic matrix

The size of a traffic matrix affects the number of demands that are routed over a network:

10 demands 100 demands 1000 demands 10000 demands …


Scaling the order of the traffic

0

50

100

150

200

250

300

350

400

450

500

0 1000 2000

Traffic streams

lab

el

sp

ac

e r

eq

uir

ed


S


Lookups Must be Fast

1. Lookup mechanism must be simple and easy to implement2. Memory access time is the bottleneck

200Mpps × 2 lookups/pkt = 400 Mlookups/sec → 2.5ns per lookup

Year Aggregate Line-rate

Arriving rate of 40B POS packets (Million pkts/sec)

1997 622 Mb/s 1.56

1999 2.5 Gb/s 6.25

2001 10 Gb/s 25

2003 40 Gb/s 100

2006 80 Gb/s 200


Direct lookup in ELS vs PBB-TE

•ELS: •Label space is 12 bits•With 64b data, this is 256 K of memory. • Label space is private to one link • Therefore, table size can be “negotiated”

SVID/MPLS-label

Addre

ss

Memory

Data

(Outgoing Port, new SVID label)

Direct Memory Lookup

•PBB-TE: •Label space is 48+12 bits•With 64b data, this is 64 EiB of memory (2^60). • Label space is global•2^60 > 2^12, therefore cannot hold all addresses in table and use direct lookup, less efficient alternatives:•Hashing •Binary/Multi-way Search Trie/Tree


Conclusion

The specific traffic matrix & topology used, clearly affect label usage

The uniformity of the traffic matrix affects PBB-TE-type domain-wide labelling: more uniform is better

The topology connectedness affects ELS-type link-local labelling: more connected is better

In typical metro-network with low node-degree, PBB-TE & ELS LL have similar performance

PBB-TE with SF and ELS with merging consistently score better than alternatives

Node local labelling techniques consistently score worse than alternatives.

The label length affects the memory space needed and accordingly affects cost and lookup speed


Appendices


Carrier-Grade Ethernet challenges

Flat address space scalability of number of MAC addresses to be learned

Beyond 100k learned addresses per node seems challenging

Scalability in terms of number of VLANs 12-bit VLAN ID (VID): 4k VLANs possible network wide

STP cannot converge faster than worst case 20s sec (root failure)

Traffic Engineering (routing) constraint by tree based structure.

Limited OA&M


IP/ MPLS

ISP1

ISPn

Video

WWW

LargeCO

Level 2remotes

RegionalPOP

RU/MTU


AggregationNetwork

[ring / star]

< 5,5 km< 1,5km 2,1 - 3,0 km

ADSL/ADSL2+

VDSL

FE/GigEor PON

ADSL

RAM /DLC

FE/GigE/SONET

CLE

BRAS

CustomerPremises

RegionalNetwork

BroadbandServices

Aggregator

AccessNetwork

[ring / star]

BNG

EthernetAggregation

(optional)

< 50 km


# of lines


100-1K

100-2K20-400

200-4K

5K-10K

1K-2K

10K-20K

25K-50K

5K-10K

50K-100K


OLT /CO DSLAM

Metro Core Core

Metro Aggregation

Metro Access

First Mile

Ethernet


IP/ MPLS

ISP1

ISPn

Video

WWW

LargeCO

Level 2remotes

RegionalPOP

RU/MTU


AggregationNetwork

[ring / star]

< 5,5 km< 1,5km 2,1 - 3,0 km

ADSL/ADSL2+

VDSL

FE/GigEor PON

ADSL

RAM /DLC

FE/GigE/SONET

CLE

BRAS

CustomerPremises

RegionalNetwork

BroadbandServices

Aggregator

AccessNetwork

[ring / star]

BNG

EthernetAggregation

(optional)

< 50 km


# of lines


100-1K

100-2K20-400

200-4K

5K-10K

1K-2K

10K-20K

25K-50K

5K-10K

50K-100K


OLT /CO DSLAM

Metro Core Core

Metro Aggregation

Metro Access

First Mile

Ethernet


Reference network

AB

C

D

E

1 2

3

4-9

10i

ii

L

i

iix

a

b


Assumptions (Luis)

Results are evaluated in terms of the maximum and average number of:

For ELS (average calculated based on the number of links): Labels per link Labels per link with agg (meaning only paths are count) Labels per link with agg and label merging

For PBB-TE Labels per destination Labels per destination with agg (meaning only paths are

count) Labels per destination with VLAN-reut (meaning link

disjoint paths can use same label) Labels per destination with INV-trees (paths that intersect

only on a common segment ending at the destination can also use the same label)B


Tests on smaller networks not normalized


Uniformity in small 3-connected network

S


Connectedness in 28n network

S


Traffic scaling in small network

0

2000

4000

6000

8000

10000

12000

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

Nr of global labels

Nr of link local labels

Nr of node local

Nr of optimized link local

Nr of optimized node local

Global SF gain

Link local merge gain

Node local merging

S

Documents

Vakgroep Informatietechnologie – Onderzoeksgroep IBCN Label scalability of Carrier Ethernet Benchmarking Carrier Ethernet Technologies Workshop Session