Carrier Strategies for Backbone Traffic Engineering and QoS

Carrier Strategies for Carrier Strategies for Backbone Traffic Engineering Backbone Traffic Engineering

and QoSand QoSDr. Vishal Sharma President & Principal ConsultantMetanoia, Inc.Voice: +1 408 394 6321Email: [email protected] Web: http://www.metanoia-inc.com

Metanoia, Inc.Critical Systems Thinking™

© Copyright 2004All Rights Reserved

Cisco Systems, NERD Lunch, 26 February 2004 2


Copyright 2004, All Rights Reserved

Agenda

Traffic engineering techniques & approaches

Global Crossing

Sprint

Backbone traffic characterization for QoS via capacity management

[Joint work with Thomas Telkamp (Global Crossing), Arman Maghbouleh

(Cariden Technologies), Stephen Gordon (SAIC, former C&W)]




Basic Service Provider Goals

The two fundamental tasks before any service provider:

Deploy a physical topology that meets customers’ needs

Map customer traffic flows on to the physical topology

Earlier (1990s) the mapping task was uncontrolled!

By-product of shortest-path IGP routing

Often handled by over-provisioning




Two Paths to TE in IP Networks With increase in traffic, emergence of ATM, and higher-speed

SONET, two approaches emerged

Use a Layer 2 (ATM) network

Build ATM backbone

Deploy complete PVC mesh, bypass use of IP metrics

TE at ATM layer

With time, evolve ATM to MPLS-based backbone

Use only Layer 3 (IP) network

Build SONET infrastructure

Rely on SONET for resilience

Run IP directly on SONET (POS)

Use metrics (systematically) to control flow of traffic




Global Crossing IP Backbone Network

100,000 route miles 27 countries 250 major cities5 continents200+ POPs

Courtesy: Thomas Telkamp, GBLX




Global Crossing IP Network

OC-48c/STM-16c (2.5Gbps) IP backbone Selected 10Gbps links operational (e.g. Atlantic)

Services offered Internet access & Transit services

IP VPNs -- Layer 3 and Layer 2

MPLS and DiffServ deployed globally

Edge Equipment

Core Equipment

Cisco GSR 12000/12400[12.0(17) SI]

Cisco 7500/7200 ESR, OSRJuniper M10/20/40




Global Crossing: Network Design Philosophy

Ensure there are no bottlenecks in normal state

On handling congestion Prevent via MPLS-TE

Manage via Diffserv

Over-provisioning Well traffic engineered network can handle all traffic

Can withstand failure of even the most critical link(s)

Avoid excessive complexity & features Makes the network unreliable/unstable




Global Crossing’s Approach: Big Picture

WebServer

HR

DR BR

AR

CR

WR

DR

HR BR

AR

CR

WR

DR

HR BR

AR

CR

WR

EthernetSwitch

ModemBank

To other ISPs

To Customers

POP1

POP2

POP3

AR = Access Router

BR = Border Router

CR = Core Router

HR = Hosting Router

WR = WAN Router

DR = DSL Aggregation

OC-3/OC-12

OC-12/OC-48

OC-48/OC-192




TE in the US IP Network: Deployment Strategy

Decision to adopt MPLS for traffic engineering & VPNs

Y2000: 50+ POPs, 300 routers; Y2002: 200+ POPs

Initially, hierarchical MPLS system 2 levels of LSPs

Later, a flat MPLS LSP full mesh only between core routers

Started w/ 9 regions -- 10-50 LSRs/region 100-2500 LSPs/region

Within regions: Routers fully-meshed

Across regions: Core routers fully-meshed

Intra-region traffic ~Mb/s to Gb/s, Inter-region traffic ~ Gb/s

Source [Xiao00]




Design Principles: Statistics Collection

A

B

C

LSP1 = 15 Mb/s

LSP2 = 10 Mb/s

LSP3 = 10 Mb/s

Statistics on individual LSPs can be used to build matrices

A

B

C

25 Mb/s

25 Mb/s

Using packets, we do not know traffic individually to B & C




Design Principles: LSP Control & Management

B

A

D

D

B

OC-48

OC-192

10% in usebefore new req.

New RequestA to D = 2.2 Gb/s

New LSP takeslonger path

Links utilization ismore balancedManually move traffic away from

potential congestion via ERO

B

A

D

D

B

B

A

D

D

B

OC-192

2 LSPs of 1.2Gb/s each

LSPs split acrossalternate routes

Lowered load, greaterheadroom to grow

Load splittingratio = 0.5 each

OC-48

Adding new LSPs with a configured load splitting ratio




Global Crossing’s Current LSP Layout and Traffic Routing

Region 1 Region 2

Region 3

Region 4

POP1POP3

POP4

POP5POP2

Full LSP Meshin Core

Core LSP betweenWRs in POPs 1 & 5

Source

Destination




SprintLinkTM IP Backbone Network

19+ countries

30+ major intl. cities5 continents(reach S. America as well)

400+ POPs

Courtesy: Jeff Chaltas Sprint Public Relations

Represents connectivity only (not to scale)

110,000+ route miles (common with Sprint LD network)




SprintLinkTM IP Network

Tier-1 Internet backbone Customers: corporations, Tier-2 ISPs, univs., ...

Native IP-over-DWDM using SONET framing

4F-BLSR infrastructure (425 SONET rings in network)

Backbone US: OC-48/STM-16 (2.5 Gb/s) links

Europe: OC-192/STM-64 (10 Gb/s) links

DWDM with 8-40 ’s/fiber

Equipment Core: Cisco GSR 12000/12416 (bbone), 10720 metro edge router

Edge: Cisco 75xxx series

Optical: Ciena Sentry 4000, Ciena CoreDirector




SprintLinkTM IP Design Philosophy

Large networks exhibit arch., design & engg. (ADE) non-linearities not seen at smaller scales

Even small things can & do cause huge effects (amplification)

More simultaneous events mean greater likelihood of interaction (coupling)

Simplicity Principle: simple n/wks are easier to operate & scale

Complexity prohibits efficient scaling, driving up CAPEX and OPEX!

Confine intelligence at edges

No state in the network core/backbone

Fastest forwarding of packets in core

Ensure packets encounter minimal queueing




SprintLinkTM Deployment StrategyL2 failure detection triggersswitchover before L3 converges

ZA

Parallel links 50% utilizationunder normal state

1

2

3

4

SONET framing forerror detection

LineCard

LineCard

SONETOverheadIP Data




SprintLinkTM Design Principles

Great value on traffic measurement & monitoring

Use it for

Design, operations, management

Dimensioning, provisioning

SLAs, pricing

Minimizing the extent of complex TE & QoS in the core




Sprint’s Monitoring Methodology

AccessRouter

AccessRouter

AccessRouter

BackboneLinks

Peering LinksProbe

BackboneRouter

Customers Customers Customers

Adapted from [Diot99]

Analysis platform located at Sprint ATL, Burlingame, CA




Sprint Approach to TE

Aim: Thoroughly understand backbone traffic dynamics

Answer questions such as:

Composition of traffic? Origin of traffic?

Between any pair of POPs

What is the traffic demand?

Volume of traffic?

Traffic patterns? (In time? In space?)

How is this demand routed?

How does one design traffic matrics optimally?




Obtaining Traffic Matrices between POPs

A

B

C

D

1.1.1.1

1.1.1/24

SADA

IP Packet DestinationSubnet

POP1POP2

POP3 POP4

DA

1.1.1.1

Exit POP

POP4POP1

POP2

POP3

POP4

ProtocolExitPOP

# pktsBuildTable

City A

City B

City C

City D

City A City B City C City D

City A

City B

City C

City D

City A City B City C City D

TrafficMatrices

ByProtocol

By Timeof Day

Combine data,Obtain matrix




A Peek at a Row of a Traffic Matrix

Summary of Data CollectedAdapted from [Bhattacharya02]

Distribution of aggregate access traffic across other POPs in the Sprint backbone

Peer 1

Peer 2

Web 2

Web 1

ISP

To Backbone

Sprint POP under study




Routing of Demands in the Sprint Backbone

Matrices provide insight into aggregate traffic behavior Do not show the paths demands follow over the backbone

In reality IS-IS link weights hand-crafted by network ops. experts

Weights chosen to restrict traffic b/ween an ingress-egress POP pair to only a few paths through the backbone

Intra-POP link weights heavily influence backbone paths

Result: Despite several alternate paths between POPs Many remain underutilized

Few have v. high utilization




Link Utilization Across the Sprint IP Backbone

Almost 50% of the links have utilization under 15%!

8% of the links are 60% utilized

Observe

Extent of link underutilization

Disparity in utilization levels

Need better load balancing rules

Require a systematic, policy-based approach to do so

Source [Bhattacharya02]




Techniques for Aggregate Load Balancing

Effective load balancing across backbone ...

Knowing how to split traffic over multiple alternate paths

Criteria used depend on purpose

Different service levels use TOS byte or protocol field

Backbone routing use destination address (DA) as basis

Gather inter-POP traffic into streams per DA-based prefixes

E.g. An N-bit prefix produces a pN stream

Assign streams to different paths to balance network load




Observations on Aggregate Streams Examine traffic volume & stability of streams over interval for which

load balancing is to be performed

Findings

Elephants and mice ...

Few very high-vol. streams, many low-vol. streams

Ranking of streams stable over large timescales

Phenomenon is recursive

E.g. p8 elephant sub-divided into p16 streams also has elephants & mice!

Result

Engineering network for elephants alone gives practically all of the benefits of TE! (good for scalability as well)

Elephants

Mice

p8 stream p8 stream

p16stream

p16stream

p16stream

p8 stream Mice

Elephants

Mice

Elephants




Actual Behavior of Streams in the Sprint Backbone

Time of day variation of elephants & mice to a busy egress POP

Elephants

Mice

Decreasing Traffic Volume

Distribution of traffic from p8 streams of POP under study to 3 egress POPs

Less than 10 of the largest streams account for up to 90% of the traffic

Elephants retain a large share of the bandwidth & maintain their ordering

Source [Bhattacharya02]




Agenda

Traffic engineering techniques & approaches

Global Crossing

Sprint

Backbone traffic characterization for QoS via capacity management

[Joint work with Thomas Telkamp (Global Crossing), Arman Maghbouleh (Cariden Technologies), Stephen Gordon (SAIC, former C&W)]




QoS for Backbone IP Networks

QoS – nature of packet delivery service realized in the network

Characterized by achieved: bandwidth, delay, jitter, loss

For backbone networks No link oversubscription achieved b/w ~ desired b/w

Controlled O/P queue size bounded packet delays

Bounded packet delays

Bounded jitter

No packet loss

Backbone QoS Latency characteristics of traffic (Packet delay and jitter)




Relevant Timescales

Long-term: > 5 minutes

Short-term: < 5 minutes

100ms 1sec 1h0 10sec 1min

Aggregate Flows

Intra-Flow

Users/Applications

TCP (RTT) Flow Sizes/Durations Diurnal variation

Timescale

Dynamics

Characteristics




Timescales Critical for QoS

Some of the most stringent QoS requirements for IP traffic arise when carrying voice (e.g. ITU G.114)

Requirements include:

Packet delay (one-way) < 150 ms

End-to-end jitter < 20 ms (for toll-quality voice)

Need resolution at millisecond timescales to understand

Trajectory of individual packets

Queueing behavior in the core

Good performance at ms extends naturally to larger time-scales




Short-term Traffic Characterization

Investigate burstiness within 5-minute intervals

Measure at timescale critical for queueing

E.g., 1 ms, 5 ms, or 10 ms

Analyze statistical properties

Variance, autocorrelation, …

Done one-time at specific locations, as it involves

Complex setup

Voluminous data collection




Data Collection and Measurement

Shomiti Fiber TapTap

Analyzer

GbE backbone link

Measurement PC

12 traces, 30 seconds each Collected over a month

Different times and days

Mean b/w 126 – 290 Mbps (<< 1 Gbps)

No queueing/shaping on O/P interface

Trace utilizations uniformly < 1Gbps over any 1 ms interval




Raw Results30 sec of data, 1ms scale

Mean = 950 Mbps

Max. = 2033 Mbps

Min. = 509 Mbps

95-percentile: 1183 Mbps

5-percentile: 737 Mbps

~250 packets per interval

Mean rate over 30 sec

Output queue rate (available link bandwidth)




Traffic Distribution Histogram (1ms scale)

Fits normal probability distribution well (Std. dev. = 138 Mbps)

No heavy-tails

Suggests small over-provisioning factor




Autocorrelation Lag Plot (1ms scale)

Scatter plot for consecutive samples of time-series

Are periods of high usage followed by other periods of high usage?

Autocorrelation at 1msis 0.13 (=uncorrelated)

High bandwidth bursts do not line up to cause marked queueing

High autocorrelation Points concentrated along 45° lineClearly not the case here

45°




Poisson versus Self-Similar Traffic

Scale Invariant!

Markovian Process Self-Similar Process

Refs. [Liljenstolpe01],[Lothberg01]

Ref. [Tekinay99]




Internet Traffic: Variance versus Timescale

Random variable X

Var(X(m)) = σ2 m-1

Self-similar process, with Hurst parameter H

Var(X(m)) = σ2m2H-2

Long range dependence (LRD)

0.5 < H < 1

Var(X(m)) converges to zero

slower than a rate m-1

150 msNote: m = sample size, σ2 = Var(X)

Variance decreases in proportion to timescale

Variance decreases slower self-similarity

Slope = -1 Poisson




Traffic: Summary

Long-term well behaved traffic

Short-term uncorrelated traffic




IP Capacity Allocation

Measurement data

5-min average utilization

Performance goals, e.g.

Packet loss < 1%

Jitter < 10 ms

End-to-end delay < 20 ms

But … we have no “Erlang formulas” for IP traffic …

Model traffic, fit parameters, evaluate parametric solution

Two approaches to a solution

Empirically derive guidelines by characterizing observed traffic

Approach in this work




Queuing Simulation: Methodology

FIFO Queue

Sampled Traffic

Fixed Service Rate

Monitor Queuing Delay

Sampled Traffic

Sampled Traffic

Feed multiplexed, sampled traffic into a FIFO queue

Measure amount of traffic that violates set delay bound

622 Mbps572 Mbps

126 Mbps

240 Mbps

206 Mbps

Example: 92% Utilization

Output Link under study




Queuing Simulation: Results89% 93%

+ Simulation 622 Mbps + Simulation 1000 Mbps---- M/M/1 622 Mbps---- M/M/1 1000 Mbps




Multi-hop Queueing: 8 hops

P99.9 delay: Hop 1 = 2 ms, Hop 8 = 5.2 ms (increase not linear!)

P99.9 = 2ms

P99.9 = 5.2ms




Queueing: Summary

Queueing simulation

Backbone link (GbE)

Over-provisioning ~7.5% to bound delay/hop to under 2 ms

Higher speeds (2.5G/10G)

Over-provisioning factor becomes very small

Lower speeds (< 0.622G)

Over-provisioning factor is significant

P99.9 multi-hop delay/jitter is not additive




Applications to Network Planning

QoS targets “Headroom” (over-provisioning %) Derived experimentally by characterizing short-term traffic

Traffic matrix Derivable from the stable, well-behaved, long-term traffic

Determine minimum capacity deployment required to meet objectives under normal and failure conditions

How to use this for planning?

Trending – study impact of growth over time

Failure analysis – impact of failures on loading Derived experimentally by characterizing short-term traffic

Optimization – LSP routing, IGP metrics




Acknowledgements

Thomas Telkamp, Global Crossing

Robert J. Rockell, Jeff Chaltas, Ananth Nagarajan, Sprint

Steve Gordon, SAIC (former C&W)

Jennifer Rexford, Albert Greenberg, Carsten Lund, AT&T Research

Wai-Sum Lai, AT&T

Fang Wu, NTT America

Arman Maghbouleh, Alan Gous, Cariden Technologies

Yufei Wang, VPI Systems

Susan Cole, OPNET Technologies




References

[Bhattacharya02] S. Bhattacharya, et al, “POP-Level and Access-Link Level Traffic Dynamics in a Tier-1 POP,” Proc. ACM SIGCOMM Internet Measurement Workshop, November 2001.

[Diot99] C. Diot, “Tier-1 IP Backbone Network: Architecture and

Performance,”Sprint Advanced Technology Labs., 1999. Available at: http://www.sprintlabs.com/Department/IP-Interworking/Monitor/

[Liljenstolpe01] Chris Liljenstolpe, Design Issues in Next Generation Carrier Networks, Proc. MPLS 2001, Washington, D.C., 7-9 October, 2001.

[Lothberg01] Peter Lothberg, A View of the Future: The IP-Only Internet, NANOG 22, Scottsdale, AZ, 20-22 May 2001, http://www.nanog.org/mtg-0105/lothberg.html




References

[Morris00] Robert Morris and Dong Lin, Variance of Aggregated WebTraffic, IEEE Infocom’00, Tel Aviv, Israel, March 2000, pp. 360-366.

[Tekinay99] Zafer Sahinoglu and Sirin Tekinay, On Multimedia Networks: Self-Similar Traffic and Network Performance, IEEE Commun. Mag., vol. 37, no. 1, January 1999, pp. 48-53.

[Xiao00] X. Xiao et al, “Traffic Engineering with MPLS in the Internet,” IEEE Network, March/April 2000, vol. 14, no. 2, pp. 28-33.

Business

Carrier Strategies for Backbone Traffic Engineering and QoS