Implementing and Maintaining an ISP Backbone

Kevin Butler

SprintNetwork

Click here fora closer lookat the Spr int

network on theEast Coast

Click here fora closer lookat the Spr intnetwork inNor thernCalifornia

Pearl Ci ty in Hawaii isa future network location

Click here for a closer look at theSprint network in Washington s tate

Legend DS3OC3OC12OC48

Seattle

Atlanta

Chicago

Roachdale

Stockton

San Jose

Anaheim

Fort Worth

Orlando

Kansas City

CheyenneNew York

PennsaukenRelayWash. DC

Tacoma

Tier 1 ISP Backbones

• Comprise some of the world’s largest IP networks

• Tier 1 companies include Sprint, AT&T, PSINet

• UUNET has the world’s largest IP data network, presence on four continents and future expansion into Latin and South America

Service Level Agreements

• SLAs are an important and prestigious tool in attracting and maintaining customers

• Comprised of uptime guarantees and bounds on latency through various geographic regions

• most ISPs currently have latency < 50ms across the US

Supporting the Customer

• Quality and expertise of first-line customer support varies wildly between companies

• depending on size, geographic location and company focus, some front-line support teams outsourced to third parties

• some in-house high level support teams have skills equivalent or superior to NOCs

Network Operations Centres

• Generally the teams concerned with backbone maintenance and support

• trend towards consolidation into “Super-NOCs” (eg. one for Americas, one for Europe)

• specialisation within NOC for product support (eg. dial, VPN, backbone NOCs)

NOC Tools

• NOCOL - Network Operations Centre On Line (freeware UNIX)

• Mediahouse monitoring (mainly web)

• Micromuse Netcool (now owned by Lucent) - used by MCI WorldCom, PSINet, BT

Dial Access

• Dial is a major selling point, especially with customers who travel a lot or are their own ISPs

• connections made through an Ascend MAX TNT, which can support up to 720 concurrent callers

• back-end is a DS-3 into a backbone router, routers advertised by an IGP (eg. RIP)

Dial-Related Technologies

• COBRA (Central Office Based Remote Access) allow building of virtual POPs by backhauling PRIs

• RADIUS (Remote Authentication Dial In User Service)

Integrated Services Digital Network

• ISDN customers authenticate by RADIUS similar to dial users

• underlying architecture similar but dial equipment often administrated differently

• ISDN maintained within same AS as backbone whereas dial often in its own AS

DS-1 and high-speed access

• Customer connections usually multiplexed, come into DSU as a channelised DS-3

• gateway routers on ISP side usually Cisco 7500 series, increasingly using Cisco 12000

• customers connect using Cisco 1604, 2621, some 3600 series, very large customers use 7500 series routers

Gateway Routers

• obtain routes from customers usually statically, but sometimes by BGP

• usually run link-state IGP within AS (eg. OSPF, IS-IS)

• Cisco 7513 backplanes 1.8 Gbps while 12008 does 40 Gbps

Where does traffic go from here?

• Most ISPs have two levels of networks above the access router

• Metropolitan networks aggregate gateway traffic, generally city-wide (if multiple POPs in city)

• transit networks aggregate metro network’s traffic, responsible for inter-city transport

ATM Switches

• Terminate long-haul OC-12, OC-48 circuits and metro rings

• Choice of vendor contingent on ISP, commonly Newbridge, Fore Systems (ASX-1000 and ASX-4000)

Example of an ATM interface

TR1.EG1:

interface ATM2/0description To HA13.BLAH1 3C1atm vc-per-vp 512atm pvc 16 0 16 ilmi!interface ATM2/0.195 point-to-pointdescription To XR1.BLAH1 ATM6/0ip address 146.188.200.98 255.255.255.252ip router isis Net-Backboneatm pvc 195 0 195 aal5snapclns router isis Net-Backbone

Implementation of BGP

• BGP run between autonomous systems and peers, as well as multi-homed customers

• monolithic AS broken up into BGP confederations for ease of work

• routes controlled using access lists and route maps

BGP

• Communities are destinations that share common attributes (eg. through access-list filters)

BGP table version is 23718690, local router ID is 205.150.242.2Status codes: s suppressed, d damped, h history, * valid, > best, i - internalOrigin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path*>i24.64.0.0/19 198.133.49.7 100 0 6327 6172 i*>i24.64.0.0/14 198.133.49.7 100 0 6327 i*>i24.64.32.0/19 198.133.49.7 100 0 6327 6172 i*>i24.64.64.0/19 198.133.49.7 100 0 6327 6172 i*>i24.64.96.0/19 198.133.49.7 100 0 6327 6172 i*>i24.64.192.0/19 198.133.49.7 100 0 6327 6172 i*>i24.64.224.0/19 198.133.49.7 100 0 6327 6172 i*>i24.65.0.0/19 198.133.49.7 100 0 6327 6172 i*>i24.65.96.0/19 198.133.49.7 100 0 6327 6172 i*>i24.65.128.0/19 198.133.49.7 100 0 6327 6172 i

Advantages of BGP for User

• Allows for load-sharing and redundancy

• routes can be biased through AS path prepending

• requirement is high-quality router with close to 100% uptime to avoid connection flaps and subsequent route dampening

Common Customer Issues

• Static routes on backbone - often difficult to spot, can cause very strange routing results

• pull-up routes for netblocks smaller than /24, required to avoid BGP dampening

• BGP recalculations - if done on a transit router, entire backbone segments can experience outages

Customer Requirements of the Backbone

• Redundancy - networks are redundant but card failures can take down whole routers

• physical connection to POP from customer is SPF

• low latency - massive increases in demand on backbone makes this difficult

• over $2 million a day spent on global backbone upgrades

DSL: low cost, high speed

• DSL might phase out ISDN connections

• difficult to troubleshoot from network standpoint

• connections pass through telco’s frame or ATM cloud between DSLAM and VR

• RedBack SMS (Subscriber Management System) 1000 commonly used as VR

RedBack SMS 1000

• Supports up to 4000 sessions

• OC-3 out to metro network

• traffic-shaping accomplished with profiles

atm profile samplecust

counters

shaping vbr-nrt pcr 1000 cdvt 100 scr 100 bt 10

Increasing Capacity

• Backbone capacity increasing at a huge rate

• Traffic engineering combined with high backplane becoming increasingly important

• many ISPs turning to Juniper routers

• UUNET rolled out production OC-192c with Juniper M160 running MPLS

Juniper Routers

• JUNOS supports MPLS and RSVP

isis { interface all;}ospf { area 0.0.0.0 { interface so-0/0/0 { metric 15; retransmit-interval 10; hello-interval 5; } }}

[edit]

Distributed DOS attacks

• Can be very detrimental to backbone (even causing switch crashes)

• Combated by rate-limiting ICMP on routers

• Most effective defense is community-wide egress filtering; requires co-operation throughout the Internet

Canadian Network Challenges

• Geographically, population resides in virtually a straight line across the south

• major focus is on southbound capacity to the US

• CRTC regulations on telcos create different arrangements

• heterogeneous network to the US, integration a big issue

Questions?

• Anything I can clarify or expand on...

• Thank you!