OSPF to ISIS

Vijay Gill

Jon Mitchell

jrmitche@aol.net

vijaygill9@aol.com

Notes

"But in our enthusiasm, we could notresist a radical overhaul of thesystem, in which all of its majorweaknesses have been exposed,analyzed, and replaced with newweaknesses."

-Bruce Leverett

Why

• Features– Convergence

• Security

• Simplicitybut to learn ISIS you have to know the secret

handshake and be a *%##%ing 33rd levelmason

-Chance Whaley

Security

• http://www.nanog.org/mtg-0006/katz.html

• Packet bombs

• Wasn’t as big of a deal for AOL– We have packet filters on most line cards

• Most is not ALL

• Runs directly on L2– Harder to spoof or attack

Simplicity

• Found out we didn’t need areas– Added complexity

• Configuration

• Typos

• Slowed it down– DV

– Flat area easy to configure and maintain• Stupid, but no stupider (apologies to Einstein)

State of the Art

POP3

BB1 BB2

L0: x.y.z.n

N.N

.N.m

/31 N.N .N.o/31

OSPF AREA 0

OSPF AREA X

ATDN OSPF

POP1

P6/0

P4/0/0

P6/1

P5/0/0

P2/0P2/0

POP2

P1/0P0/0 P1/0P0/0

P0/0 P1/0 P6/0P0/0 P1/0

L0: x.y.z.m

L0: A.B.C.DBlock: X.Y.A.B/28

P6/2to

bb2-ZZZ

to bb2 -XX X

BB Sample Config

router ospf 1log-adjacency-changesarea 0 authenticationarea x authentication

passive-interface Loopback0network A.B.C.0 0.0.3.255 area 0network A.B.D.0 0.0.1.255 area Xnetwork A.B.C.0 0.0.7.255 area X

maximum-paths 6

area X range A.B.C.x 255.255.255.240area X range A.B.C.y 255.255.255.240

Note: Area X is the BGP cluster-ID ofthe site

POP Sample Config

router ospf 1log-adjacency-changesarea X authenticationredistribute connected subnetspassive-interface Loopback0network A.B.C.0 0.0.1.255 area X….Maximum-paths 6

X. Y.A.B/31

N.N.M.Y/31P6/2

A.B.C

.E/3

1to p

op1-YYY

P4/0

A. B.C

.D/30

L0: A.B.C.EBlock: X.Y.A.C/28

L0: A.B.C.FBlock: X.Y.A.D/28

Strategy

Ships in night– Run parallel– Verify routes

• Raise OSPF admin distance• Verify network after change• Remove OSPF

The plan is in the works, but we have not activated the implementation phase.-Frank Caddeo

Main Backbone Nodes

bb2-mtc

bb2-dtc

bb2-dcl

bb2-nyc

bb1-dtc

bb1-frabb1-nye

bb2-spo

bb1-nyc

bb1-mtc

bb1-dcl

bb1-frr

bb2-frabb2-nye

bb1-spo

bb2-frr

bb1-loh

bb2-loh

bb1-tkn

bb2-sun

America OnlineInternet Operations

12100 Sunrise Valley Drive, Reston, VA 20191

Date:

Revision:

Drawn:

October 17, 2003

6.2

tdo

Architect:

AOL Proprietary and Confidential

Updated: tdo

bb2-sje

bb2-ash

bb1-ash

bb1-den bb2-den

bb1-new

bb2-new

bb1-alb

bb2-alb

bb1-hon

bb2-hon

bb2-seabb1-sea

bb1-kcybb2-kcy bb1-ch1

bb2-chi

bb1-sun

bb2-ntc

bb1-ntc

bb1-sje

bb2-tkn

bb2-las

bb1-las

bb2-phobb1-pho

bb1-col

bb2-col

bb1-hou

bb2-hou

bb1-tbybb2-tby

bb1-atm

bb2-atm

bb2-cha

bb1-cha

bb2-vie

bb1-vie

bb2-rtc

bb1-rtc

2x 48

2x48

2x 48

2x48

bb1-rtl

bb2-rtl

bb1-prs

bb2-prs

bb1-cin bb2-cin

bb1-sjg

bb2-sjg

bb2-dls

bb1-dls

Out of Band

“OOB is the saving throw when you @#$%up”

-RS

• Verified OOB reachability to all POPsbeforehand

IS-IS Migration Prep

Pre-Migration– Load IS-IS configuration built with scripts on RTL

routers• Non Customer PoP

– Develop/test scripts to check IS-IS neighborrelationships and route consistency

Migration Week

• Load IS-IS configuration

• Verify IS-IS neighbor relationships

• Verify LSPs in IS-IS database

• Change OSPF administrative distance to 254

– On some edge routers

Some mornings, it's just not worth chewing through the leather straps.-Emo Phillips

Migration Week (cont)

• Compare IS-IS and OSPF routes on pair of pop routers

• IS-IS vs. OSPF cost check on all interfaces in network

• Change OSPF administrative distance of all remainingrouters to 254

• Verify no OSPF routes in fowarding table

• Basic network reachability

– Ping all routers

– Check connectivity to some external sites• Standard NOC monitoring

Your rules are really beginning to annoy me-Snake Plissken

Post-Migration

• No verification– Verification done as part of migration

• Run a script to remove the OSPF configurationfrom all ATDN routers

We had more than enough genuine headaches as it was, and trivial aestheticconcerns weren't even close to making it onto our agenda.

-Geoff Miller

Current Setup

S1/0/0:0 S1/1/0:0

POP2

BB1 BB2

To bb2-den P7/0

to bb1-chi p6/0

POP1 POP3

P0/0

P0/0

P0/0P0/1 P0/1

P1/0P1/0

P1/0P1/0P7/0 P7/0

P0/0

P8/0P8/0

P5/0/0 P8/0/0

to bb2-dal P6/0

P6/0

CustomerAS: Blah

Low Speed CustomerAS: Blah

P3/0P3/0 P3/1 P3/1

PeerAS: Blah

P0/2

503 503 503 505 503 505

1

1010

10

# IS-ISMETRIC

OC-192

OC-48

OC-12

OC-3

DS1

GSR 12410GSR 12410

GSR 12410 GSR 12410 7513

Config Bits• !• interface Loopback0• isis metric 1 level-2• !• interface POS5/0• description P5/0: bb1-nye-P5-0-pop1-nye-P5-0 (66.p.x.y/31 direct-cabled)(T=pbNYE)• ip router isis• isis metric 503 level-2• isis password ISISPASSWORD(hint, this isn’t the real password) level-2• !• router isis• passive-interface Loopback0• maximum-paths 6• net 39.752f.0100.0014.0000.5000.1668.router.id.inIPv4.00

• is-type level-2-only !Why Level 2?• domain-password this-isn’t-the-real-password-either

• metric-style wide !• external overload signalling ! Ensure that IS-IS will tear down

adjacencies when dCEF is disabled on an interface

• set-overload-bit on-startup wait-for-bgp ! Avoid placingrouter on IGP SPF before bgp

• max-lsp-lifetime 65535• lsp-refresh-interval 65000

• no hello padding ! Hello padding to mtu is deprecated• log-adjacency-changes all• !

Design

• All connected interfaces are redistributed intoBGP

• IS-IS will be preferred

• Redistribution into BGP chosen to reduce thenumber of links in the SPF– Is it an issue in practice

• Not really

cluelessness leads to flapping... flapping leads todampening... dampening leads to suffering

-RS

Cost and RR Design

• Backbone links– Used OSPF metrics– BB-POP Interconnects

• OSPF metric + 500• Avoids Inversion on BB-BB link failure• Mirrors OSPF w/ Areas behavior

– MED oscillation issue• Full mesh of POP routers• No client-to-client reflection• Cost (InterPOP) > cost differences IntraPOP

• New cost out procedure– add 10000 to the interface

Timeline

LoadISIS config

Day

Tim

e

Verify routes

SwitchDistance

ConfirmReach

RemoveOSPF

Loading ISIS Config

• Non Disruptive

• Config was loaded in a three hour window,Monday 6-9 am

• Script (OSPF) -> IS-IS

• Output was copied to each router

• No IS-IS routes in use

If you can't remember, then the claymore is pointed at you

Route Verification

• Compare IS-IS neighbor topology with OSPF– show clns neighbor

– show ip ospf neighbor

• Check IS-IS database on all routers– Ensure all other routers LSP’s installed in IS-IS

database (sh isis database)

Route Verification

• On selected edge routers– Change OSPF admin distance to 254

– Verify traffic to peers

– Compare IS-IS and the OSPF routes• All routes in the network are correctly in IS-IS?

• Go or No Go

Great ideas, in theory, should not be hampered bypesky reality

-Dys

MED

• Changing metric affects MEDs– New metric in the BGP one minute after distance

change– Ratchet down

• Does not propagate for another 10 minutes

– One Large Peer – LP• Listened to MEDs• Not enough capacity to fit all of traffic in one circuit

– All routers connected to LP• Migrated at roughly the same time• Manually cleared soft out after the metric advertisement

updated

The Big One

• Flip Admin Distance– IS-IS routes are preferred

• Current network metrics are consistent with config files?

• Slow Start– Manually change admin distance to 254 on more edge POPS

• Go No-Go?

• Script to flip the rest– From the edge to the center (with respect to ops2)

– In order - LP, europe, asia, brazil, us-pop, us-bb, and dc

• External routes in OSPF now in iBGP

Routing

• Convergence time for the installation– <1 second

• No CEF updates– Costs changed but PATHS didn’t

• All production traffic is routed to Edgerouter loopbacks (n-h-s)

• Rollback• Remove admin distance command• Pre-written script

This thing severely violated the Rule of Complexity as applied to the problem. The Rule ofComplexity states that if an answer seems too complicated to be the right answer, it is the

wrong answer.-Steve Cutchen

Removal of OSPF configuration

• After burn in– 0300 EDT

– OSPF configuration removed• Non-disruptive change

– Old OSPF configs archived via RANCID

We are jolly green giants, walking the earth with routers.-Christopher Morgan (after no router ospf 10 at MFN)

Subject: From the install fileDate: 6/25/2003To: network-ops-list@aol.netCC: John

Network Install Doc for Non-Bounce June 25, 2003General Maintenance (times noted with attribution):

c) Switching ATDN backbone from OSPF to ISIS as the igp. 0300 Expected Impact: None

Dog will hunt/vijay

Line of Truth

Traffic

Questions?

You thinking about smoking off the MPLS hookah?-Brook Bailey

There is a difference between making something foolproof andreducing the number of fools

-Bill Barns