Upload
hadung
View
285
Download
0
Embed Size (px)
Citation preview
Session ID-BRKRST-3320
Troubleshooting BGP
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 2
Agenda
• Housekeeping Items
• Knowing Your Crime Scene and Your Suspects
• Troubleshooting References
• Troubleshooting Peers
• Best Path
• BGP Convergence
• High Utilization
• BGP Routing Problems
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 3
Housekeeping
Cell Phones
Who are you?
“Traditional” Service Provider
Enterprise/State/School Offering Service Provider like services
Enterprise with a BGP Core
Enterprise Doing BGP Cause your SP told you to
CCIE
“Advanced” Class
Assume BGP Operational Experience
Basic config
Show commands
Understand BGP attributes
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 4
Knowing Your Crime Scene and Your Suspects
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 5
Knowing Your Crime Scene and Your Suspects
The Usual Suspects Configs
Layer 1
IGP
Memory
CPU
MTU
Design
Input Queues
Bug
………. BGP Table
BGP Best Path
RIB
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 6
AS 1BGPTCP
IP
AS 2
Layer 2 Cloud
eBGP
R2R1
BGPTCP
IP
Knowing Your Crime Scene and Your Suspects
Your Usual Suspects Not true point-to-point
Down detection
Presentation doesn’t match provisioning
………
+ the “usual” suspects
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 7
AS 65500
Intranet/IGP
iBGP
R2R1
Knowing Your Crime Scene and Your Suspects
Your Usual Suspects Network stability
IGP churn
........
+ the “usual” suspects
BGPTCP
IP
BGPTCP
IP
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 8
Troubleshooting References
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 9
References
• BGP Support Pagehttp://www.cisco.com/en/US/customer/tech/tk365/tk80/tsd_technology_support_sub-protocol_home.html
(much more )
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 10
Troubleshooting BGP
http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a008009478a.shtml#main
BGP neighbor is flapping or unable toEstablish BGP peering
BGP Routes not in the Routing Table
BGP Route Advertisement
Multihoming/Loadsharing
Malloc Failures & BGP Memory Consumption
BGP and Firewalls
Troubleshooting Flapping BGP Routes
Issues with BGP Conditional Advertisement
References
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 11
ReferencesTroubleshooting BGP Neighbor Establishment
http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a008009478a.shtml#bgp_trouble_neighbor
AS Number
IP Address
multihop
update-source
Extended Ping: peering address to
peering address
Access-lists
Extended Ping: With Max size Packet
(MTU)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 12
ReferencesTroubleshooting Routes Missing from the Routing Table
http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a008009478a.shtml#bgp_trouble_route_missing
Synchronization
Valid Next Hop?
Inbound filters
Does the path exist in the BGP Table?
Inbound filters
BGP NeighborNot Advertising
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 13
ReferencesTroubleshooting BGP Route Advertisement
Network command
Redistributing
Route-map or filter list
Aggregate-address
Classful network
Auto-summary Access-list or route-maps
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 14
ReferencesTroubleshooting Multihoming Inbound
http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a008009478a.shtml#bgp_trouble_multi_in
Different ISPs
Single ISP andsingle router
Single ISP andmultiple routers
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 15
Troubleshooting Peers
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 16
router bgp {R1_AS#}neighbor {R2_BGP_Peer_Add} remote-as {R2_AS#}
R2router bgp {R2_AS#}neighbor {R1_BGP_Peer_Add} remote-as {R1_AS#}
R1
R1_BGP_Peer_Add
R2_BGP_Peer_Add
router bgp {R1_AS#}neighbor {R2_BGP_Peer_Add} remote-as {R2_AS#}neighbor {R2_BGP_Peer_Add} update-source {R1_BGP_Peer_Add}
R2router bgp {R2_AS#}neighbor {R1_BGP_Peer_Add} remote-as {R1_AS#}neighbor {R1_BGP_Peer_Add} update-source {R2_BGP_Peer_Add}
R1
Configs
BGP Speakers Won’t Peer
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 17
interface Loop0ip address 1.1.1.1/32
router bgp 100neighbor 2.2.2.2 remote-as {R2_AS#}neighbor 2.2.2.2 update-source Loop0
R2
interface Loop0ip address 2.2.2.2/32
router bgp 100neighbor 1.1.1.1 remote-as {R1_AS#}neighbor 1.1.1.1 update-source Loop0
R1
Extended ping - between peering addresses
R1 BGP session to 2.2.2.2 not successful
BGP Debugs on R1 showBGP: 2.2.2.2 open active, local address 1.1.1.1BGP: 2.2.2.2 open failed: Connection timed out; remote host not responding
Extended Ping Between Configured BGP Peering AddressesR1#ping Protocol [ip]: Target IP address: 2.2.2.2Repeat count [5]: Datagram size [100]: Timeout in seconds [2]: Extended commands [n]: ySource address or interface: 1.1.1.1..Sending 5, 100-byte ICMP Echos to 2.2.2.2.....Success rate is 0 percent (0/5)R1#
BGP Speakers Won’t Peer
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 18
BGP Speakers Won’t Peer
• Remember that BGP runs on top of IP, and can be affected by:
IP reachability problems (the underlying routing isn’t working)Access ListsTCP problemsMTU Issues – extended ping and sweep address rangesRate limitingTraffic shapingTunneling problemsEtc.
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 19
BGP Speakers Won’t PeerUseful Peer Troubleshooting Commands
show tcp brief all TCB Local Address Foreign Address (state)
64316F14 1.1.1.1.12345 2.2.2.2.179 ESTAB
6431BA8C *.179 2.2.2.2.* LISTEN
62FFDEF4 *.* *.* LISTEN
show tcp statistics Rcvd: 7005 Total, 10 no port
0 checksum error, 0 bad offset, 0 too short
....
0 out-of-order packets (0 bytes)
....
4186 ack packets (73521 bytes)
....
Sent: 9150 Total, 0 urgent packets
4810 control packets (including 127 retransmitted)
2172 data packets (71504 bytes)
....
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 20
BGP Speakers Won’t PeerUseful Peer Troubleshooting Commands
debug ip tcp transactions R1#sh log | i TCP0:
TCP0: state was ESTAB -> FINWAIT1 [12345 -> 2.2.2.2(179)]
TCP0: sending FIN
TCP0: state was FINWAIT1 -> FINWAIT2 [12345 -> 2.2.2.2(179)]
TCP0: FIN processed
TCP0: state was FINWAIT2 -> TIMEWAIT [12345 -> 2.2.2.2(179)]
TCP0: Connection to 2.2.2.2:179, advertising MSS 1460
TCP0: state was CLOSED -> SYNSENT [12346 -> 2.2.2.2(179)]
TCP0: state was SYNSENT -> ESTAB [12346 -> 2.2.2.2(179)]
TCP0: tcb 6430DCDC connection to 2.2.2.2:179, received MSS 1460, MSS is 1460
This can be very chatty, so be careful with this debug!
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 21
BGP Speakers Won’t Peer
• If the connectivity is good, the next step is to check BGP itself
• debug ip bgpUse with cautionConfigure so the output goes to the log, rather than the console
logging buffered <size>
no logging console
It’s easier to find the problem points this wayrouter#show log | i NOTIFICATION
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 22
BGP Speakers Won’t Peer
• show ip bgp neighbor 1.1.1.1 | include last reset
This should give you the resets for a peer
The same information as is shown through debug ip bgp
• bgp log-neighbor changesProvides much of the same information as debug ip bgp, as well
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 23
R2
R1
View from R2 Only - Wrong AS
10.1.2.1
10.1.2.2
R2# show log | include NOTI%BGP-3-NOTIFICATION: sent to neighbor 10.1.2.1 2/2 (peer in wrong AS) 2 bytes 0064 FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0104 0064 00B4 0101 0101 1002 0601 0400 0100 0102 0280 0002 0202 00
Sniff of BGP Notification Sent from R1 to R2
router bgp 200no synchronizationbgp log-neighbor-changesneighbor 10.1.2.1 remote-as 10no auto-summary
X ‘0064’ = decimal 100
BGP Speakers Won’t Peer
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 24
R2
R1
View from R2 Only - Wrong AS
10.1.2.1
10.1.2.2
R2# debug ip bgpBGP: 10.1.2.1 bad OPEN, remote AS is 100, expected 10BGP: 10.1.2.1 went from OpenSent to Closing%BGP-3-NOTIFICATION: sent to neighbor 10.1.2.1 2/2 (peer in wrong AS) 2 bytes 0064 FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0104 0064 00B4 0101 0101 1002 0601 0400 0100 0102 0280 0002 0202 00BGP: 10.1.2.1 send message type 3, length (incl. header) 23BGP: 10.1.2.1 local error close after sending NOTIFICATION
router bgp 200no synchronizationbgp log-neighbor-changesneighbor 10.1.2.1 remote-as 10no auto-summary
router bgp 100no synchronizationbgp log-neighbor-changesneighbor 10.1.2.2 remote-as 200no auto-summary
BGP Speakers Won’t Peer
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 25
R2
R1
View from R2 Only - Wrong AS
10.1.2.1
10.1.2.2 router bgp 200no synchronizationbgp log-neighbor-changesneighbor 10.1.2.1 remote-as 10no auto-summary
Question: What did R1 see
R1#sh log | include NOTI%BGP-3-NOTIFICATION: received from neighbor 10.1.2.2 2/2 (peer in wrong AS) 2 bytes 0064
router bgp 100no synchronizationbgp log-neighbor-changesneighbor 10.1.2.2 remote-as 200no auto-summary
BGP Speakers Won’t Peer
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 26
BGP Speakers Won’t Peer
BGP uses a TTL of 1 for eBGP peers
For eBGP peers that are more than 1 hop away a larger TTL must be used
neighbor x.x.x.x ebgp-multihop[2-255] R1
AS65000
R2AS65001
Default TTL
Configured TTL
eBGP Peers and Time to Live
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 27
BGP Speakers Won’t PeerDirectly Connected eBGP Peers
router bgp 100no synchronizationbgp log-neighbor-changesneighbor 2.2.2.2 remote-as 200neighbor 2.2.2.2 disable-connected-checkneighbor 2.2.2.2 update-source Loopback0no auto-summary
router bgp 100no synchronizationbgp log-neighbor-changesneighbor 2.2.2.2 remote-as 200neighbor 2.2.2.2 ebgp-multihop 2neighbor 2.2.2.2 update-source Loopback0no auto-summary
R2
R1
10.1.2.1
10.1.2.2
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 28
BGP Speakers Won’t PeerDirectly Connected eBGP Peers
router bgp 100no synchronizationbgp log-neighbor-changesneighbor 2.2.2.2 remote-as 200neighbor 2.2.2.2 disable-connected-checkneighbor 2.2.2.2 update-source Loopback0no auto-summary
R2
R1
10.1.2.1
10.1.2.2
“neighbor disable-connected-check
To disable connection verification to establish an eBGP peering session with a single-hop peer that uses a loopback interface, use the neighbor disable-connected-check command in Address Family or Router configuration mode. To enable connection verification for eBGP peering sessions, use the no form of this command.”
http://www.cisco.com/en/US/docs/ios/12_3t/ip_route/command/reference/ip2_n1gt.html#wp1109875
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 29
BGP Speakers Won’t Peer
unknown subcode The peer open notification subcode isn’t known
incompatible BGP version The version of BGP the peer is running isn’t compatible with the local version of BGP
peer in wrong AS The AS this peer is locally configured for doesn’t match the AS the peer is advertising
BGP identifier wrong The BGP router ID is the same as the local BGP router ID
unsupported optional parameter
There is an option in the packet which the local BGP speaker doesn’t recognize
authentication failure The MD5 hash on the received packet does not match the correct MD5 hash
unacceptable hold time The remove BGP peer has requested a BGP hold time which is not allowed (too low)
unsupported/disjoint capability The peer has asked for support for a feature which the local router does not support
%BGP-3-NOTIFICATION: sent to neighbor 2.2.2.2 2/2 (peer in wrong AS) 2 bytes 00C8 FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0104 00C8 00B4 0202 0202 1002 0601 0400 0100 0102 0280 0002 0202 00
Bad Messages
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 30
BGP Speaker Flap
• Here we see a message from bgp log-neighbor-changes telling us the hold timer expired
• We can double check this by looking at show ip bgp neighbor x.x.x.x | include last reset
R1
R2
%BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down BGP Notification sent%BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 4/0 (hold time expired)
R2#show ip bgp neighbor 1.1.1.1 | include last resetLast reset 00:01:02, due to BGP Notification sent,hold time expired
Case Study
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 31
BGP Speaker Flap
There are lots of possibilities here
R1 has a problem sending keepalives?
The keepalives are lost in the cloud?
R2 has a problem receiving the keepalive?
R1
R2
Case Study
%BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down BGP Notification sent%BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 4/0 (hold time expired)
R2#show ip bgp neighbor 1.1.1.1 | include last resetLast reset 00:01:02, due to BGP Notification sent,hold time expired
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 32
BGP Speaker Flap
• Did R1 build and transmit a keepalive for R2?debug ip bgp keepaliveshow ip bgp neighbor
• When did we last send or receive data with the peer?
R2#show ip bgp neighbors 1.1.1.1BGP neighbor is 1.1.1.1, remote AS 100, external linkBGP version 4, remote router ID 1.1.1.1BGP state = Established, up for 00:12:49Last read 00:00:45, last write 00:00:44, hold time is 180,
keepalive interval is 60 seconds
• If R1 did not build and transmit a KAHow is R1 on memory?What is the R1’s CPU load?Is R2’s TCP window open?
Case Study
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 33
BGP Speaker Flap
R1#show ip bgp sum | begin NeighborNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd2.2.2.2 4 2 53 284 10167 0 97 00:02:15 0
Case Study
R1#show ip bgp summary | begin NeighborNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd2.2.2.2 4 2 53 284 10167 0 98 00:03:04 0
At least one BGP keepalive interval apart
The number of packets generated is increasing
But the number of packets transmitted is not increasing
The keepalives aren’t leaving R1!
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 34
BGP Speaker Flap
• Go back to square one and check the IP connectivityThis is a layer 2 or 3 transport issue, etc.
R1#ping 2.2.2.2Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 16/21/24 m
R2#ping 2.2.2.2 size 1500 df-bit
Type escape sequence to abort.Sending 5, 1500-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:Packet sent with the DF bit set…..Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 msR2#
Case Study
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 35
Best Path
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 36
Best Path
http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094431.shtml#bestpath
Algorithm 1 Weight Highest wins Scope is router only
2 Local Preference Highest wins Scope is AS only
3 Locally Originated
4 AS_Path Shortest wins Skipped if bgp bestpath as-path ignore configured
5 Origin Type Lowest IGP, EGP, Incomplete
6 MED Lowest MEDs are compared only if the first AS in the AS_SEQUENCE is the same for multiple paths.
7 eBGP over iBGP If bestpath is selected, go to #9 (multipath)
8 Metric to Next Hop Lowest Continue, even if bestpath is already selected
9 Multiple Paths in RIB
Continue, if bestpath not yet selected
10 Oldest Wins Unless BGP best path compare router-id configured
11 BGP Router ID Lowest
12 Cluster List Smallest Only present in BGP RR
13 Neighbor Address Lowest
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 37
Best Path
Code
Real Term Who Wins
Numerous N Next-hop
1 Wise W Weight Highest
2 LiP LP Local Preference
Highest
3 LOvers LO Locally Originated
4 APply AP AS PATH Shortest
5 ORal OR Origin
6 MEDication MED MED Lowest (see note 1 below)
Note 1: The default behavior for MED comparison is only done if the first AS in the AS_SEQUENCE is the same for multiple paths.
Policy Tools
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 38
BGP Convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 39
BGP Slow Convergence
• Hey—Who are you calling slow?Slow is a relative term....
BGP probably won’t ever converge as fast as any of the IGPs
• Two general convergence situationsInitial startup between peers
Route changes between existing peers
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 40
BGP Slow Convergence
• Initial convergence is limited byThe number of packets required to transfer the entire BGP database
The number of routes
The ability of BGP to pack routes into a small number of packets
The number of peer specific policies
TCP transport issues
How often does TCP go into slow start?
How much can TCP put into one packet?
Initial Convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 41
BGP Slow Convergence
BGP starts a packet by building an attribute set
It then packs as many destinations (NLRIs) as it can into the packet
Only destinations with the same attribute set can be placed in the packetDestinations can only be put into the packet until it’s full
Initial Convergence
Attr
ibut
eN
LRI
Attr
ibut
eN
LRI
Attr
ibut
eN
LRI
NLR
I
Less
Effi
cien
t
Mor
e E
ffici
ent
First rule of thumb: to increase convergence speed, decrease unique sets of attributes
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 42
BGP Slow Convergence
The larger the packet BGP can build, the more destinations it can put in the packet
The more you can put in a single packet, the less often you have to repeat the same attributes
Initial Convergence
Attr
ibut
eN
LRI
Attr
ibut
eN
LRI
Attr
ibut
eN
LRI
NLR
I
Less
Effi
cien
t
Mor
e E
ffici
ent
Attr
ibut
eN
LRI
NLR
IA
ttrib
ute
NLR
IN
LRI
Attr
ibut
eN
LRI
NLR
IN
LRI
NLR
I
Less
Effi
cien
t
Mor
e E
ffici
ent
Second rule of thumb: allow BGP to use the largest packets possible
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 43
BGP Slow Convergence
• BGP must create packets based the policies towards each peer
Initial Convergence
Attr
ibut
eN
LRI
NLR
I
Less
Effi
cien
t
Attr
ibut
eN
LRI
NLR
I
Attr
ibut
eN
LRI
NLR
IM
ore
Effi
cien
t
Third rule of thumb: Minimize the number of unique policies towards eBGP peers
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 44
BGP Slow Convergence
• TCP Interactions
Each time a TCP packet is dropped, the session goes into slow start
It takes a good deal of time for a TCP session to come out of slow start
Initial Convergence
Fourth rule of Thumb: Try and reduce the circumstances under which a TCP segment will be dropped during initial convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 45
BGP Slow Convergence
• Bottom Line:Hold down the number of unique attributes per route
Don’t send communities if you don’t need to, etc
Hold down the number of policies towards eBGP peersTry to find a small set of common policies, rather than individualizing policies per peer
Stop TCP segment dropsIncrease input queues
Increase SPD thresholds
Make certain links are clean
Initial Convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 46
BGP Slow Convergence
There are two elements to route change convergence for BGP
How long does it take to see the failure?How long does it take to propagate information about the failure?
For faster peer down detection, there are several tools you can use
Fast layer two down detectionFast external fallover for directly connected eBGP peers
Faster keepalive and dead interval timers
Down to 3 and 9 are commonly used today
Route Change Convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 47
BGP Slow Convergence
Fast Session Deactivation• The address of each peer is registered with
the Address Tracking Filter (ATF) system
• When the state of the route changes, ATF notifies BGP
• BGP tears down the peer impacted
• BGP does not wait on the hold timer to expire
Route Change Convergence
BGP/RIB ATF Interface
eBGP Multihop SessionLink fails; IGP converges
ATF Notifies BGP
BGP Tears Down eBGP Session
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 48
BGP Slow Convergence
• Very dangerous for iBGP peersIGP may not have a route to a peer for a split second
FSD would tear down the BGP session
Imagine if you lose your IGP route to your RR (Route Reflector) for just 100ms
• Off by defaultneighbor x.x.x.x fall-over
Route Change Convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 49
BGP Slow Convergence
• ATF can also be used to track changes in next hopsiBGP recurses onto an IGP next hop to find a path through the local AS
Changes in the IGP cost or reachability are normally seen only by the BGP scanner
Since the scanner runs every 60 seconds, by default, this means iBGP convergence can take up to 60 seconds on an IGP change....
Route Change Convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 50
• BGP Next Hop TrackingEnabled by default
[no] bgp nexthop trigger enable
• BGP registers all nexthops with ATFHidden command will let you see a list of nexthops
show ip bgp attr nexthop
• ATF will let BGP know when a route change occurs for a nexthop
• ATF notification will trigger a lightweight “BGP Scanner” runBestpaths will be calculated
None of the other “Full Scan” work will happen
BGP Slow ConvergenceRoute Change Convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 51
• Once an ATF notification is received BGP waits 5 seconds before triggering NHT scan
bgp nexthop trigger delay <0-100>
May lower default value as we gain experience
• Event driven model allows BGP to react quickly to IGP changes
No longer need to wait as long as 60 seconds for BGP to scan the table and recalculate bestpaths
Tuning your IGP for fast convergence is recommended
BGP Slow ConvergenceRoute Change Convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 52
• Dampening is used to reduce frequency of triggered scans
• show ip bgp internalDisplays data on when the last NHT scan occurred
Time until the next NHT may occur (dampening information)
• New commandsbgp nexthop trigger enable
bgp nexthop trigger delay <0-100>
show ip bgp attr next-hop ribfilter
debug ip bgp events nexthop
debug ip bgp rib-filter
• Full BGP scan still happens every 60 secondsFull scanner will no longer recalculate bestpaths if NHT is enabled
BGP Slow ConvergenceRoute Change Convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 53
• How is the timer enforced for peer X?Timer starts when all routes have been advertised to X
For the next MRAI (seconds) we will not propagate any bestpath changes to peer X
Once X’s MRAI timer expires, send him updates and withdraws
Restart the timer and the process repeats…
• User may see a wave of updates and withdraws to peer X every MRAI
• User will NOT see a delay of MRAI between each individual update and/or withdraw
BGP would probably never converge if this was the case
BGP Slow ConvergenceRoute Change Convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 54
• MRAI timeline for iBGP peer
• Bestpath Change #1 at t7 is TXed immediately
• MRAI timer starts at t7, will expire at t12
• Bestpath Change #2 at t10 must wait until t12 for MRAI to expire
• Bestpath Change #2 is TXed at t12
• MRAI timer starts at t12, will expire at t17
• MRAI expires at t17…no updates are pending
t0 t5 t10 t15 t20 t25
•TX update #1•Start MRAI
BestpathChange #2
BestpathChange #1
•MRAI Expires•TX update #2•Start MRAI
•MRAI Expires
BGP Slow ConvergenceRoute Change Convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 55
• BGP is not a link state protocol
• May take several “rounds/cycles” of exchanging updates and withdraws for the network to converge
• MRAI must expire between each round!
• The more fully meshed the network and the more tiers of ASes, the more rounds required for convergence
• Think aboutHow many tiers of ASes there are in the Internet
How meshy peering can be in the Internet
BGP Slow ConvergenceRoute Change Convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 56
• Internet churn means we are constantly setting and waiting on MRAI timers
One flapping prefix slows convergence for all prefixes
Internet table sees roughly 6 bestpath changes per second
• For iBGP and PE-CE eBGP peersneighbor x.x.x.x advertisement-interval 0
Will be the default in 12.0(32)S
• For regular eBGP peersLowering to 0 may get you dampened
OK to lower for eBGP peers if they are not using dampening
BGP Slow ConvergenceRoute Change Convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 58
High Utilization
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 59
High Utilization
• High Processor Utilization
• Next Hop Tracking
• High Memory Utilization
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 60
High Processor Utilization
• Why?This could be for several reasons
High route churn is the most likely
Router#show process cpuCPU utilization for five seconds: 100%/0%; one minute: 99%; five minutes:
81%....139 6795740 1020252 6660 88.34% 91.63% 74.01% 0 BGP Router
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 61
High Processor Utilization
• Check how busy the peers areThe Table Version
You have 150k routes and see the table version increase by 150k every minute … something is wrong
You have 150k routes and see the table version increase by 300 every minute … sounds like normal network churnThe InQ
Flood of incoming updates or build up of unprocessed updates
The OutQFlood of outgoing updates or build up of untransmitted updates
Router#show ip bgp summaryNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd10.1.1.1 4 64512 309453 157389 19981 0 253 22:06:44 111633172.16.1.1 4 65101 188934 1047 40081 41 0 00:07:51 58430
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 62
High Processor Utilization
• If the Table Version is Changing QuicklyAre you in initial convergence with this peer?Is the peer flapping for some reason?
Examine the table entries from this peer: why are they changing?
If there is a group of routes which are constantly changing, consider route flap dampening
• If the InQ is highYou should see the table version changing quickly
If it’s not, the peer isn’t acting correctlyConsider shutting it down until the peer can be fixed
• If the OutQ is highLots of updates being generatedCheck table versions of other peers
Check for underlying transport problems
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 63
High Processor Utilization
• If the Table Version is Changing Quickly
RR#sh ip bgp vpnv4 rd 81:4BGP table version is 8216218, local router ID is 128.109.210.148Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S StaleOrigin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight PathRoute Distinguisher: 81:4*>i24.181.203.0/24 128.109.210.217 0 100 0 65502 i*>i128.109.5.44/30 128.109.210.217 0 100 0 ?*>i152.1.99.0/24 128.109.210.197 0 100 0 11442 i*>i152.1.254.0/24 128.109.210.197 0 100 0 11442 I*>i152.1.255.115/32 128.109.210.197 0 100 0 11442 ?*>i152.1.255.205/32 128.109.210.197 0 100 0 11442 I*>i172.26.38.128/27 128.109.210.217 0 100 0 65502 ?*>i192.168.1.0/30 128.109.210.197 0 100 0 ?
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 64
High Processor Utilization
• If the Table Version is Changing Quickly
RR#sh ip bgp vpnv4 rd 81:4 24.181.203.0BGP routing table entry for 81:4:24.181.203.0/24, version 8216205Paths: (1 available, best #1, no table)Advertised to update-groups:
165502, (Received from a RR-client)128.109.210.217 (metric 131584) from 128.109.210.217 (128.109.210.217)
Origin IGP, metric 0, localpref 100, valid, internal, bestExtended Community: RT:81:4Connector Attribute: count=1type 1 len 12 value 81:4:128.109.210.217mpls labels in/out nolabel/30
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 65
High Processor Utilization
• If the Table Version is Changing Quickly
RR#sh ip bgp vpnv4 all version ? <1-4294967295> Version number and aboverecent Offset from the current table version
RR#sh ip bgp vpnv4 all version recent 1BGP table version is 8216322, local router ID is 128.109.210.148Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S StaleOrigin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight PathRoute Distinguisher: 81:2*>i128.109.70.104/30
128.109.210.248 0 100 0 ?Route Distinguisher: 81:3*>i1.3.1.0/24 128.109.210.248 0 7500 0 19401 i
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 66
High Processor Utilization
• Check on the BGP ScannerWalks the table looking for changed next hops
Checks conditional advertisement
Imports from and exports to VPNv4 VRFs
Router#show processes | include BGP Scanner172 Lsi 407A1BFC 29144 29130 1000 8384/9000 0 BGP Scanner
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 67
High Processor Utilization
To relieve pressure on the BGP ScannerUpgrade to newer code
Most of the work of the BGP Scanner has been moved to an event driven modelThis has reduced the impact of BGP Scanner significantly
Reduce route and view count
Reduce or eliminate other processes which walk the RIBSNMP routing table walks, for instance
Deploy BGP Next Hop Tracking (NHT)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 68
Next Hop Tracking
• ATF is a middle man between the RIB and RIB clients BGP, OSPF, EIGRP, etc are all clients of the RIB
• A client tells ATF what prefixes he is interested in
• ATF tracks each prefixNotify the client when the route to a registered prefix changes
Client is responsible for taking action based on ATF notification
Provides a scalable event driven model for dealing with RIB changes
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 69
Next Hop Tracking
• BGP tells ATF to let us know about any changes to 10.1.1.3 and 10.1.1.5
• ATF filters out any changes for 10.1.1.1/32, 10.1.1.2/32, and 10.1.1.4/32
• Changes to 10.1.1.3/32 and 10.1.1.5/32 are passed along to BGP
RIB10.1.1.1/3210.1.1.2/3210.1.1.3/3210.1.1.4/3210.1.1.5/32
ATF
BGP BGP Nexthops10.1.1.310.1.1.5
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 70
Next Hop Tracking
• BGP Next Hop TrackingEnabled by default[no] bgp nexthop trigger enable
• BGP registers all nexthops with ATFHidden command will let you see a list of nexthopsshow ip bgp attr nexthop
• ATF will let BGP know when a route change occurs for a nexthop
• ATF notification will trigger a lightweight “BGP Scanner” runBestpaths will be calculatedNone of the other “Full Scan” work will happen
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 71
Next Hop Tracking
• Once an ATF notification is received BGP waits 5 seconds before triggering NHT scan
bgp nexthop trigger delay <0-100>
May lower default value as we gain experience
• Event driven model allows BGP to react quickly to IGP changes
No longer need to wait as long as 60 seconds for BGP to scan the table and recalculate bestpathsTuning your IGP for fast convergence is recommended
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 72
Next Hop Tracking
• Dampening is used to reduce frequency of triggered scans
• show ip bgp internalDisplays data on when the last NHT scan occurred
Time until the next NHT may occur (dampening information)
• New commandsbgp nexthop trigger enable
bgp nexthop trigger delay <0-100>
show ip bgp attr next-hop ribfilter
debug ip bgp events nexthop
debug ip bgp rib-filter
• Full BGP scan still happens every 60 secondsFull scanner will no longer recalculate bestpaths if NHT is enabled
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 73
High Memory Utilization
Why is BGP taking up so much memory?
A BGP speaker generally receives a number of copies of the same route or set of routes
Each of these copies of the same route or routes is called a view
A has two views of 10.1.1.0/24
10.1.1.0/24
A
B
C
D
First ViewSecond View
Views and Routes
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 74
High Memory Utilization
Multiple views can come from:iBGP peers peering with the same remote AS
iBGP peers peering with remote AS’ with (generally) the same tableThis is common in the case of the global Internet
eBGP peers peering with the same remote AS
eBGP peers peering with remote AS’ with (generally) the same tableThis is common in the case of the global Internet
Views and Routes
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 75
High Memory Utilization
• Multiple views exist in IGPs, as wellBut not on the same scale
Neighbor adjacencies in IGPs are generally on a lower scaleIn the hundreds, not the thousands
Neighbor adjacencies in IGPs normally pick up different routes, rather than the same route multiple times
• Each view takes up some amount of space250,000 routes x 100 views == a lot of memory usage
Views and Routes
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 76
High Memory Utilization
• To reduce memory consumption:
Reduce the number of routesThis is particularly true in providers supporting L3VPN servicesThe route and view count can escalate quickly when supporting many customer’s L3VPNs
Filter aggressively
Accept partial routing tables, rather than full routing tables
Reduce the number of viewsUse route reflectors rather than full mesh iBGP peeringPeer only when needed
Views and Routes
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 77
High Memory Utilization
• BGP implementations build their memory structures around minimizing storage
• Attributes are stored onceRather than once per routeEach route references an
attribute set, rather than storing the attribute set
• This is similar to the way BGP updates are formed
Attributes
10.1.1.0/24
10.1.2.0/24
10.1.3.0/24
10.1.4.0/24
AS Path 1
AS Path 2
Community Set 1
Community Set 2
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 78
High Memory Utilization
• The more unique attribute sets you’re receiving, the more unique attribute sets you need to store
You might have the same number of routes and views over time, but memory utilization can increase
Attributes
10.1.1.0/24
10.1.2.0/24
10.1.3.0/24
10.1.4.0/24
AS Path 1
AS Path 2
Community Set 1
Community Set 2
AS Path 3
Community Set 3
Community Set 4
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 79
High Memory Utilization
• To Conserve MemoryStrip unneeded attributes on the inbound side of eBGP peering sessions
Verify you don’t really need them, or they aren’t useful after the route has transited your AS
Communities are the biggest/only target
Use Communities wisely within your networkA large mishmash of communities can consume memory
Attributes
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 80
High Memory Utilization
• B advertises 10.1.1.0/24 to A
• A filters the route locally
• The filters on A are changed to permit 10.1.1.0/24
• But how does A relearn 10.1.1.0/24?
Soft Reconfiguration
10.1.1.0/24
Advertise 10.1.1.0/24
Blocked by filterA
B
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 81
High Memory Utilization
• With soft reconfiguration, A saves all the routes it receives from B
• Applies any inbound filters between this saved copy of B’s updates and the local BGP table
• If the local filters change, they can be reapplied by simply pulling all the updates from the saved table into the local BGP table
Soft Reconfiguration
10.1.1.0/24
Blocked by filterA
B Advertise 10.1.1.0/24
Local Copy
Local Filter
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 82
High Memory Utilization
• Keeping this local copy uses a lot of memory
• In general, don’t use soft-reconfiguration
• BGP now uses the route refresh capability to rebuild the local table if the local filters change
Soft Reconfiguration
10.1.1.0/24
Blocked by filterA
B Advertise 10.1.1.0/24
Local Copy
Local Filter
Consumes large amounts of memory
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 83
Routing Problems
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 84
BGP Routing Problems
• Route Reflector Loops
• Route Reflector Suboptimal Routes
• BGP Multihoming
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 85
Route Reflector Loops
• Router BBGP Next-Hop: Router ALocal Next-Hop: Router ASet: Next-Hop-Self
• Router CBGP Next-Hop: Router BLocal Next-Hop: Router D
• Router DBGP Next-Hop: Router ELocal Next-Hop: Router C
• Router EBGP Next-Hop: Router ALocal Next-Hop: Router ASet: Next-Hop-Self
B
C
D
E
A
RR
C
RR
C
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 86
Route Reflector Loops
This results in a permanent routing loop
Route reflectors must always follow the topology
Never peer through a route reflector client to reach a route reflector
B
C
D
E
A
RR
C
RR
C
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 87
Route Reflector Suboptimal Routing
Route reflectors can also cause routing to be different (or suboptimal) compared to full mesh iBGP
E advertises 10.1.1.0/24 through eBGP to both B and C
The local preference, MED, AS Path length, and all other attributes are the same for 10.1.1.0/24 at both B and C
10 5
10
5
10
5
IGP costsA
B DC
E
eBGP
eBGP
10.1.1.0/24
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 88
Route Reflector Suboptimal Routing
• Assume A, B, C, and D are configured for full mesh iBGP
• A chooses B as its exit point because of the IGP cost
• D chooses C as its exit point, because of the IGP cost
10 5
10
5
10
5
IGP costsA
B DC
E
eBGP
eBGP
10.1.1.0/24
Best Path
Best Path
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 89
Route Reflector Suboptimal Routing
• Assume B, C and, D are configured as route reflector clients of A
• A chooses B as its best path because of the IGP cost
• A reflects this choice to C, but C chooses its locally learned eBGP route over the internal through B
• A reflects this choice to D, and D chooses the path through B, even though the path through C is shorter
10 5
10
5
10
5
IGP costsA
B DC
E
eBGP
eBGP
10.1.1.0/24
Best Path
Best Path
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 90
Route Reflector Suboptimal Routing
• There is little you can do about this
• Whenever you remove routing information, you risk suboptimal routing
• Keeping the route reflector topology in line with the layer 3 topology helps
• iBGP multipath can resolve some of these problemsAt the cost of additional memory
• Otherwise, use policy to choose the best exit point
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 91
BGP Multihoming
You must announce assigned address block to Internet
You may also announce subprefixes—reachability is not guaranteed
Current RIR* minimum allocation is /21Several ISPs filter RIR blocks on this boundary
Several ISPs filter the rest of address space according to the IANA assignments
This activity is called “net police” by some
*RIR: Regional Internet Registryhttp://en.wikipedia.org/wiki/Regional_Internet_Registry
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 92
Assigned Netblock Filters
The RIRs publish their minimum allocation sizes at:AfriNIC: www.afrinic.net/docs/policies/afpol-v4200407-000.htm
APNIC: www.apnic.net/db/min-alloc.html
ARIN: www.arin.net/reference/ip_blocks.html
LACNIC: lacnic.net/en/registro/index.html
RIPE NCC: www.ripe.net/ripe/docs/ripe-ncc-managed-address-space.html
(Above picture from http://www.iana.org/numbers)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 93
Assigned Netblock Filters
IANA* publishes the address space it has assigned to end-sites and allocated to the RIRs:
www.iana.org/assignments/ipv4-address-space
Several ISPs use this published information to filter prefixes on:
What should be routed (from IANA)
The minimum allocation size from the RIRs
Example….“Sprint reserves the right to aggregate any announcement for a network smaller than /19 when advertising to external peers such as AT&T, UUnet etc.”https://www.sprint.net/index.php?p=policy_bgp
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 95
One Path as Backup
This does not direct all the traffic through the one link, however
The AS path length doesn’t impact forwarding decisions within provider 2
Virtually all providers set the local preference to prefer routes learned from customers over routes learned from peers
Problem #1: We wanted the link between RtrB and RtrD to be used only as a backup but that is not happening
Provider 1 Provider 2
Local Pref Set to Prefer Routes Learned from Customers
10.1.1.0/20{AS#}
B
DC
A
10.1.1.0/20{AS#, AS#}
Upstream Customer 2
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 96
One Path as Backup
If customer 2 prefers the path through provider 2
They could have a default only to provider 2
They could be accepting a partial routing table
Etc. …
Then provider 2 will prefer the B->D link
Rather than taking the path through the Upstream->provider 1->A->C
All the traffic coming from provider 2’s customers will follow the B->D link
AS Path Prepend
Provider 1 Provider 2
Local Pref Set to Prefer Routes Learned from Customers
B
DC
A
Upstream Customer 2
10.1.1.0/20{AS#}
10.1.1.0/20{AS#, AS#}
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 97
10.1.1.0/24
Inbound Load Sharing
65600
65300 65200
65100
Why can’t we just AS path prepend for this?
Traffic from AS65500 will still flow through 65200
Traffic from AS65400 will still flow through AS65300
Only the traffic sourced from AS65600 will be impacted by AS path prepend by itself
This might work, or it might not
6550065400
Problem #2: We wanted to inbound load share between these 2 connections
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 98
Inbound Load Sharing
What are our other options?
Longer match prefixes are still your friend…*
Advertise 10.1.0.0/22 through one connection
Advertise 10.1.0.0/22 and 10.1.0.0/23 through the other connection
Customer 2
Provider 2Provider 1
Upstream
A
10.1.0.0/2210.1.0.0/2310.1.0.0/22
DC
B
* If your service provider will support this
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 99
Inbound Load Sharing
You can extend this concept by
Adding various longer prefix matches on both linksCombining advertisements of
longer prefixes out both links with AS path prepending
For instance, here we are Prepending the /22 to influence
traffic towards CAdvertising a longer prefix
to influence other traffic towards D
10.1.0.0/2210.1.0.0/2310.1.0.0/22
Customer 2
Provider 2Provider 1
Upstream
A
DC
B
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 100
Inbound Load Sharing
This example is more commonplace
Shows how ISPs and end-sites subdivide address space frugally, as well as use the AS-PATH prepend concept to optimise the load sharing between different ISPs
Notice that the /22 aggregate block is always announced
10.1.0.0/2210.1.0.0/2310.1.0.0/22
Customer 2
Provider 2Provider 1
Upstream
A
DC
B
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 101
Show BGP Neighbors
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 102
XR – Show BGP NeighborRP/0/RP1/CPU0:CRS#sh bgp neighbor 128.109.210.148 detail
BGP neighbor is 128.109.210.148Remote AS 81, local AS 81, internal linkRemote router ID 128.109.210.148BGP state = Established, up for 02:46:51Last read 00:00:41, hold time is 180, keepalive interval is 60 secondsLast write 00:00:51, Last write before reset 02:47:50Last write pulse rcvd Jul 1 15:18:00.690 last full Jul 1 12:15:23.200 pulse count 25209Socket not armed for io, armed for read, armed for writePrecedence: internetNeighbor capabilities: Adv Rcvd
Route refresh: Yes Yes4-byte AS: Yes YesAddress family VPNv4 Unicast: Yes YesAddress family IPv4 MDT: Yes Yes
Message stats:InQ depth: 0, OutQ depth: 0
Last Sent Sent Last_Rcvd RcvdOpen: Jul 1 15:18:00.689 39 Jul 1 15:18:00.688 39Notification: Jul 1 15:17:40.486 33 --- 0Update: Jul 1 15:28:24.021 81254 Jul 1 16:31:32.984 60310Keepalive: Jul 1 18:04:01.011 12867 Jul 1 18:04:10.549 12597Route Refresh: Jun 26 22:47:30.307 4 --- 0Total: 94197 72946
Minimum time between advertisement runs is 0 seconds
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 103
XR – Show BGP Neighbor..For Address Family: VPNv4 UnicastBGP neighbor version 39025483Update group: 0.2 (Update Generation Throttled)
Route refresh request: received 0, sent 2Policy for incoming advertisements is pass-allPolicy for outgoing advertisements is pass-all1521 accepted prefixes, 1521 are bestpathsPrefix advertised 1503, suppressed 0, withdrawn 1500, maximum limit 524288Threshold for warning message 75%An EoR was received during read-only modeLast ack version 39025483, Last synced ack version 0Outstanding version objects: current 1, max 5
For Address Family: VPNv6 UnicastBGP neighbor version 82192Update group: 0.1Route refresh request: received 0, sent 2Policy for incoming advertisements is pass-allPolicy for outgoing advertisements is pass-all6 accepted prefixes, 6 are bestpathsPrefix advertised 1, suppressed 0, withdrawn 0, maximum limit 524288Threshold for warning message 75%An EoR was received during read-only modeLast ack version 82192, Last synced ack version 0Outstanding version objects: current 0, max 2
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 104
XR – Show BGP NeighborConnections established 39; dropped 38Last reset 03:19:07, due to BGP Notification sent: illegal networkTime since last notification sent to neighbor: 03:19:07Error Code: illegal networkNotification data sent:
FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF002A0200 00001380 0F100001 4260000000510000 0004806D D2D9
RP/0/RP1/CPU0:CRS#
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 105
7600 – Show BGP Neighbor7600#sh ip bgp all neighbors 128.109.210.148 .For address family: VPNv4 UnicastBGP neighbor is 128.109.210.148, remote AS 81, internal linkMember of peer-group RRs for session parametersBGP version 4, remote router ID 128.109.210.148BGP state = Established, up for 03:12:13Last read 00:00:31, last write 00:00:04, hold time is 180, keepalive interval is 60 secondsNeighbor sessions:
1 active, is not multisession capableNeighbor capabilities:
Route refresh: advertised and received(new)Address family IPv4 Unicast: advertisedAddress family IPv4 MDT: advertised and receivedAddress family VPNv4 Unicast: advertised and receivedAddress family VPNv6 Unicast: advertised and receivedMultisession Capability: advertised
Message statistics:InQ depth is 0OutQ depth is 0
Sent RcvdOpens: 1 1Notifications: 0 0Updates: 46 98Keepalives: 213 186Route Refresh: 0 0Total: 260 285
Default minimum time between advertisement runs is 0 seconds
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 106
7600 – Show BGP Neighbor..Address tracking is enabled, the RIB does have a route to 128.109.210.148Connections established 1; dropped 0Last reset neverTransport(tcp) path-mtu-discovery is enabledGraceful-Restart is disabled
Connection state is ESTAB, I/O status: 1, unread input bytes: 0 Connection is ECN DisabledMininum incoming TTL 0, Outgoing TTL 255Local host: 128.109.210.197, Local port: 18367Foreign host: 128.109.210.148, Foreign port: 179Connection tableid (VRF): 0
Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)
Event Timers (current time is 0xB248B0):Timer Starts Wakeups NextRetrans 222 0 0x0TimeWait 0 0 0x0AckHold 215 185 0x0SendWnd 0 0 0x0KeepAlive 0 0 0x0GiveUp 0 0 0x0PmtuAger 1 1 0x0DeadWait 0 0 0x0Linger 0 0 0x0
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 107
7600 – Show BGP Neighbor..iss: 4216790207 snduna: 4216799431 sndnxt: 4216799431 sndwnd: 15890irs: 839066505 rcvnxt: 839118621 rcvwnd: 16308 delrcvwnd: 76
SRTT: 300 ms, RTTO: 303 ms, RTV: 3 ms, KRTT: 0 msminRTT: 0 ms, maxRTT: 300 ms, ACK hold: 200 msStatus Flags: noneOption Flags: higher precendence, nagle, path mtu capable
Datagrams (max data segment is 536 bytes):Rcvd: 527 (out of order: 0), with data: 306, total data bytes: 52115Sent: 507 (retransmit: 0 fastretransmit: 0),with data: 230, total data bytes: 9223
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 108
Summary
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 109
Knowing Your Crime Scene and Your Suspects
The Usual Suspects Configs
Layer 1
IGP
Memory
CPU
MTU
Design
Input Queues
Bug
………. BGP Table
BGP Best Path
RIB
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 110
AS 1BGPTCP
IP
AS 2
Layer 2 Cloud
eBGP
R2R1
BGPTCP
IP
Knowing Your Crime Scene and Your Suspects
Your Usual Suspects Not true point-to-point
Down detection
Presentation doesn’t match provisioning
………
+ the “usual” suspects
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 111
AS 65500
Intranet/IGP
iBGP
R2R1
Knowing Your Crime Scene and Your Suspects
Your Usual Suspects Network stability
IGP churn
........
+ the “usual” suspects
BGPTCP
IP
BGPTCP
IP
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 112
Questions?
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 113
Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily.
Receive 20 Cisco Preferred Access points for each session evaluation you complete.
Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.