04/18/23 1

CompSci 514: Computer Networks

Lecture 5: BGP problemsXiaowei Yang

04/18/23 2

Today

• Known problems of BGP– Multi-homing– Instability– Delayed convergence

• Slow failover

• Discussing fixes– Root cause, ghost flushing etc.

04/18/23 3

Failover

• BGP is designed for scaling more than fast failover– Many mechanisms favor this balance– Route flap damping, for example.

• If excess routing changes (“flapping”), ignore for some time.

• Has unexpected effects on convergence times.– Route advertisement/withdrawal timers in the 30

second range– Effect: tens of seconds to many minutes to

recover from “simple” failures.– 15-30 minute outages not uncommon.

04/18/23 4

Multi-homing

• Connect to multiple providers– Goal: Higher availability, more capacity

• Problems:– Provider-based addressing breaks– Everyone needs their own address space

04/18/23 5

Multi-homing increases routing table size

Mutil-home.com

128.0.0.0/8204.0.0.0/8

204.1.0.0/16

ISP2 ISP1

You can reach 128.0.0.0/8And 204.1.0.0/16 via ISP1

ISP3

204.1.0.0/16 ISP1204.1.0.0/16128.0.0.0/8 ISP1

204.1.0.0/16 ISP2

204.0.0.0/8 ISP2

04/18/23 6

Global routing tables continue to grow

Source: http://bgp.potaroo.net/as6447/

04/18/23 7

Other BGP problems

• Convergence: BGP may explore many routes before finding the right new one.– Labovitz et al., SIGCOMM 2000

• Correctness: routes may not be valid, visible, or loop-free.

• Security: There is none!– Some providers filter what announcements their

customers can make. Not all do.– See paper discussion site for pointers

04/18/23 8

Measurement studies

• Two papers (measurement)– End-to-end traffic– Routing messages

• Experimental techniques

• Results

04/18/23 9

Internet Routing Instability

• Goals: how often BGP sends updates to change routes

• Methodology:– Analyzing BGP logs for a long time

Terms

• WADiff: withdrawal announcement

• AADiff: announcement announcement

• WADup: same route withdrawal announcement

• AADup: same route announcement announcement

• WWDup: same route withdrawal wthdrawal

04/18/23 CPS 214 10

Observed pathologies

• Repeated WWDup, WADup, AADup

• Why are they pathologies?

04/18/23 CPS 214 11

04/18/23 12

• Majority of BGP updates are WWDup• WWDup belong to ASes that never announce them• Why?

– Stateless BGP, does not remember what have sent to peers– Send withdrawals to all peers

Possible origins of instability

• Stateless BGP

• Physical link errors

• Unjittered timers

• IGP, BGP interactions

• Conflicting routing policies

04/18/23 13

04/18/23 14

Data analysis techniques

• Time series analysis

• Frequency analysis– Fast Fourier transform – Maximum entropy spectral estimation

• Different estimation methods, but both find significant frequencies at seven days, and 24 hours

Main results

• Much more updates than expected– 99% is pathological. Impressive!– A taxonomy to analyze pathologies

• Speculation of causes– Configuration errors, router bugs– Correlate with traffic load, perhaps due to router

architectures– Open research questions: root cause of updates

• Motivated much follow-up work

04/18/23 15

04/18/23 16

End-to-end routing behavior

• Goals: study routing pathologies, route stability, and routing asymmetry

• Methodology:– End-to-end measumrents– Traceroutes from N sites, N2 paths– Exponentially spaced-out sampling

• Nice properties!• Unbiased• PASTA: the fraction of measures that observe a given

state is equal to the time that the system spends in that state

– Two datasets: D1 and D2 for different intervals• 1~2 days, 2 hrs, 2.75 days

Measurement infrastructure

• 37 hosts

• 8% of Internet at that time

• Pair-wise traceroute

04/18/23 17

04/18/23 18

Pathologies• Persistent loops: 0.13-0.16%, some lasted hours

– Same router appearing in traceroute more than three times

• Erroneous routing: packets to UCL sent to Israel

• Mid-stream change, 0.16% (D1) and 0.44% (D2)– Suggests route change

• Infrastructure Failure: availability 99.8%, 99.5%– Unreachable to host– Telephone networks: 2 hours in 40 years, five nines

• Outages: more than six packet losses in a traceroute

04/18/23 19

Stabilities

• Prevalence: the probability of observing a particular route

• Persistence: how long a route lasts• Examples:

– R1, R2, R1, R2– R1, R1, R2, R2– Same prevalence, different persistence

04/18/23 CPS 214 20

Prevalence of dominant route• Pdomp =kp/np

• Internet paths are strongly dominated by one path

04/18/23 21

2/3 of routes persist for days

04/18/23 22

Path asymmetry

• Common– Don’t assume path symmetry in your

design– 49% of measures have asymmetric paths

differed by at least one city– 30% observed different ASes– 20% differ by more than one city/AS– Q: what might cause it?

Comments on this paper

• Seminal work on Internet measurement

• Solid data

• Rigorous analysis

04/18/23 23

04/18/23 24

Delayed Internet Convergence

MeasurementProblem

discoveryModeling& analysis

Improvement

• Methodologies

04/18/23 25

Experiments setup

• Actively inject BGP faults– How is fault injected?

• Passively listen at peering sessions, and use NTP synchronized machines to calculate the convergence time

• Actively send probe packets to observe end-to-end packet loss and latency

• Much BGP work later uses similar measurement techniques.

04/18/23 26

Results show delayed convergence

• Bad news travels slow.

04/18/23 27

Slow routing convergence results in poor end-to-end performance

04/18/23 28

What causes the delayed routing convergence?

• A simple BGP convergence model reveals that in the worse case, all possible paths are explored before a prefix is withdrawn.

• No minimum advertisement timer: synchronized network, global message queue

0

1 2

R

(*0R, ∞,, 2R) (∞, ∞, *2R)(∞, ∞, *20R)

(*0R, 1R, ∞,)(01R, *1R, ∞)(*01R, 10R, ∞,)

(∞, *1R, 2R)(∞, ∞, *2R)(∞, ∞, ∞)

01R 01R

10R

10R

20R

20R

04/18/23 29

Min router Advertisement interval timer(MRAI) reduces message

count• Why?

– MATI introduces synchronization. Multiple announcements are combined into one announcement, reducing the total message count.

• However, the convergence time becomes proportional to timer_interval * (n-3)

04/18/23 30

Let’s brain storm…

• How can we fix the slow convergence problem?– What is the solution proposed by the authors?

• Sender-side loop detection. When a sender detects a loop, it sends a withdrawal to a neighbor immediately. Since withdrawal is not subject to MATI delay, this improvement reduces both message count and convergence time.

– What exactly is the root cause of BGP’s slow convergence problem?

– Can you come up with any solution?

04/18/23 31

Sender-side loop detection

• Without sender-side loop detection:– AS3 AS1: 301R– This announcement is sent out when MRAI

timer expires

• With sender-side loop detection:– AS3->AS1: withdrawal– Withdrawal is sent out immediately. AS1

knows it has no path.

04/18/23 32

BGP assertion

• Detect path inconsistency between different neighbors• If inconsistency is found, give path learned from direct

neighbors high priority• Sensitive to topology• Does not eliminate all invalid paths

N1R D

X Y

N2

XN1 R: N1 X Y D

R: N1 D N2 N1 D

04/18/23 33

Ghost flushing

• If new path is worse than last announced path, and router advertisement timer has not expired yet, send a withdrawal immediately.

• The withdrawal flushes “ghost” information.• Reduces the convergence time because withdrawals

are not delayed by MRAI, but does not help much with “Tlong.”

N1R D

X Y

N2

XN1 R: withdrawal

R: N1 D N2 N1 D

04/18/23 34

BGP root cause notification

• Neither BGP-assertion nor ghost flushing works well in this topology.– Why?– BGP-assertion: 3 and 6 are both direct neighbors to 5, but their

announcements may be inconsistent– BGP ghost flushing: the newer path is subject to MATI delay

• Explicitly send out link up/down information• Essentially adds link-state information into BGP• Sequence number is used to order the notifications.• Open research problem: can you get rid of sequence number?

5

3

4

2

1

0X

6

04/18/23 35

Summary

• BGP’s slow convergence problem and other problems

• It represents a message overhead, processing overhead, and latency tradeoff. We do not yet know the best solution to address this problem.

Comments

• Measurement paper– Data– Data collection techniques– Data analysis

• When to write a measurement paper

04/18/23 36