How to get good TCP performance over

asymmetric networks Hari Balakrishnan

MIT Lab for Computer Science

http://wind.lcs.mit.edu/hari@lcs.mit.edu

With: Venkata Padmanabhan (Microsoft Research) and Randy Katz

(UC Berkeley)

Motivation

• Many emerging (in some cases, “emerged!”) networks are asymmetric– Cable modems, “wireless” cable, (A)DSL– Direct Broadcast Satellites– Packet radio networks

• How does TCP behave over asymmetric networks?– TCP relies on ACK feedback to work, which

might be an issue

When is a network asymmetric?

Bandwidth: 10-1000X more in forward direction

Latency: Due to MAC protocol interactions Loss-rate: Higher loss-rate in one direction

The network characteristics in one directionsignificantly affect performance in the other

Sender Receiver

Forward

Reverse

Router

Router

Why is this a problem for TCP?

• TCP relies on ACKs for reliability and for congestion control

• New, cumulative ACKs “slide” and “bump up” window, allowing new transmissions (ACK clocking)

• Smooth ACK flow smooth transmits;flaky ACK flow bursty transmits

7

43

65

Cumulative Acknowledgments (ACK)

12

34

2

0

TCP Overview

– Window-based algorithm to determine sustainable rate

– Upon congestion, reduce window– “ACK clocking” sends data smoothly

8

11

76

13

4

12 lost1

5

Timeout = srtt + 4 * rtt_deviation (coarse-grained)Fast retransmissions when duplicate ACKs arrive

1. Loss recovery

2. Congestion control

0

109

TCP Dynamics

4.13E+06

4.14E+06

4.15E+06

4.16E+06

4.17E+06

4.18E+06

4.19E+06

33.52 33.54 33.56 33.58 33.6 33.62 33.64 33.66 33.68 33.7 33.72

Data

ACKs

Window

Fast retransmission

Duplicate ACKs

Seq

uenc

e nu

mbe

r (b

ytes

)

Time (s)

RTT

Typical Performance

Technology RatedBandwidth

Typical TCPThroughput

LucentWaveLAN

2 Mbps 50 Kbps-1.5 Mbps

MetricomRicochet

100 Kbps 10-35 Kbps

Hybridwireless cable

10 Mbps 0.5-3.0 Mbps

Goal: To bridge the gap between perceived and rated performance

Bandwidth Asymmetry

• Shows up in asymmetric access technologies– ADSL, (wireless) cable modems, DBS

• Low-bandwidth ACK channel• ACKs don’t come back fast enough or get

lost

Problem: Imperfect ACK feedback degrades TCP performance

Problems

SenderReceiver

Forward

ACKs

Router

Bottleneck Router

0

Data 8Data 11

Data 10 Data 9

321 45 6 71. Acks arrive slowly (large buffer)

2 5

4 71

3 6 2. Acks are dropped (small buffer)

1Data Data 3. Acks are queued behind data packets

Lakshman & Madhow 97Kalampoukas et al. 97Balakrishnan et al. 97

Performance depends on 1. Normalized asymmetry ratio 2. Bottleneck buffer sizes 3. Competing reverse traffic

Hybrid Wireless Cable

0

1

2

3

4

5

6

0 20 40 60 80 100 120 140 160 180 200

T

CP T

hro

ugh

pu

t (M

bp

s)

10 Mbps Ethernet

28.8 C-SLIP

28.8 SLIP9.6 C-SLIP

9.6 SLIP

Return channel speed and latency affects performance

Receiver socket buffer size (KB)

Latency Asymmetry: Packet Radio Networks

Internet

PTER

FH

GW

MH

Modem PRER

ER

PT

PT

PT

PT

PT

PT

Fixed HostEthernet Radios

Poletop Radios

Mobile Host

RTS

CTS

Half-duplex radiosSynchronization before communication

MAC Protocol Interactions

Internet

PTER

FH

GW

MH

Modem PRER

ER

PT

PT

PT

PT

PT

PT

Fixed HostEthernet Radios

Poletop Radios

Mobile Host

Pkt

Ack

RTS

No response

Exponentialbackoff

Problem: Large, variable communication latency

Problem: Large Round-Trip Time Variations

Metricom Ricochet Wireless Network

RTT Estimate

0

1000

2000

3000

4000

5000

6000

1 3 5 7 9 11 13 15 17 19Sample number

RT

T E

stim

ate

(m

sec)

• Mean rtt = 2.45s, std deviation = 1.5s long timeout!

• Long idle periods after multiple losses (~ 20 Kbps)• In contrast, UDP throughput = 50-64 Kbps• ACK flow affects data latency

Sequence Number trace

0

50000

100000

150000

200000

250000

300000

0 20 40 60 80 100Time (sec)

Se

que

nce

Num

be

r (b

yte

s) Fast retransmissions

Timeouts

Solutions

• Problems arise because of imperfect ACK feedback

• 1. Reduce frequency of acks– ACK Filtering (AF)– ACK Congestion Control (ACC)

• 2. Handle infrequent acks– Sender Adaptation (SA)– ACK Reconstruction (AR)

General solutions for different asymmetric situations

ACK Filtering (AF)

9 7511 13

3 75

Forward

Router

1Server Client

Router

• Purge all redundant, cumulative ACKs from constrained reverse queue

• No persistent filter state

• Used in conjunction with sender adaptation or ACK reconstruction

Server

Client

Router

8

Data 21

Data 22

Data 20

1816

14

Data 19

10 Delack factor = 2

Adaptive extension of TCP delayed ACKs

Forward

ACK Congestion Control (ACC)

ACK Congestion Control (ACC)

Sender

Router

Data

Data

Data

22

Data

12

Data

RED marking of ECN bit (Explicit Congestion

Notification)

Delack factor = 2

Receiver

Forward

ACK Congestion Control (ACC)

Router

Data

Data 40

Data

Data

22

Echo ECN marking to receiver

Delack factor = 2

Forward

Sender

Receiver

ACK Congestion Control (ACC)

Router

Data 42

Data 43

Data 41

3640

Data 40

Delack factor = 4

Forward

Sender

Receiver

Sender Adaptation (SA)• Infrequent ACKs slow window growth• Sender tends to be bursty

Server

1 9 15

1. cwnd += 8

cwnd += 8/cwnd

Incr window by amount acked

19 20 21 22 . . .

2.

Regulation: pace packets out at rate estimated by cwnd/srtt

This reduces burstiness

Forward

Receiver

ACK Reconstruction (AR)

975

111 13

3 75

ACK filterACK reconstructor753

9Server

Forward

Client

• Regenerate ACKs at other end of reverse link

• Shield sender from gaps in ACK sequence (no TCP changes needed)

• AR rate determined by input ACK rate and target ACK spacing

• Reconstructor state is soft

1

Performance: Bandwidth Asymmetry

– TCP transfers in the forward direction alone– Maximum window size 100 KB; no forward losses

– Header compression helps– Large reverse buffer hurts for Reno and ACC– Fairness greatly improves using AF and ACC

02468

10

Thr

ough

put

(Mbp

s)

10

pkt

C/10

pkt

50

pkt

C/50

pkt

Reno

ACC

AF

AF+AR

Multihop Wireless Simulations

– 1 to 3 wireless hops on path– Radio turnaround time = 3-12 ms– Radio queue size = 10 packets– Exponential backoff in multiples of 20 ms

slots

ERPT

Receiver

Sender

100 Kbps, 10 ms

10 Mbps1 ms

PT

PT

Single TCP Transfer• AF reduces chances that peer radio is busy

– MAC backoffs less frequent

• RTT std deviation reduces from 1.5s to 0.6s

0

10

20

30

40

50

60

1 hop 2 hops 3 hops

Reno

Reno+ACC

Reno+AF

Th

roughput

(Kbps)

AF: 20-35% throughput improvement compared to Reno

Concurrent Transfers• Metrics: utilization and fairness • Simultaneous connections over 2-hop network

– Predictable and consistent performance with AF

• Long timeouts cause unpredictable performance

0

0.2

0.4

0.6

0.8

1

2 4 6 8 10 12

Number of connections

Jain

's fa

irne

ss in

dex

AF

Reno

AF: 25% improvement in fairness over Reno

Combining Technologies

Internetserver

Internet

Hybrid PoP Client

10 Mbps, 2 ms

Wireless transmitter

PTER

Requests & ACKs

Web data

Cable forward channel with packet radio reverse channelWorkload: Multiple concurrent Web-like transfers

Result: Ack filtering greatly improves scaling behavior (average completion time vs. # of concurrent transactions)

Status

• Extensive simulation results under various bandwidth and latency asymmetric conditions

• Implementation and evaluation under BSD/OS 3.0 (code available)

• Documentation– Paper to appear ACM MONET (extended,

revised version from MOBICOM 97– Internet-draft submitted (PILC WG)

Cross-layer optimizations

• A way of performing protocol-specific optimizations at link-layer

• Lightweight, soft-state agents in network infrastructure

• State is soft: not essential for correctness; improves performance

• Easy to implement!• Examples include ACK filters and

reconstructors

Summary• Asymmetry affects TCP performance

– ACK channel affects forward throughput

• Asymmetry comes in various flavors– Bandwidth asymmetry (technology)– Latency asymmetry (MAC interactions)

• Fortunately, good performance can be obtained using a two-pronged approach– Reduce frequency of ACKs (AF, ACC)– Handle infrequent ACKs (SA, AR)

• Cross-layer optimizations are A Good Thing!