Measurement Driven Deployment of a Two-Tier Urban Mesh Access Network

J. Camp, J. Robinson, C. Steger, E. KnightlyRice Networks Group

MobiSys 20066/20/06

Two-Tier Mesh Architecture

• Limited Gateway Nodes wired to Internet

• Mesh Nodes wirelessly forward bandwidth

• Backhaul Tier (Blue) - mesh node to mesh node

• Access Tier (Red) - mesh node to client node

City-wide Two-tier Mesh: Houston RFP

• Three tiers of access: public service, Internet access, safety

• 620 square miles, Coverage:

• 95% Outside

• 90% Inside (window)

• $1 million startup capital downtown, $100 million total

• 18k mesh nodes, 3k gateways, Over 1 million end nodes!

Technology For All/Rice Deployment

• TFA MISSION: “Empower low income communities through technology”

• Pilot neighborhood: Houston’s East End

• Per capita income 1/3 national average ($10k), 37% of children below poverty

• 64.2% of adults without GED

• 4.2 km2 covering 40,000 residents

• Education and work-at-home (“Learn-and-Earn” and Job-Tech)

Outline

• TFA/Rice Background

• Objectives and Hardware

• Measurement Driven Deployment

• Single Hop Measurements

• Multihop Measurements

• Computational Placement Model

• Related Work

• Conclusion

TFA Design Objectives

• Single wireline gateway (burstable to 100 Mb/sec)

• Coverage for entire 4 sq. km neighborhood (vs. only homes with mesh nodes)

• 1 Mb/sec minimum access rate

• $15k per square km

• Programmable platform for protocol design and measurement

Off-the-Shelf Hardware• Mesh Node

• 802.11b, 200mW, Linux OS

• 1GHz x86, 4GB Flash

• 15 dBi antennas at 10 m (serve both access and backhaul)

• Client Node

• 802.11b, 200mW

• Engenius CB-3 Ethernet Bridge (like DSL/Cable modem)

Methodology• Single Link Behavior = pathloss (ɑ), throughput as a f(SNR)

• Multihop Measurements = traffic matrices (β)

• Long-lived TCP flows, Static Rate Limited Flows, Web Traffic

• Placement Study = wire ratio (w), topology (regular grid, regular grid w/ perturbations, and random)

• Average mesh node throughput

• Network Reliability

ComputationalPlacement

Model

Single Link! & T(SNR)

Trafc Matrices"

Topology Generator(spacing, wire ratio)

Topology Information

(x, y)

Mesh PerformanceMetrics

Mesh Node Throughput(Average)

Mesh Node Reliability

Single Hop Experiments

• Empirically measure importance of critical deployment factors

• Accurate understanding of propagation environment (pathloss)

• Accurate throughput to signal strength mapping

• Link Measurements

• Backhaul and Access* Links

* Not shown here, see results in paper

Backhaul Link Measurements

• 235 Measurements (30 seconds)

• Concentrated measurements at distances > 175 meters

• Experiment Set-up

• 10 m height for fixed node

• 10 m height for portable node

• UDP traffic

• RTS/CTS enabled

• PHY layer autorate enabled

800 1000 1200 1400 1600 1800

600

700

800

900

1000

1100

1200

Pixels (Meters/1.9)

Pixe

ls (M

eter

s/1.

9)

Throughput(SNR)

!100 !90 !80 !70 !60 !500

1000

2000

3000

4000

5000

6000

7000Slope = 240 kb/s/dBmX!Intercept = !86.4 dBm

Received Signal Strength (dBm)

UD

P Th

roug

hput

(kb/

s)

• Need to find throughput as a function of signal strength for model

• Manufacturer specification not sufficient (overly optimistic)

• Linear Approximation on logarithmic scale

• Target throughput for backhaul links 3 Mbps

• -75 dBm signal strength

• Access: -86 dBm for 1Mbps (DSL speeds)

ManufactureSpecification

Pathloss

• Pathloss (α) = 3.3

• Theory - urban pathloss from 2 to 5

• Increased pathloss (access links 3.7)

• Increased height and better antenna

• Increased Shadowing, 5.9 where access links (4.1)

• 200-250 m range at -75 dBm for 3 Mbps

• Access: 150-200 m range at -86 dBm for 1 Mbps

0 100 200 300 400 500!100

!90

!80

!70

!60

!50

!40

!30

!20

!10

0

Distance (m)

Rec

eive

d Si

gnal

Pow

er (d

Bm)

Pathloss Exp = 3.2683Shadowing Std = 5.9427

MeasurementsFree SpaceMean+1 Stdev!1 Stdev+2 Stdev!2 Stdev

Deployment Findings

• Accurate propagation measurements critical (theory says 2 to 5)

• Findings from placement model when we set internode spacing according to theory

• 2 yields completely disconnected network

• 3.5 yields overprovision factor of 55%

• 4 yields overprovision factor of 330%

• 5 yields 9 times overprovisioning (approx. $1 Billion vs. $100 million for Houston!)

• Accurate throughput-signal-strength function critical - manufacturers values overestimate link range 3X -> disconnected

• Requires only a few measurements

• 15 random measurements = std. dev. 3% about average

• 50 random measurements = std. dev. 1.5% about average

• Issue: Spatial Bias

• Single Active Flow known

• Multiple active flow experiments measured

• Interflow and self-contention

• Experiments:

• Long-lived TCP Flows (Upload, Download*, Bidirectional*)

• Rate Limiting

• Web traffic* (download)

Multihop Experiments

* Not shown here, see results in paper

TCP Long-lived Upload

• Upload experiences severe spatial bias

• Packet loss (from contention/collision)exacerbates effect

• RTS/CTS overhead outweighs fairness improvement for starved nodes

1 2 3 40

500

1000

1500

2000

2500

3000

Number of Hops from Gateway Node

Uplo

ad T

hrou

ghpu

t (kb

ps)

Concurrent Flows, RTS/CTS OffConcurrent Flows, RTS/CTS On

71%

58%

1%3%

Rate Limiting

• Experiment: statically rate limit all nodes to the same rate (x-axis)

• Expect fair rate to be 1/9 (444 kbps) of effective capacity (4 Mbps)

• Result: 450 kbps has fair per-node throughput

• Upload still has spatial bias at 450 kbps

100 200 300 400 500 600 700 800 900 10000

100

200

300

400

500

600

700

800

900

1000

Static Rate Limit (kbps)

Down

load

Thr

ough

put (

kbps

)

1st Hop Node2nd Hop Node3rd Hop Node4th Hop Node

1234

567

89

Download

100 200 300 400 500 600 700 800 900 10000

100

200

300

400

500

600

700

800

900

1000

Static Rate Limit (kbps)

Uplo

ad T

hrou

ghpu

t (kb

ps)

1st Hop Node2nd Hop Node3rd Hop Node4th Hop Node

1234

567

89

Upload

Multihop Measurement Findings

• Imperative to consider contending flows

• Single active flow measurements lead to large fraction of starving and disconnected nodes (comparing the β values yields avg. throughput per mesh node of twice actual -- 2x overestimation)

• Starvation in fully backlogged upload

• Compounding of MAC-induced loss & equal prioritization of intermediate node’s and forwarded traffic

• RTS/CTS overhead outweigh gains in starved nodes

• Proper limiting of flows alleviates starvation

• Web traffic allows statistical multiplexing to alleviate starvation (even without rate limiting)

Placement Study

• Results

• Effect of Traffic Matrices

• Effect of Perturbations

• Reliability*

• Grid vs. Grid w/ Perturbations vs. Random*

• Case Study Network Deployment*

• Experimental Set-up

• Square (Manhattan) Grid, add perturbations

• Poisson placement for random topologies

* Not shown here, see results in paper

Effect of Traffic Matrices

• Rate limited flows achieve approx. 1/2 of long-lived flows

• Unfair traffic pattern

• Web traffic able to achieve both high throughput and fairness

• Employing dynamic rate limiting w/ web traffic would be ideal

0 10 20 30 40 50 60 700

500

1000

1500

2000

2500

3000

3500

Node Density (nodes/km2)

Aver

age

Mes

h No

de T

hrou

ghpu

t

Parking Lot DownParking Lot UpRate Limited DownRate Limited UpWeb Emulation1/h Falloff

Perturbations

• Increase in std. deviation as perturbation increase

• 6% increase in throughput up to 40 meters (1/6 of inter-node spacing)

• Not In My Backyard Scenario

• Uniform perturbation distribution

• 225 m spacing (grid)

0 10 20 30 40 50 60 70 801000

1100

1200

1300

1400

1500

1600

1700

1800

Average Perturbation (meters)

Aver

age

Mes

h No

de T

hrou

ghpu

t (kb

ps)

w=1/16

Status of TFA Deployment

• 12 Nodes (Red) Deployed

• Approximately 2 square kilometers covered and growing

• 700+ users and rapidly growing

• New Devices (PDAs) and applications (health care)

Related Work

• MIT Roofnet Measurements

• Strong line-of-sight component, Single Tier

• Single active multihop flows

• Philadelphia Wireless Spectrum Analysis

• Spectral scan of 49 points (135 mi.2)

• Signal strength measurements of access links

• Optimizing the Placement of iTAPs (Microsoft)

• Wired mesh node placement problem

• Analytical optimization formulation

Conclusion

• Critical factors to consider in deploying mesh networks

• Accurate knowledge of propagation environment

• Accurate throughput-signal-strength function

• Traffic characterization must include concurrently contending multihop flows

• Rate Limiting to avoid starvation of multihop flows

• Placement Study to explore mesh deployment factors

• Random suitable for one-tier small scale but not large scale

Future Work

• Management of TFA Network

• Traffic Management with QoS

• Dynamic rate allocation scheme to instill fairness

• Capacity Planning

• From Coverage Limited to Capacity Limited: The evolution of a mesh network

• Security, DDoS

• Mesh traffic force single point of failure, security is critical

• Rice TAPs/WARP Platform Collaboration

Questions...?

Joseph Campcamp@rice.edu

www.ece.rice.edu/~camp

Rice Networks Groupwww.ece.rice.edu/networks

Web Traffic

• Expect load to saturate at 30 users, where the first link becomes saturated

• 80 users - similar spatial bias to fully backlogged case

• Web traffic acts as singly active flows on small-scale of time

• On average, only one flow active up to 25 users

10 20 30 40 50 60 70 800

500

1000

1500

2000

2500

Number of Users Per Hop

Down

load

Thr

ough

put (

kbps

)

1st Hop Node2nd Hop Node3rd Hop Node4th Hop Node

Web-emulation script (C) - Two minute trials5 to 80 constant users on nodes B through E

Download 30 kB webpage, 7 second “think” time, exponentially distributedAssume gateway node is not the bottleneck