Collaborators: Mark Coates, Rui Castro, Ryan King, Mike Rabbat, Yolanda Tsang, Vinay Ribeiro, Shri...

collaborators: Mark Coates, Rui Castro, Ryan King, Mike Rabbat, Yolanda Tsang, Vinay Ribeiro, Shri Sarvotham,Rolf Reidi

Network Bandwidth Estimation and Tomography

Rob Nowak & Rich Baraniuk UW-Madison Rice University

spin.rice.edu

• too complex to measure everywhere, all the time• traffic measurements expensive (hardware, bandwidth)

1969 1993

Internet Boom

companies do not share data or performance information

Proprietary Concerns

• bits are bundled into packets• packets go through routers• queues absorb bursts of packets• congestion: queues fill up, large delays, packet drops

Networking 101

Network Measurement & Inference

Internet equivalent model

Path Modeling and Bandwidth Estimation

Network Tomography

Brain Tomography

unknown object

statistical model

measurements

Maximumlikelihood estimate

maximizelikelihood

physics

data

prior knowledge MRF model

counting &projection

Poisson

unknown object

statistical model

measurements

Maximumlikelihood estimate

maximizelikelihood

physics

data

prior knowledge

Network Tomography

queuing behavior

routing &counting

binomial /multinomial

Network Tomography

From link-level traffic measurements, infer end-to-end traffic flow rates

Vardi ’96, Tebaldi & West ’98

Cao, Davis, Vander Wiel, Yu ’00

y = packet losses or delays measured at the edge

A = routing matrix (graph)

= packet loss probabilities or queuing delays for each link

= randomness inherent traffic measurements

),|(),( AypAl likelihood function

Ay

Network Tomography (MINC Project, Towsley-Duffield)

Probe packets experience similar queuing effects and may interact with each other

Probing the Network

probe =packet stripe

cross-traffic(2)packet (1)packet

delay

Network Tomography: The Basic Ideasender

receivers

Network Tomography: The Basic Ideasender

receivers

Maximum Likelihood Estimation via EM

)()1( ny

)()2( ny

Suppose we were able to measure losses/delays on each link

)(1 nx

)(2 nx)(3 nx

)(3 nx

Expectation-Maximization (EM) alternates between computing expectation of unobserved internal measurements and the desired estimates of link-by-link loss/delay distributions

Problem: How to computemaximum likelihood estimates of link-by-link loss/delay distributions from end-to-end measurements ?

0

1

2

53 4

Topology ID via Probe Interactions

35

d

Topology ID via Probe Interactions

0

1

2

53 4

34

d

35 we can infer that receivers 3 & 4 have a longer shared path than 3 & 5

Finding the Maximum Likelihood Tree

Stochastic search through “forest” via Metropolis-Hastings

True topology

estimated topology

Internet measurement experiments

UNO

What have we learned?

• Clever probing and sampling schemes reveal “hidden” network structure and behavior

• Simple inference algorithms are effective, intuitive, easy to implement, scale nicely

• MLE criteria are easily modified to include prior information: Bayesian or regularized MLE methods are straightforward

Complex interplay between measurement/probing techniques, statistical modeling, and computational methods for optimization

Open Problems: Placement/Coverage

How should measurement devices be deployed ?

Logical graph coverage of physical topology ?

Can random graph models shed some light ?

Open Problems: Spatio-temporal Correlation

competing traffic

Can we detect correlations? Can we exploit them in measurement and mapping applications?

Fuse tomography and bandwidth estimation

Long-range dependence of network traffic

Correlations due to competing traffic flows

Open Problems: Detection and Localization

Detecting and locating anomalous behavior rather than estimating everything

Estimation Hypothesis Testing

How can we capitalize on conventional wisdom: most links are ‘good’ and only a few are ‘bad’ ?

Open Problems: Timing and Synchronization

• Hardware solutions (expensive)• Software solutions (more practical)

- sophisticated software clocks (Veitch ’02)

- crude software clocks (ICMP timestamping) and statistical averaging

sender network

sendermonitor

receivermonitor

receiver

How to accurately measure time ?

Open Problems: Network Security

How can measurement and monitoring across the Internet help detect and prevent malicious activities ?