Upload
britton-tyler
View
215
Download
1
Embed Size (px)
Citation preview
COULD THE INTERNET HANG UP ON US? A LONGITUDINAL GLOBAL STUDY OF INTERNET PACKET AND DATA ARRIVAL PREDICTABILITYUlrich Speidel, 'Etuate Cocker, Firas Ghazzi, Nevil
BrownleeDepartment of Computer Science, The University of Auckland
Ulrich Speidel – [email protected]
AGENDA
VoIP and other real-time protocols Causes and implications of variability in packet
arrival Observing long-term trends in packet arrival
variability with a beacon network Why the Pacific Islands are a great test bed –
and why other parts of the world are important, too
Preliminary observations Measures of arrival quality: jitter and entropy
Ulrich Speidel – [email protected]
CONTRIBUTORS TO DATE Niko Rebenich, Stephen Neville, Aaron
Gulliver, University of Victoria, B.C., Canada Kashif Nisar, Zeeshan Aziz, Suhaidi Hassan,
Universiti Utara Malaysia Ming-Chui Dong, Vincent Wong, University of
Macau Raimund Eimann, Zurich, Switzerland (in a
private capacity) Bernhard Platter, Bernhard Ager, ETHZ
Zürich Hendrik Ferreira et al., University of
Johannesburg, South Africa AJ Han Vinck, University of Duisburg-Essen,
Germany Hirosuke Yamamoto, University of Tokyo Hidetoshi Yokoo, Gunma University Hiroyoshi Morita, University of Electro-
Communication ITS, University of Auckland Muriel Médard, Jason Cloud, Massachusetts
Institute of Technology Randy Bush (in a private capacity)
Ministry of Lands, Survey, and Natural Resources, Tonga
Connect ISP, Fiji Ministry of Infrastructure and Planning, Cook
Islands Oyster ISP (Telecom Cook Islands), Rarotonga,
Cook Islands Ministry of Fisheries, Government of Kiribati IT Department, Government of Tuvalu E.M. Jones Ltd., Tonga Ministry of Revenue, Tonga Solomon Telekom, Solomon Islands
Ulrich Speidel – [email protected]
INDUSTRY CONTRIBUTORS TO DATE
Ministry of Lands, Survey, and Natural Resources, Tonga
Connect ISP, Fiji Ministry of Infrastructure and Planning, Cook Islands Oyster ISP (Telecom Cook Islands), Rarotonga, Cook
Islands Ministry of Fisheries, Government of Kiribati IT Department, Government of Tuvalu E.M. Jones Ltd., Tonga Ministry of Revenue, Tonga Solomon Telekom, Solomon Islands
Ulrich Speidel – [email protected]
WHY ARE REAL-TIME PROTOCOLS IMPORTANT?
Voice over IP (VoIP) telephony / videoconferencing / remote call centre industry
Telemedicine (e.g. remote surgery / diagnostics / consultation)
Remote control in industrial applications
Internet of Things
Financial applications (rapid trading)
Ulrich Speidel – [email protected]
VOICE OVER IP AND OTHER REAL-TIME PROTOCOLS
blah
A/D Encoder
Internet
Decoder
Buffer
D/A
Packets (~50 per second)
blah
• analog voice signal is digitally encoded
• encoded byte stream is chopped into a packet train
• packets are transmitted via Internet to receiver
• receiver buffers packets to establishconstant-rate dataflow to decoder
• decoder recovers analog signal and replays it
Transmitter
Receiver
Ulrich Speidel – [email protected]
VOICE OVER IP AND OTHER REAL-TIME PROTOCOLS
Delay between signal generation and replay is undesirable – must be kept as small as possible
Replay delay is composed of three components:1. Physical latency of cables or satellite links along
packet paths (fixed for a given path)2. Time spent in router queues (variable)3. Size of buffer at receiver (governed by variation in
arrival times)
Ulrich Speidel – [email protected]
IRREGULAR PACKET ARRIVAL - CONSEQUENCES
Need to buffer more Real-time services such as VoIP, video
conferencing, and remote manipulation may become impractical
Downloads and streaming video/audio would still work
Significant economic impact Effects are most likely to be felt between
remote endpoints first
Ulrich Speidel – [email protected]
CAUSES OF VARIABILITY IN PACKET ARRIVAL
Length and number of router queues Grow with increase in traffic and network
mesh
Overflowing router queues cause packet drops
Load balancing at routers Could this cause problems?
Ulrich Speidel – [email protected]
LOAD BALANCING ROUTERS
Load balancing routers send packets to the same destination across different links
If packets from the same stream are load balanced, it causes them to take different paths and experience different latencies
In some cases, packets may overtake each other (out-of-order arrivals)
Ulrich Speidel – [email protected]
PACIFIC RIM SUBMARINE CABLES
Which way do packetstravel between Asia and NZ?
What are the possiblelatencies?
http://www.submarinecablemap.com/
Ulrich Speidel – [email protected]
GO FIGURE!>tracert 161.64.45.10
Tracing route to 161.64.45.10 over a maximum of 30 hops
1 1 ms 3 ms 3 ms tmk-net-207-gw.net.auckland.ac.nz [130.216.207.254] 2 2 ms 1 ms 1 ms cx-alpha-sxj-700.net.auckland.ac.nz [172.18.0.6] 3 3 ms <1 ms <1 ms dxj-260-1-to-cx-alpha.net.auckland.ac.nz [172.18.0.26] 4 2 ms 2 ms 2 ms br1-to-cx-alpha.net.auckland.ac.nz [130.216.246.41] 5 1 ms 1 ms 1 ms 203.167.201.41 6 4 ms 5 ms 5 ms unknown.telstraglobal.net [134.159.174.41] 7 29 ms 27 ms 31 ms i-0-0-1-1.sydo-core02.bx.telstraglobal.net [202.84.143.134] 8 29 ms 27 ms 27 ms i-0-3-2-0.sydo-core01.bi.telstraglobal.net [202.84.220.190] 9 212 ms 215 ms 215 ms i-0-5-2-0.hkth-core01.bx.telstraglobal.net [202.84.143.22] 10 141 ms 142 ms 141 ms i-3-0-0.hkth01.bi.telstraglobal.net [202.84.153.233] 11 155 ms 155 ms 155 ms tata-peer.hkth01.pr.telstraglobal.net [134.159.128.6] 12 144 ms 144 ms 144 ms 116.0.67.70 13 * 150 ms * z0l161.static.ctm.net [202.175.0.161] 14 160 ms 161 ms 160 ms cr1-191.macau.ctm.net [202.175.26.241] 15 161 ms 160 ms 160 ms ctm-int-umac.macau.ctm.net [202.175.0.58] 16 * * * Request timed out. …
Trace complete.
Traceroute to our beacon at Universiti Utara Malaysia:
>tracert 103.5.183.14
Tracing route to 103.5.183.14 over a maximum of 30 hops
1 4 ms 1 ms 1 ms tmk-net-207-gw.net.auckland.ac.nz [130.216.207.254] 2 2 ms 1 ms 2 ms cx-alpha-sxj-700.net.auckland.ac.nz [172.18.0.6] 3 <1 ms <1 ms <1 ms dxj-260-1-to-cx-alpha.net.auckland.ac.nz [172.18.0.26] 4 1 ms 1 ms <1 ms br1-to-cx-alpha.net.auckland.ac.nz [130.216.246.41] 5 1 ms 2 ms 1 ms 203.167.201.41 6 154 ms 166 ms 155 ms te7-3.ccr01.sjc05.atlas.cogentco.com [38.122.92.105] 7 155 ms 155 ms 155 ms te0-2-0-4.ccr21.sjc01.atlas.cogentco.com [154.54.84.53] 8 133 ms 137 ms 135 ms be2162.mpd21.lax01.atlas.cogentco.com [154.54.27.173] 9 133 ms 133 ms 135 ms be2021.ccr21.lax04.atlas.cogentco.com [154.54.87.250] 10 133 ms 132 ms 144 ms 38.104.210.82 11 239 ms 239 ms 240 ms ae-2.cr-gw-2-sin-pip.sg.globaltransit.net [124.158.224.17] 12 238 ms 238 ms 238 ms 124.158.224.91 13 239 ms 240 ms 248 ms 124.158.226.150 14 248 ms 247 ms 248 ms ae-1.cr-02-glsfb.ni.time.net.my [223.28.2.121] 15 244 ms 245 ms 245 ms xe-0-0-0-0.er-01-glsfb.ni.time.net.my [223.28.2.2] 16 245 ms 244 ms 247 ms xe-0-1.es-01-glsfb.ni.time.net.my [223.28.16.130] 17 345 ms 345 ms 345 ms 203.121.112.18 18 345 ms 344 ms 345 ms 211.25.227.212 …
Trace complete.
Traceroute to our beacon at the University of Macau:
via Sydney and Hong Kong
via San Jose (California) and Los Angeles
…and back across the Pacific!
Ulrich Speidel – [email protected]
WHICH EFFECT WILL WIN?
Technological progress or the forces of chaos it unleashes?
extra bandwidth new links
more powerfulrouters
more efficient codecs
more users
more traffic
more congestion
more malware
less predictable packetarrivals
more inefficient routing
Ulrich Speidel – [email protected]
CAN WE PREDICT A TREND?
Basic idea: Pick a sample application, e.g., VoIP Reality check: Applications don't stay stable, so need one
whose behaviour we can control
Repeatedly use the application over time between remote end points where deterioration is likely to be seen early
See whether you observe any long-term changes
Do this for multiple pairs of endpoints so you can ascertain whether a change is specific to a particular endpoint or pair of endpoints
Ulrich Speidel – [email protected]
HOW? THE BEACON NETWORK
Basic idea:
"Run a global network of beacon computers that act pair-wise as endpoints"
To date (December 2013), 28 beacons have been installed in Canada, Cook Islands, Fiji, Germany, Japan, Kiribati, Macau, Malaysia, New Zealand, Solomon Islands, South Africa, Switzerland, Tonga, Tuvalu, and the United States.
…and the network is growing! Additional beacons in Hawaii and Canada are under construction
Ulrich Speidel – [email protected]
THE BEACON NETWORK
Ulrich Speidel – [email protected]
BEACON SOFTWARE
The beacon software is a simple C program that runs one experiment, writes it logs and then shuts down
Scheduled via cron jobs & configured via command line parameters
Writes plain text logs
Requires sufficient privileges to run raw sockets (to determine TTL of received packets)
Ulrich Speidel – [email protected]
BEACON HARDWARE
PC or Mac platform with Ubuntu/BSD/OSX
Doesn't need to be overly fast – we use 600 MHz ALIX boards among others
Must have real-time clock, but GPS / atomic clock synchronisation is not required
Virtual machines are not a good idea – tend to spend too much time away from task at hand
Ulrich Speidel – [email protected]
BEACON UDP EXPERIMENTS
Beacon exchanges 10,000 UDP packets of 110 bytes with a partner beacon
Packets transmitted every 20 ms
Packets are timestamped and serial-numbered
At receiving end, packets are logged with arrival time stamp, serial number, arrival sequence number, and TTL observed
This experiment typically runs 3 times a day between selected beacon pairs
Ulrich Speidel – [email protected]
BEACON TCP EXPERIMENTS
Two beacons exchange a predefined set of data via TCP
Two modes available: VoIP (constant payload rate) and Download (maximum payload rate)
Receiving end logs data arrival at application layer plus TTL and size of packets involved in communication (from raw socket)
Ulrich Speidel – [email protected]
WHY HAVE BEACONS ALL OVER THE WORLD?
Want long-term global trend, not just local effects
Want a "developed world" baseline but also see what it is like in remote places on the fringe – many of our beacons run in the Pacific for that reason (need I mention Africa?)
Long paths generally are of interest – both in terms of latency and number of hops
A lot of international traffic passes through "hub regions" (North America, Europe, SE Asia). What effect do these regions have on traffic that passes through them?
Last but not least: We're looking for input (and your own experiments)!
Ulrich Speidel – [email protected]
BEACON CHALLENGES
Proper timing without proper timebase
Precise timing on multitasking computers
Getting two beacons to communicate (firewalls, ports, policies/politics)
Scheduling
Data volume backhaul (each beacon contributes ~3MB / experiment / day)
Data processing & presentation (currently UDP only)
Coordination with many partners, each with a unique network setup
Making it longitudinal
Ulrich Speidel – [email protected]
INITIAL OBSERVATIONS: PACKET LOSS
Ulrich Speidel – [email protected]
INITIAL OBSERVATIONS: LOSS VS OUT-OF-ORDER
Ulrich Speidel – [email protected]
PACKET TRAIN QUALITY
Subjective approaches, e.g., Mean Opinion Score (MOS) – reliably replicable only with very large sample
Objective approaches, e.g., jitter measurements.
But: jitter can be random (=problem) or predictable (=less of a problem)
How can we tell the difference?
Ulrich Speidel – [email protected]
NON-RANDOM JITTER
Consider the following simplified scenario:
Assume uncongested routers but differences in latency, blue router load balances
Have two paths/latencies -> high jitter, but not really queue-induced
Jitter ignores systematic patterns in arrivals
A good entropy estimator (i.e., not Shannon) can detect the difference
Ulrich Speidel – [email protected]
COMPLEMENT: ENTROPY (RATE)
Entropy: "measure of disorder in a system". Usually given in the form of an entropy rate rather than absolute entropy
Entropy rate: number of bits of actual information required to represent an object divided by number of bits actually used to represent it.
Dimensionless ("bits/bit")
Examples: Shannon entropy (well known). Better: Lempel-Ziv compression ratios, T-entropy, …
Predictable arrivals or arrival patterns: low entropy
"Really unpredictable arrivals" = high entropy
Ulrich Speidel – [email protected]
ENTROPY HOW-TO
Map inter-arrival times of successive UDP packets to symbol bins, e.g.: t < 17 ms: "A" 17 ms < t < 19 ms: "B" 19 ms < t < 21 ms: "C"
… Form string from these symbols: "CCCBDCACF…"
Determine entropy rate for string (e.g., as Lempel-Ziv compression ratio or T-entropy)
"Perfect" string will be "CCCCCCC…" – highly compressible, low entropy
Chaotic arrivals generate many new pattern combinations: harder to compress, higher entropy
Ulrich Speidel – [email protected]
ENTROPY VS. JITTER
Ulrich Speidel – [email protected]
ENTROPY VS. JITTER
0.001 0.01 0.1 1 10 1000
0.2
0.4
0.6
0.8
1
1.2
1.4
Jitter [s]
T-e
ntr
opy 5
-bin
[bit
s/s
ym
bol]
"Direction of worry"
Ulrich Speidel – [email protected]
WHERE IS ENTROPY INTRODUCED?
Ulrich Speidel – [email protected]
PICKING A TREND?
Ulrich Speidel – [email protected]
CHANGING TTLS – PROOF OF PATH INSTABILITY
Ulrich Speidel – [email protected]
WHAT ABOUT TCP?
Not ideal for real-time applications due to handshake delays
Still very widely used for real-time apps as UDP is often blocked at firewalls (because it's historically been used infrequently)
E.g., Skype fails over to TCP if UDP can't get through
Ulrich Speidel – [email protected]
TCP IN STREAMING APPLICATION
Time
Data[bytes]
Cumulative amount of bytes received
Bytes needed for continuous rate immediate replay
Bytes needed for continuous rate buffered replay (no underruns)
Actual replay with "boredom wheel"
minimum buffer period
minimum buffersize
Replay buffer underruns
Goal: Avoid buffer underruns with minimum buffer period / size
Ulrich Speidel – [email protected]
TCP MINIMUM BUFFER TIME TRENDS?
Ulrich Speidel – [email protected]
CONCLUSIONS
Real-time traffic and best-effort protocols are uneasy companions
We propose to use a large beacon network to measure arrival shape of packet trains
Our beacons already see interesting effects
Effects are strongly path-specific and sometimes not easily explained
Jitter and entropy are both useful for arrival shape evaluation
A lot of work remains to be done!
Ask us if you're interested in hosting a beacon
https://iibex.auckland.ac.nz (https://130.216.197.6)
Questions ?
?
? ?
??
?
?
?
https://iibex.auckland.ac.nz