22-Sept-2005

Google Summer of Code Projects:Lightweight Precision Timestamps

Jeff Boote

2005-Sept-22

Summer of Code 2005

•Program funded by Google ($2M)

•Supported students to work on open-source projects ($4500 per student)

•400 students were selected

•Each student was required to have a mentoring organization

•40 mentoring organizations

•Internet2 mentored 10 students

2005-Sept-22

Internet2 mentored projects

•Mostly transport-related things (http://transport.internet2.edu/)

•Two projects to characterize internal delays

•Three projects to work on a transport tool

•Two projects to work on thrulay

•One project to build a Gaim plug-in

•8 success, 2 failures

2005-Sept-22

Lightweight Precision Timestamps

•Implement a lightweight timestamp API

•Basically gettimeofday() without a system call

•2 projects successful

2005-Sept-22

Uses

•For Internet2 mostly network delay measurements

•Owamp•Thrulay•Bulk-transport protocol (delay based congestion control)

•Wide applicability (code profiling, etc…)

2005-Sept-22

OWAMP (the problem)

•One-way network latency measurements• Control protocol• Test protocol• Sample implementation

•Find problems in the network• Congestion usually happens in one direction first…

• Routing (asymmetric, or just changes)• SNMP polling intervals mask high queue levels that active probes can show

2005-Sept-22

Sample Implementation

Applications•owampd daemon•owping client

Built upon protocol abstraction library•Supports one-off applications•Allows authentication/policy hooks to be incorporated

2005-Sept-22

Functionality (owping client)

•owping client requests OWD tests from an OWAMP server

•Client can be sender or reciever•Communication can be “open”, “authenticated”, or “encrypted”

•Supports the setup of many tests concurrently

•Supports the buffering of results on the server for later retrieval

2005-Sept-22

Functionality (owampd)

owampd•Accepts requests for OWD tests•Responds with accepted/denied•Tests are formally started with a StartSessions message from the client.

•Runs tests•Sessions with packets received at the server are buffered for later retrieval

2005-Sept-22

Architecture

2005-Sept-22

Systemic Application Errors

Our tests indicate a systemic error of 73 usec *• Experiments with two systems connected via cross-over

cable• Two concurrent sessions (send,recv)• 10 packets/second

• Intel SCB2 motherboard

• 2x512 MB ECC registered RAM

• Intel PRO/100+ integrated NIC

• FreeBSD 4.6* 95% confidence level (RFC 2679)

* Error is specific to this OS/hardware/intensity level

* Tests with older version of owamp, should be a little better now.

2005-Sept-22

Calibration Distribution

2005-Sept-22

Characterization of Error

•Owamp (Min delay was 56 usec)•Timestamp fetching: 14.34 usec•Serialization: 10.24 usec•Unaccounted: 31.42 usec

•Ping (RTT 57 usec)•Timestamp Fetching: 5.07 usec•Serialization: 20.48 usec•Unaccounted: 31.45 usec

2005-Sept-22

Precision Related Context Switches

2005-Sept-22

Problems with “system” time for latency measurement

•Poor clock resolution• (typically 1-10 msec)

•Large amount of time to actually fetch timestamp

• Some systems better than others

•Large jitter• System-call - context switch threat

•No error estimates

•Drift estimate only happens at boot time for some systems

• Cold hardware

2005-Sept-22

Project Definition

Use clock register (TSC) along with system-wide state to incorporate drift adjustment and conversion to UTC without system calls.

2005-Sept-22

Implementation

Daemon•Treat “system” clock as an upstream NTP server

•Synchronize TSC values to “system” clock–Saves transformation information in shared memory so all local process have access to identical transformation

–Does not use locks - implemented with a rotating buffer with an index to the “current” set of coefficients (index updated with “atomic” operation)

2005-Sept-22

Implementation

Client•Read TSC values directly•Read transformation information from shared memory

2005-Sept-22

Validation of Implementation

See websites

2005-Sept-22

Features

•No system calls

•Clock resolution related to processor speed

•Potential to provide error estimates

•Quick convergence

•Spike supressor to deal with system time jitter

•PLL/FLL state machine based on NTP algorithms

2005-Sept-22

Future Enhancements to support OWAMP

•Add interaction with NTP to determine error of system clock

•Expose error of TSC synchronization as well as NTP

2005-Sept-22

Project Pages

Fasttime (Alex Holkner)

http://fasttime.sourceforge.net/

TSC-xluo (Xun Luo)

http://tsc-xluo.sourceforge.net

(Both sites site references for related work)

www.internet2.edu