Quality and Performance Quality and Performance Evaluation of VoIP End-pointsEvaluation of VoIP End-points

Wenyu JiangHenning Schulzrinne

Columbia UniversityNYMAN 2002

September 3, 2002

MotivationsMotivations The quality of VoIP depends on both

the network and the end-points Extensive QoS literature on network

performance, e.g., IntServ, DiffServ Focus is on limiting network loss & delay

Little is known about the behavior of VoIP end-points

Performance Metrics for Performance Metrics for VoIP End-pointsVoIP End-points Mouth-to-ear (M2E) delay

C.f. network delay Clock skew

Whether it causes any voice glitches Amount of clock drift

Silence suppression behavior Whether the voice is clipped (depends much

on hangover time) Robustness to non-speech input, e.g., music

Robustness to packet loss Voice quality under packet loss

Measurement ApproachMeasurement Approach Capture both original and output audio Use “adelay” program to measure M2E delay Assume a LAN environment by default

Serve as a baseline of reference, or lower bound

ethernet

IP phonecoupler

InOut

notebookoriginalaudio

PCsignalstereo

LAN

speaker

ethernet

coupler IP phoneInOut

line in

(mouth)

(ear)

outputaudio

VoIP End-points TestedVoIP End-points Tested Hardware End-points

Cisco, 3Com and Pingtel IP phones Mediatrix 1-line SIP/PSTN Gateway

Software clients Microsoft Messenger, NetMeeting (Win2K,

WinXP) Net2Phone (NT, Win2K, Win98) Sipc/RAT (Solaris, Ultra-10)

Robust Audio Tool (RAT) from UCL as media client Operating parameters:

In most cases, codec is G.711 -law, packet interval is 20ms

Example M2E Delay PlotExample M2E Delay Plot 3Com to Cisco, shown with gaps > 1sec Delay adjustments correlate with gaps,

despite 3Com phone has no silence suppression

35

40

45

50

55

60

0 50 100 150 200 250 300 350

M2E

del

ay (m

s)

time (sec)

experiment 1-1experiment 1-2

silence gaps

Visual Illustration of M2E Visual Illustration of M2E Delay Drop, Snapshot #1Delay Drop, Snapshot #1

3Com to Cisco 1-1 case

Left/upper channel is original audio

Highlighted section shows M2E delay (59ms)

Snapshot #2Snapshot #2 M2E delay

drops to 49ms, at time of 4:16

Snapshot #3Snapshot #3 Presence

of a gap during the delay change

Effect of RTP M-bits on Effect of RTP M-bits on Delay AdjustmentsDelay Adjustments Cisco phone sends M-bits, whereas Pingtel

phone does not Presence of M-bits results in more adjustments

20

30

40

50

60

70

80

90

100

0 50 100 150 200 250 300

M2E

del

ay (m

s)

time (sec)

Cisco to 3Com 1-1Pingtel to 3Com 2-1

new talkspurt (M-bit=1)

Sender CharacteristicsSender Characteristics Certain senders may introduce delay

spikes, despite operating on a LAN

50

100

150

200

250

300

0 50 100 150 200 250 300

M2E

del

ay (m

s)

time (sec)

Mediatrix to 3Com 3-1Mediatrix to Cisco 1-1

Mediatrix to Pingtel 1-1

Average M2E Delays for IP Average M2E Delays for IP phones and sipcphones and sipc Averaging the M2E delay allows more compact

presentation of end-point behaviors Receiver (especially RAT) plays an important role

in M2E delay

0

50

100

150

200

250

3Com Cisco Mediatrix Pingtel RAT

Receiver

Ave

rag

e M

2E d

elay

(m

s)

Sender: 3Com Sender: Cisco

Average M2E Delays for Average M2E Delays for PC Software ClientsPC Software Clients Messenger 2000 wins the day

Its delay as receiver (68ms) is even lower than Messenger XP, on the same hardware

It also results in slightly lower delay as sender NetMeeting is a lot worse (> 400ms) Messenger’s delay performance is similar to or

better than a GSM mobile phone.

A B AB BA

MgrXP (pc) MgrXP (notebook) 109ms 120ms

Mgr2K (pc) 96.8ms

68.5ms

NM2K (pc) NM2K (notebook) 401ms 421ms

Mobile (GSM)

PSTN (local number)

115ms 109ms

Delay Behaviors for PC Delay Behaviors for PC Clients, contd.Clients, contd. Net2Phone’s delay is also high

~200-500ms V1.5 reduces PC->PSTN delay PC-to-PC calls have fairly high delays

A B AB BA

N2P v1.1 NT P-2 (pc2)

PSTN (local number)

292ms 372ms

N2P v1.5 NT P-2 (pc2)

201ms 373ms

N2P v1.5 W2K K7 (pc)

196ms 401ms

N2P v1.5 W2K K7 (pc)

N2P v1.5 W98 P-3(notebook2)

525ms 350ms

Effect of Clock Skew: Cisco Effect of Clock Skew: Cisco to 3Com, Experiment 1-1to 3Com, Experiment 1-1

Symptom of playout buffer underflow

Waveforms are dropped

Occurred at point of delay adjustment

Bugs in software?

Clock Skew RatesClock Skew Rates Mostly symmetric between two devices RAT (Ultra-10) has unusually high drift rates, > 300

ppm (parts per million) High clock skews confirmed in many (but not all) PCs

and workstations

Drift Rates (in ppm)

3Com Cisco Mediatrix

Pingtel RAT

3Com -8.3 55.4 43.3 41.2 -333

Cisco -55.2 -0.4 -11.8 -12.1 -381

Mediatrix -43.1 11.7 1.3 -0.8

Pingtel -40.9 12.7 2.8 -3.5 -380

RAT 343 403 376 12.3

Drift Rates for PC ClientsDrift Rates for PC Clients Drift Rates not always symmetric!

But appears to be consistent between Messenger 2K/XP and Net2Phone on the same PC

Existence of 2 clocking circuits in sound card?

A B AB BA

MgrXP (pc) MgrXP (notebook)

172 87.7

Mgr2K (pc) 165 85.6

NM2K (pc) NM2K (notebook)

? -33?

Net2Phone NT (pc2)

PSTN 290 -287

Net2Phone 2K (pc) 166 82

Mobile (GSM) 0 0

Packet Loss ConcealmentPacket Loss Concealment Common PLC methods

Silence substitution (worst) Packet repetition, with optional fading Extrapolation (one-sided) Interpolation (two-sided), best quality

Use deterministic bursty loss pattern 3/100 means 3 consecutive losses out of

every 100 packets Easier to locate packet losses Tested 1/100, 3/100, 1/20, 5/100, etc.

PLC BehaviorsPLC Behaviors Loss tolerance (at 20ms interval)

By measuring loss-induced gaps in output audio

3Com and Pingtel phones: 2 packet losses Cisco phone: 3 packet losses

Level of audio distortion by packet loss Inaudible at 1/100 for all 3 phones Inaudible at 3/100 and 1/20 for Cisco phone,

yet audible to very audible for the other two. Cisco phone is the most robust

Probably uses interpolation

Effect of PLC on DelayEffect of PLC on Delay No affirmative effect on M2E delay

E.g., sipc to Pingtel

50

55

60

65

70

75

80

0 10 20 30 40 50 60

mou

th-t

o-ea

r de

lay

(ms)

time (sec)

0/1003/100

1/20

Silence SuppressionSilence Suppression Why?

Saves bandwidth May reduce processing power (e.g., in

conferencing mixer) Facilitates per-talkspurt delay adjustment

Key parameters Silence detection threshold Hangover time, to delay silence

suppression and avoid end clipping of speech

Usually 200ms is long enough [Brady ’68]

Hangover TimeHangover Time Measured by feeding ON-OFF

waveforms and monitor RTP packets Cisco phone’s is the longest (2.3-2.36

sec), then Messenger (1.06-1.08 sec), then NetMeeting (0.56-0.58 sec)

A long hangover time is not necessarily bad, as it reduces voice clipping Indeed, no unnatural gaps are found Does waste a bit more bandwidth

Robustness of Silence Robustness of Silence Detectors to MusicDetectors to Music On-hold music is often used in customer

support centers Need to ensure music is played without any

interruption due to silence suppression Tested with a 2.5-min long soundtrack Messenger starts to generate many

unwanted gaps at input level of -24dB Cisco phone is more robust, but still

fails from input level of -41.4dB

Acoustic Echo CancellationAcoustic Echo Cancellation Important for hands-free/conferencing

(business) applications Primary metric: Echo Return Loss (ERL)

Measured by LAN-sniffing RTP packets Most IP phones support AEC

ERL depends slightly on input level and speaker-phone volume

Usually > 40 dB (good AEC performance)

IP Phone

3Com

Cisco

ipDialog

Pingtel

Snom-100

ERL (dB)

40-45

53- 49-54 33-42 -5 (no AEC)

M2E Delay under JitterM2E Delay under Jitter Delay properties under the LAN environment

serves as a baseline of reference When operating over the Internet:

Fixed portion of delay adds to M2E delay as a constant Variable portion (jitter) has a more complex effect

90100110120130140150160170180

0 20 40 60 80 100 120 140 160 180

mou

th-t

o-ea

r de

lay

(ms)

time (sec)

High jitter (uplink)Low jitter (downlink)

Initial test Used typical cable

modem delay traces Tested RAT to Cisco No audible distortion

due to late loss Added delay is normal

M2E Delay under Jitter, M2E Delay under Jitter, contd.contd.

Cisco phone generally within expectation Can follow network delay change timely

Takes longer (10-20sec) to adapt to decreasing delay Does not overshoot playout delay

More end-points to be examined

0

100

200

300

400

500

600

700

800

0 10 20 30 40 50 60 70 80

M2E

del

ay (m

s)

time (sec)

Tracetest1test2

0

20

40

60

80

100

120

140

160

0 10 20 30 40 50 60 70 80 90 100

M2E

del

ay (m

s)

time (sec)

Tracetest1test2

Artificial Trace Real Trace with Spikes

ConclusionsConclusions Average M2E Adelay:

Low (mostly < 80ms) for hardware IP phones Software clients: lowest for Messenger 2000 (68.5ms) Application (receiver) most vital in determining delay

Poor implementation easily undoes good network QoS Clock skew high on SW clients (RAT, Net2Phone) Packet loss concealment quality

Acceptable in all 3 IP phones tested, w. Cisco more robust

Silence detector behavior Long hangover time, works well for speech input But may falsely predict music as silence

Acoustic Echo Cancellation: good on most IP phones

Playout delay behavior: good based on initial tests

Future WorkFuture Work

Further tests with more end-points on how jitter influences M2E delay

Measure the sensitivity (threshold) of various silence detectors

Investigate the non-symmetric clock drift phenomena

Additional experiments as more brands of VoIP end-points become available