TechnischeUniversität Berlin URL: www.tkn.tu-berlin.de

Predicting the Perceptual Service QualityUsing a Trace of VoIP Packets

C. Hoene, S. Wiethölter, A. WoliszSeptember 29th, 2004.

Qofis’04, Barcelona, Spain

Content

Introduction/BackgroundOur ApproachPlayout SchedulingListening-only Tests

Delay SpikesNon-Random Packet Losses

Conclusions

VoIP transmission of a telephone call

How to Judge the Quality of a Call?

Measure the quality of a VoIP system:Gather packet loss rates, mean delay, etc.

Easy but inaccurate

In the end the service quality is importantHuman based listening tests are extensive

ITU P.862 (PESQ algorithm) measures speech qualityCompares original sample with the transmitted versioncalculates Mean Option Score (MOS) (1=bad, 5=excellent)

ITU G.107 (E-Model) predicts quality of tel.-systemConsiders echo, loudness, coding, packet loss rate, delay, …Result: R Factor (0=bad, 70=toll quality, 100=excellent)

Limits of PESQ and E-Model

PESQ and E-Model cannot judge the impact of:Impact of playout scheduling cannot be predicted.Variable packet delay

adaptive playout timeNon-random packet losses Not applicable to assess VoIP packet traces resulting from simulations or experiments

Predicting Playout Scheduling

A VoIP phone can implement any playout scheduler. How to know which is going to be used? Actually, you cannot know it.But you know the most common, published playout scheduler schemes, which are

Fixed-deadline and adaptive from Van Jacobsen, Mills, Schulzrinne, Ramjee, and Moon

Calculate packets' playout times and the mean transmission delay.

Consider only speech frames during voice activity because during silence a human cannot identify the delay.

We can predict the behavior of most phones !

Assessing a Trace of VoIP Packets

Our solution:Using the most common different playout schedulers and encoding schemescombine of E-Model and PESQ (MOS to R formula approved by the ITU)

PESQ calculate speech qualityE-Model combinesMOS rating and transmission delays

Speech Quality and Transmission Delay

Speech Recordings

SpeechCoding

(coding distortion)

Experimentor Simulation

(loss or delay)

Decoder PLC

PESQReferenceDegraded

Audio

AudioMOS

Playout Buffer

(loss or scheduling)

E-ModelIe Result:Id R Factor

Degraded Audio

Mean Delay

Equations – referring to the SPECTS’04 paper…

Content

Introduction/BackgroundOur ApproachPlayout SchedulingListening-only Tests

Delay SpikesNon-Random Packet Losses

Conclusions

PESQ not verified conduct listening tests

Can PESQ measure playout rescheduling?or non-random packet losses?

PESQ can measure variable playout but its performance has never been verified.It has been tested only for random/burstybut not for non-random losses.

Conduct formal listening-only tests!

Test Sample Design

Construct packet traceswith delay spikes!Delay spikes with a height of 50 to 300ms and a width of height+10%.1 Spike in one sample

What to do?1. Drop late packets2. Or delay playout?

Playout Decisions

Original

a) Packet loss b) Positive adaptation c) Negative adaptation

1 2 2+ 2++ 5 6 1 2 2+ 2++ 3 4 5 6 1 2 5 6

Legend:1...6 Frame seq. no. 2+;2++ Concealed frame

1 2 3 4 5 6

1. DROP: Drop late packets (a)2. ADAPT: Delay playout to late packets (b)3. ADAPT&FALLBACK: Delay playout (b) and return to

normal during the next silence period (c)

Listening-Only Tests

Follow ITU P.830 butMultiple person at the same timeUse native and foreign speakers Use linear MOS scale (no discrete scale!)Sample Sequence: For each sentence

1. Original at the beginning, 2. MNRU with 5dB (worst),15,25,35 (best)3. Randomly distributed: packet loss samples

Produced on CD containing all the samples

Studio...

Listening-Test and Sample DesignDifferent encoding schemes:

G.711, G729

Produced delay spikes (only during voice activity)with length 50,100,150,200,250,300ms with width=length*1.1

Different samples with different length 5-10s), German (using Kiel Corpus Vol. I)220 samples (including MNRU samples as a reference)

Coding Loss Generator

DecodingPLC

Listening-only Tests

Frame Analysis PESQ

reference

speechproperties

importancePESQ MOS

X

seed

Speech Recordings R

(Correlation)

pack

etiz

atio

n

loss

rate

rate

/mod

e

algo

rithm

sam

ple

spea

ker

lang

uage

MOS

Rating Performance

Three parts: intro, first and second half. First is best.Native speaker and foreign speaker rate as good.Some persons did not kept track with the sample no.

Human LQS-MOS vs. PESQ LQO-MOS

y = 0,9403x - 0,2139R= 0,8316

1

2

3

4

1 2 3 4

MOS (Humans)

MO

S (P

ESQ

)

MNRU vs. MOS: Correlation: R=0.977

1

2

3

4

5 15 25 35 45

MNRU

MO

S

MOS PESQ MOS MOS cited scaled MOS

Results for G.711 coding

MOS Variance vs. Prediction Performance

trend line: R=0.768

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1,0

0 0,2 0,4 0,6 0,8 1PESQ sample variance for different impairment patterns

Cor

rela

tion

betw

een

PESQ

and

hum

ans

Content

Introduction/BackgroundOur ApproachPlayout SchedulingListening-only Tests

Delay SpikesNon-Random Packet Losses

Conclusions

Introduction

Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range,depending on

the frame speech propertiesthe encoder/decoder/concealment algorithmsdecoders resynchronization time after loss (especially low-rate decoders might maintain a wrong state after loss for the following frames.)the surrounding speech.

Example: Discontinuous Transmission (DTX)Speech frames during silence are less importantLower frame rate during silence

Packet Importance

The packet’s importance is the quality degradation that its loss would cause.

Definition: The importance of frame losses is the difference between the speech quality due to coding loss and the quality due to coding loss and frame losses, times the length of the analyzed sample.

Conducted some million PESQ ratings Dropped only one VoIP packet in a sample

Just try it: www.tkn.tu-berlin.de/research/mongoliaSelect predefined parameter sets

chose sample

Compression?

Listen to it!

amount of loss

Choose an another loss pattern

Drop only frames with an importance between

min. and max.

Talking or silence? Voiced or unvoiced?

Judge the speech quality by your self!

Sample statistics

Conclusion

Presented an approach to assess VoIP packet tracesCombined PESQ, E-Model and playout scheduling because

Speech frames differ in their importancePlayout scheduler are not standardizedPlayout rescheduling harms speech quality.

Verified PESQ for non-random packet losses (R=0.94) and delay spikes (R=0.87).

Precision performances depend on the variance of the samples’ speech quality.

Assessing VoIP: SummaryOpen-source software including

Implementation of all common playout schedulersG.711 and G.729 codingSample database including human rating results

Verified software in various research projects, e.g. voice over WLAN, impact of handover, ad-hoc.http://www.tkn.tu-berlin.de/reseach/qofis

C. Hoene, S. Wiethölter, and A. Wolisz, "Predicting the Perceptual Service Quality Using a Trace of VoIP Packets", In Proceedings of QofIS’04, Barcelona, Spain, September 2004.C. Hoene, H. Karl, and A. Wolisz, "A Perceptual Quality Model for Adaptive VoIP Applications", In Proceedings of International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS'04), San Jose, California, USA, July 2004, Paper won the Best Paper Award of the conference. C. Hoene and E. Dulamsuren-Lalla, "Predicting Performance of PESQ in Case of Single Frame Losses", In Proc. MESAQIN 2004, Prague, CZ, June 2004.S. Möller and C. Hoene, "Information About a New Method For Deriving the Transmission Rating Factor R From MOS in Closed Form", ITU, May 2002, Temporary Document for the study group 12.