D1 - 21/04/23

The present document contains information that remains the property of France Telecom. The recipient’s acceptance of this document implies his or her acknowledgement of the confidential nature of its contents and his or her obligation not to reproduce, transmit to a third party, disclose or use for commercial purposes any of its contents whatsoever without France Telecom’s prior written agreement.

confidential

Comparison of Narrowband and Wideband

Speech codecs in noisy environnement

Noel Chateau noel.chateau@francetelecom.com, Laetitia Gros laetitia.gros@francetelecom.com,

Catherine Quinquis catherine.quinquis@francetelecom.com, Jean Yves Monfort

jeanyves.monfort@francetelecom.com

Workshop on Wideband Speech Quality in Terminals and Networks: Assessment and

Prediction

8th and 9th June 2004 - Mainz, Germany

France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D2 - 21/04/23

confidential

Content

1. 3Gpp/VoIP context

2. Methodology

3. Description of Experiment

4. Results

5. Conclusion

France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D3 - 21/04/23

confidential

3GPP/VoIP Context

s Rel 99 : March 2000Q only narrow band speech and Circuit switched communications

s Rel4: March 2001QPacket switched for download QSelection of wide band speech codec

s Rel5: March 2002QPacket switched conversationnal communicationsQnarrow band and wide band codecs

s Rel6: September 2004QCharacterisation of default codecs for Packet switched conversationnal communications

France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D4 - 21/04/23

confidential

3GPP/ Characterisation of default codecs for Packet switched conversationnal communications

s 1st phase QAMR codec in quiet and noisy environmentQAMRWB codec in quiet and noisy environment

s 2nd phaseQAMR and AMRWB versus legacy codecs in quiet environment

s Further experimentQAMR and AMRWB versus legacy codecs in noisy environment : car noise

France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D5 - 21/04/23

confidential

Methodology

s VoIP in UMTSQ delay is one major issue

–Interleaving due to radio environment–Buffering in decoder side to cope with jitter–IP transport–Packetisation in IP as well as on radio leg

QNot only quality of codecs but acceptability of service

s Conversation test QMore accurate to assess acceptability of service

France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D6 - 21/04/23

confidential

Experiment: Description

s Noisy environment : car noise @ 60dB PaQ Car noise is in one of the room, the other room is quiet

s Codecs under testQAMR (6.7 and 12.2 modes)QAMRWB (12.65 and 15.85 modes)QG.723.1@ 6.3 kb/sQG.729@ 8 kb/sQG.711@ 64 kb/sQG.722 @ 64 kb/s

s Environmental conditionsQIP packet loss : 0% or 3%

France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D7 - 21/04/23

confidential

Experiment: Description

PC 1 : VOIP Terminal

Simulator

Network Board A

PC 2 : VOIP Terminal

Simulator

Network Board B

PC 3 : Network Simulator

Hub 1 Hub 2

France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D8 - 21/04/23

confidential

Results on voice quality criterion s ANOVA

Qsignificant main effects: noise, PLP & codecQsignificant interactions: codec x PLP

s Planned comparisons QInteraction bandth x noise for PLP=0%: NSQInteraction Bandth x PLP for Noise: NS

s Tuckey tests show a superiority of some codecs on others for noisy and/or lossy conditions, but not systematic. Without noise and packet losses: no significant differences.

s The superiority of WB Codecs is higher for noisy and packet-loss conditions, but there is no systematic benefit of WB in noise.

Factor F

noise * F(1,30) = 20.23 - p<0.001

packet loss percentage (PLP) * F(1,30) = 13.61 - p<0.001

codec * F(7,210) = 14.26 - p<0.001

noise x PLP F(1,30) = 0.68 - p=0.42

noise x codec F(7,210) = 1.25 - p=0.28

codec x PLP* F(7,210) = 2.3 - p<0.05

noise x codec x PLP F(7,210) = 0.76 - p=0.62

Voice Quality

Ca r noise W ithout noise

Pa cke t loss = 0%

AM

R N

B 6.7 kbs

AM

R N

B 12.2 kbs

AM

R W

B 12.65 kbs

AM

R W

B 15.85 kbs

G.723.1 6.4 kbs

G.729 8 kbs

G.722 64 kbs +

plc

G.711 +

plc2 ,5

3,0

3,5

4,0

4,5

5,0

MO

S

P a cke t loss = 3%

AM

R N

B 6.7 kbs

AM

R N

B 12.2 kbs

AM

R W

B 12.65 kbs

AM

R W

B 15.85 kbs

G.723.1 6.4 kbs

G.729 8 kbs

G.722 64 kbs +

plc

G.711 +

plc

A N

O V

A

France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D9 - 21/04/23

confidential

Results on understanding criterion

Factor F

noise * F(1,30) = 68.11 - p<0.001

packet loss percentage (PLP) * F(1,30) = 11.07 - p<0.01

codec * F(7,210) = 20.3 - p<0.001

noise x PLP F(1,30) = 0.53 - p=0.47

noise x codec F(7,210) = 1.86 - p=0.08

codec x PLP* F(7,210) = 2.09 - p<0.05

noise x codec x PLP F(7,210) = 1.88 - p=0.07

s ANOVAQsignificant main effects: noise, PLP & codecQsignificant interactions: codec x PLP

s Planned comparisons QInteraction bandth x noise for PLP = 0%: Signif.QInteraction Bandth x PLP for Noise: NS

s Tuckey tests show an almost systematic superiority of WB and G.711 codecs on NB codecs (except AMR NB 12.2 kbs) for noisy and/or lossy conditions. Without noise and packet losses: no significant differences.

s Considered globally, WB and G.711 codecs better resist to noise for PLP=0%.

U nderstanding of the partner

Ca r noise W ithout noise

Pa cke t loss = 0%

AM

R N

B 6.7 kbs

AM

R N

B 12.2 kbs

AM

R W

B 12.65 kbs

AM

R W

B 15.85 kbs

G.723.1 6.4 kbs

G.729 8 kbs

G.722 64 kbs +

plc

G.711 +

plc2 ,5

3,0

3,5

4,0

4,5

5,0

MO

S

P a cke t loss = 3%

AM

R N

B 6.7 kbs

AM

R N

B 12.2 kbs

AM

R W

B 12.65 kbs

AM

R W

B 15.85 kbs

G.723.1 6.4 kbs

G.729 8 kbs

G.722 64 kbs +

plc

G.711 +

plc

A N

O V

A

France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D10 - 21/04/23

confidential

Results on interaction criterion

Factor F

noise * F(1,30) = 24.58 - p<0.001

packet loss percentage (PLP) * F(1,30) = 12.66 - p<0.001

codec * F(7,210) = 6.97 - p<0.001

noise x PLP F(1,30) = 0.3 - p=0.59

noise x codec F(7,210) = 2.67 - p<0.05

codec x PLP* F(7,210) = 0.63 - p=0.73

noise x codec x PLP F(7,210) = 0.96 - p=0.46

s ANOVAQsignificant main effects: noise, PLP & codec

Qsignificant interactions: codec x noise

s Planned comparisons QInteraction bandth x noise for PLP = 0%: Signif.

QInteraction Bandth x PLP for Noise: NS

s Tuckey tests show only a few significant differences for noisy conditions only.

s As for the understandy criterion, considered globally, WB codecs better

resist to noise for PLP=0%.

A N

O V

A

Interactivity

Ca r noise W ithout noise

Pa cke t loss = 0%

AM

R N

B 6.7 kbs

AM

R N

B 12.2 kbs

AM

R W

B 12.65 kbs

AM

R W

B 15.85 kbs

G.723.1 6.4 kbs

G.729 8 kbs

G.722 64 kbs +

plc

G.711 +

plc2 ,5

3,0

3,5

4,0

4,5

5,0

MO

S

Pa cke t loss = 3%

AM

R N

B 6.7 kbs

AM

R N

B 12.2 kbs

AM

R W

B 12.65 kbs

AM

R W

B 15.85 kbs

G.723.1 6.4 kbs

G.729 8 kbs

G.722 64 kbs +

plc

G.711 +

plc

France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D11 - 21/04/23

confidential

Results on default perception criterion

Factor F

noise * F(1,30) = 29.5 - p<0.001

packet loss percentage (PLP) * F(1,30) = 11.95 - p<0.01

codec * F(7,210) = 6.48 - p<0.001

noise x PLP F(1,30) = 0.21 - p=0.65

noise x codec F(7,210) = 1.67 - p=0.12

codec x PLP* F(7,210) = 2.47 - p<0.05

noise x codec x PLP F(7,210) = 0.82 - p=0.57

A N

O V

A

s ANOVAQsignificant main effects: noise, PLP & codec

Qsignificant interactions: codec x PLP

s Planned comparisons QInteraction bandth x noise for PLP = 0%: NS

QInteraction Bandth x PLP for Noise: NS

s Tuckey tests show no significant differences between codecs for any condition.

s The default perception criterion does not seem relevant to detect a possible advantage of WB codecs in noise.

D efault perception

Ca r noise W ithout noise

Pa cke t loss = 0%

AM

R N

B 6.7 kbs

AM

R N

B 12.2 kbs

AM

R W

B 12.65 kbs

AM

R W

B 15.85 kbs

G.723.1 6.4 kbs

G.729 8 kbs

G.722 64 kbs +

plc

G.711 +

plc2 ,0

2,5

3,0

3,5

4,0

4,5

5,0

MO

S

Pa cke t loss = 3%

AM

R N

B 6.7 kbs

AM

R N

B 12.2 kbs

AM

R W

B 12.65 kbs

AM

R W

B 15.85 kbs

G.723.1 6.4 kbs

G.729 8 kbs

G.722 64 kbs +

plc

G.711 +

plc

France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D12 - 21/04/23

confidential

Results on global quality criterion

Factor F

noise * F(1,30) = 57.37 - p<0.001

packet loss percentage (PLP) * F(1,30) = 14.51 - p<0.001

codec * F(7,210) = 13.54 - p<0.001

noise x PLP F(1,30) = 0.29 - p=0.59

noise x codec F(7,210) = 1.28 - p=0.26

codec x PLP* F(7,210) = 1.68 - p=0.11

noise x codec x PLP F(7,210) = 0.4 - p=0.9

A N

O V

A

s ANOVAQsignificant main effects: noise, PLP & codecQsignificant interactions: none

s Planned comparisons QInteraction bandth x noise for PLP = 0%: NSQInteraction Bandth x PLP for Noise: NS

s Tuckey tests show an almost systematic superiority of WB and G.711 codecs on NB codecs (except AMR NB 12.2 kbs) for noisy and/or lossy conditions. Without noise and packet losses: no significant differences.

s As for the voice-quality criterion, the superiority of WB Codecs is higher for noisy

and packet-loss conditions, but there is no systematic benefit of WB in noise.

Globa l qua lity of the com m unica tion

Ca r noise W ithout noise

Pa cke t loss = 0%

AM

R N

B 6.7 kbs

AM

R N

B 12.2 kbs

AM

R W

B 12.65 kbs

AM

R W

B 15.85 kbs

G.723.1 6.4 kbs

G.729 8 kbs

G.722 64 kbs +

plc

G.711 +

plc2 ,0

2,5

3,0

3,5

4,0

4,5

5,0

MO

S

Pa cke t loss = 3%

AM

R N

B 6.7 kbs

AM

R N

B 12.2 kbs

AM

R W

B 12.65 kbs

AM

R W

B 15.85 kbs

G.723.1 6.4 kbs

G.729 8 kbs

G.722 64 kbs +

plc

G.711 +

plc

France Telecom R&D Distribution of this document is subject to France Telecom’s authorization D13 - 21/04/23

confidential

Summary of subjective tests resultss ANOVA is not useful for determining if WB codecs are significantly better than NB

codecs in noisy environment but show that noise was the strongest experimental factor that affected the subjective data.

s Planned comparisons and Tuckey tests show that:Qthe MOS differences between WB and NB are significantly larger in noisy conditions than in silent conditions for no packet losses, for "Understanding" and "Interactivity" criteria.

Q the MOS differences between WB and NB are often but not systematically larger in noisy and/or 3% packet-loss conditions than in the silent and 0 % packet-loss condition, for the "Voice quality" and "Global quality" criteria.

Qthe differences between WB and NB is independant on the presence of noise and packet loss for the "Default perception" criterion.

s It can be concluded that without packet loss, in noise, the advantage of WB compared to NB is that it enhances the comprehension and the interaction with the other partner, but not voice and global quality, since these criteria are highly affected by the presence of noise.