August 3-4, 2004 San Jose, CA Troubleshooting Echo in VoIP Network Deployments Brad Steinka Director – IP Technology Telica,inc

August 3-4, 2004 • San Jose, CA • www.voipdeveloper.com

Troubleshooting Echo in VoIP Network Deployments

Brad Steinka

Director – IP Technology

Telica,inc.


Agenda

• Introduction– Sources of echo– Echo cancellation basics

• Troubleshooting echo problems– EC instrumentation– Diagnostics and statistics

• Customer and Lab experience


Introduction


Sources of Echo: Hybrid Echo

PSTN

Four-wireTrunk

Two-WireSubscriberLine


Tx

Tx

Rx

Rx

S1 S2

S1’s voice

Echo of S1

PBXPBX

Hybrid Transformer

• Hybrid transformers convert four-wire trunk interfaces into two-wire local subscriber lines.

• This conversion process results in a fraction of the four-wire rx signal to be reflected back on the four-wire tx signal causing Hybrid Echo.

• Subscriber S1 hears a delayed and attenuated version of his original voice.• Output impedance of the Hybrid needs to be balanced with the input

impedance of the terminating device. Mismatch in impedance leads to louder echo. Properly balanced, returned echo should be around 20dB.

Rx

Tx


Sources of Echo: Acoustic Echo

earpiece

microphone

echo

echo

Poor isolation between the earpiece and the microphone can result in acoustic echo


Network Topologies

PSTN Phone

IP Network IP Phone

Voice GW

PSTNMG

HybridEcho

IP Phone User hears EchoCaused by Hybrid echo in PSTN network

PSTN Phone

IP Network

Voice GW

PSTN MG IAD

PSTN Phone

Internet AccessDevice

MGMG

IP NetworkPSTN

PSTN Phone

PSTNPSTN Phone

HybridEcho

VoIP Packet Tandem Network

VoIPAccessNetworks


Round Trip Delay

PSTNPBX

Four-wireTrunk



Tx

Tx

Rx

RxS1 S2

S1’s voice

Echo of S1

• Round trip delay is the time it takes a word to go from S1 across the network to the echo source and then back.

• Echo that arrive after short delays, 25-35 ms, is not noticeable and is masked by the phone’s side tone.

• As round-trip delay increases and sufficient echo amplitude exits, echo becomes subjectively more annoying.

Round Trip Delay

Tx

Hybrid Transformer


VoIP impact on Echo

VOIPPSTN PSTNMGMG

Long round trip delay,

no echo sourcesEcho sources

• VOIP round trip network delay is 70+ ms in length depending on codec (711,723,729) selected, packetization time, and jitter buffer.

• Increased delay causes both subscribers to hear echo introduced in the PSTN networks.

• Delivering carrier grade voice quality becomes the responsibility of the Media Gateways and must stop echo from reaching the VOIP network.


Removing Echo

VOIP PSTN

MG

PSTNPhone

Echo source

• Echo Canceller (EC) is allocated by the MG to cancel the echo returned from the echo source in the PSTN network.

• Residual echo remaining after echo cancellation should NOT be perceptible to the human ear.

EC

Near-endSpeech from PSTN phone

Near-endspeech fromPSTN phone

Far-endspeech

Near-end speech plus

residual echo


EC Operation

Hybrid

NLP

Echo estimation Function

Rin Rout

Sout

PSTNTailCircuit

Sub() e(t)

VoIP

Sin

• EC predicts what the return echo will be once voice signal X(t) enters the PSTN tail circuit and becomes the echo Y(t).

• The Echo prediction is subtracted from the real echo and the resulting signal is ideally zero. EC is continuously adapting its estimate based on error estimate e(t) until it has good estimate for Y(t). This is the convergence time of the EC and is required to be < 200ms.

• The Non Linear Processor (NLP) replaces any residual echo with comfort noise provided no near-end voice is detected from PSTN phone.

X(t)

Y(t)

Returned echo

Inbound speech

PSTN


ECAN Terminology

Hybrid

NLP


Rin Rout

Sout

PSTNTailCircuit

Sub() e(t)

SinReturned echo

Input speech

PSTN

VoIP ERL

• Echo Return Loss (ERL) is level of attenuation provided by the PSTN network and Hybrid. Given an input speech signal at X dB, the returned echo at interface Sin of the echo canceller is (X-ERL) dB.

• Echo Path Delay (EPD) is the delay from the Rout port to the Sin port due to the delays inherent in the echo path transmission facilities through the PSTN tail circuit.

• After EC convergence, changes in EPD require the echo canceller to re-converge on the new echo path delay.

EPD


ECAN Terminology

Hybrid

NLP


Rin Rout

Sout

PSTNTailCircuit

Sub() e(t)

SinReturned echo

Input speech

PSTN

VoIP

ERLE

• Echo Return Loss Enhancement (ERLE) refers to the additional echo loss obtained through the operation of the echo canceller. An echo canceller is not a perfect device, and the best it can do is attenuate the level of the returning echo. ERLE is a measure of this echo attenuation through the echo canceller.

• ACOM is simply the total echo return loss seen across the Rin and Sout terminals of the echo canceller, and is the sum ERL + ERLE + NLP.

ACOM


Troubleshooting echo problems


Identify the problem

• Which subscriber on the call hears echo?• Does the subscriber hear echo all the time? Only to

certain phone numbers?• What type of phone is being used? POTS or VoIP

(SIP/MGCP). • If POTS, is it connected to an Internet Access Device?• What number was being dialed (local / long distance)?• Was the remote party on a speaker phone?

– Maybe acoustic echo, remove the speaker phone

• Whenever someone hears echo, the source of the echo is on the remote end of the call


Echo Canceller Instrumentation

• Per call information from ECs used in analyzing echo problems– ERL – Echo Return Loss– ERLE – Echo Return Loss Enhancement– Echo Path Delay (ms)– Number of times echo path changed during the life of the call.– Rout signal level (dB)– Sin signal level (dB)

Hybrid

NLP


Rin Rout

Sout

PSTNTailCircuit

Sub() e(t)

Returned echo

Input speech

PSTN

VoIP

ERLESin

ERLEPD


Echo Canceller Parameters

Hybrid

NLP


Rin Rout

Sout

PSTN

Sub() e(t)

PSTN

VoIP

Sin

Echo Tail Length

• Output gain control enables the EC to attenuate or boost the signal level AFTER the echo canceller has ‘seen’ the original output signal.• Input Gain control enables the EC to attenuate or boost the signal level BEFORE the echo canceller ‘sees’ the echo.• ERL can be enhanced through changes in Output / Input gain control.• Echo Tail Length (ETL) is the amount of time the Echo Canceller will ‘Remember’ a signal that has been output. ETL must cover the Echo Path Delay (EPD)

Output Gain

Input Gain

ERL


Measuring Echo Return Loss (ERL)

Hybrid

PSTN

PSTN

VoIP ERL

IP PhoneEC

Rin Rout

SinSout

MG

TestTone1004 hz@ -10dB

• Over an established phone call, a test tone is injected at a deterministic level, ex. 1004 hz @ -10dB

• Analyze the EC statistics– Echo Return Loss (ERL) < 6dB? Insufficient ERL.

– Changes in Echo Path Delay (EPD)? EC is periodically reconverging

– EPD > Echo Tail Length. Insufficient coverage to cancel echo.


Customer and Lab Experience


No Echo Cancellation

PSTN IP Network

Voice GW

MGEPD

• Problem Reported– Customer reported hearing strong and continuous echo.

• Analysis– Customer upgraded his service offering to include VoIP access. – Customer didn’t account for the additional delay introduced by VoIP.– Local Area trunk groups were not configured for Echo cancellation.– In the past, no need for echo cancellation coverage for local-local calls. Round-trip delay < 30ms.

• Resolution– Provisioned Echo Cancellation on local trunk groups for calls originating / destined for VoIP devices.

Hybrid

VoIP Phone

PSTN Phone


PSTN

No Echo Cancellation: Low ERL

IP Network

Voice GW

MGERL T1

ChannelBank

T1 interface (4-wire)

Analog(2-wire)

Echo Source• Problem Reported

– Customer reported hearing strong and continuous echo

• Analysis– Injected 1004 hz test tone @ -10db, ERL from channel bank was 3dB.– EC was unable to differentiate the returned echo from the PSTN near end speech.– Increasing ERL was required to achieve G.168 minimum of 6dB.

• Resolution– Increased the output gain on the EC for this subscriber by 4dB to achieve an effective ERL of 7dB to meet G.168.

VoIP Phone

PSTN Phone


PSTN

Mid-Call Echo – EPD Changes

IP Network

Voice GW

MGEPD T1

ChannelBank

T1 interface (4-wire)

Analog(2-wire)

Echo Source• LAB Experience

– Random echo during the call.

• Analysis– Changes in Echo Path Delay was causing the EC to reconverge and as a result echo was leaking to the VoIP subscriber.– Identified that the T1 interface on the channel bank was set for internal timing and NOT network / loop timing.– This caused T1 frame slips which caused the EC to detect changes in the echo path.

• Resolution– Changed T1 interface timing configuration for network / loop timing.– All equipment must be timed from a common reference to prevent frame slips / errors otherwise EC will detect changes in echo path delay.

VoIP Phone

PSTN Phone


PSTN

Mid-Call Echo – IAD Impact

IP Network

Voice GW

MG

Echo Source• Problem Reported– Customer reported random echo during the call.

• Analysis– Captured voice conversation using Etherreal.– Analyzed Rin signal from IAD, severe clipping noted by cool Edit, 0dB transmit levels.– Rin signal @ 0dB was causing non-linear behavior in the Hybrid.– Returned echo signal level > PSTN near end speech, so EC was fooled into thinking echo was PSTN speech and didn’t cancel echo.– IAD was configured for 0db of attenuation to PSTN tx/rx signals

• Resolution– Added 6dB of attenuation to the PSTN rx signal to the IAD (EC Rin) signal.– Resulted in an additional 20dB of attenuation of the returned echo (ACOM).

PSTN Phone

IAD

PSTN Phone Etherreal


Summary

• EC vendors must provide necessary instrumentation to assist in troubleshooting.

• MG vendors must make this information available for EC troubleshooting.

• Add EC statistics to call detail records to assist in post mortem analysis of escalated issues.

• Remember that these EC problems already existed prior to introduction of VoIP. They are now noticeable due to increase in network round trip delay.

• When subscribers report Echo problems, majority of the time the problem is at the far end.


Thank You

Brad SteinkaDirector, IP TechnologyEmail: [email protected]: 508-804-8154

Documents

August 3-4, 2004 San Jose, CA Troubleshooting Echo in VoIP Network Deployments Brad Steinka Director – IP Technology Telica,inc