Digital Audio Signal Processing DASPhomes.esat.kuleuven.be/~dspuser/dasp/material/Slides_2017_2018... · 1 Speech & Audio Processing / Part-I Digital Audio Signal Processing DASP

1

Speech & Audio Processing / Part-II

Digital Audio Signal Processing DASP

Marc Moonen Dept. E.E./ESAT-STADIUS, KU Leuven

[email protected] homes.esat.kuleuven.be/~moonen/

Digital Audio Signal Processing: Introduction Version 2017-2018 Lecture-1: Introduction 2 / 40

Overview

•  Aims/scope •  Case study: Hearing instruments •  Overview •  Prerequisites •  Lectures/course material/literature •  Exercise sessions/project •  Exam

2


Aims/Scope

Aim is 2-fold :

•  Speech & audio per se S & A industry in Belgium/Europe/…

•  Develop basic signal processing tools/principles Spatial filter design Adaptive filter algorithms (e.g. filtered-X LMS,..) Kalman filters Time-frequency analysis/processing Etc. …which are also used in many other application fields


•  Course covers a number of (‘typical’) DASP topics, e.g. •  Noise reduction •  Microphone array processing/beamforming •  Dereverberation •  Acoustic feedback cancellation •  Active noise control

•  All of these (*) appear in the design of modern digital hearing instruments

•  Hence very relevant DASP case…

(*) and many more…: •  Dynamic range compression •  Auditory scene analysis •  Etc.

Case Study: Hearing Instruments 1/20

= remarkable/spectacular

= remarkable/spectacular

3


Hearing •  Outer ear/middle ear/inner ear •  Tonotopy of inner ear: spatial arrangement of where sounds of

different frequency are processed

= Cochlea

Low-freq tone

High-freq tone

Neural activity for low-freq tone

Neural activitity for high-freq tone

© w

ww

.cm

.be




[Source: Lapperre]

Hearing loss types: •  Conductive (~outer/middle ear) •  Sensorineural (~inner ear) •  Mixed

One in six adults (Europe) suffers from hearing loss …and still increasing Typical causes:

•  Aging •  Exposure to loud sounds •  …

4


1921

20

07 (O

ticon

)

Case Study: Hearing Instruments 4/20 èHearing Aids (HAs) •  Audio input/audio output (`microphone-processing-loudspeaker’) •  ‘Amplifier’, but so much more than an amplifier!! •  History:

•  Horns/trumpets/… •  `Desktop’ HAs (1900) •  Wearable HAs (1930) •  Digital HAs (1980)

•  State-of-the-art: Digital HAs with.. •  MHz’s clock speed •  Millions of arithmetic operations/sec, … •  Multiple microphones


Ale

ssan

dro

Volta

174

5-18

27

© C

ochl

ear L

td


Electrical stimulation for low frequency

Electrical stimulation for high frequency

èCochlear Implants (CIs)

•  Audio input/electrode stimulation output •  Preprocessing similar to HAs + stimulation strategy +

•  History: •  Volta’s experiment… •  First implants (1960) •  Commercial CIs (1970-1980) •  Digital CIs (1980)

•  State-of-the-art: •  MHz’s clock speed, Mops/sec, … •  Multiple microphones

èOther: Bone anchored HAs, middle ear implants, …

Intra-cochlear electrode

5


èCochlear Implants (CIs)

•  External Processor Analog/digital-conversion Digital processing & filterbank Etc.. •  Coil Inductive/magnetic coupling •  Implant Electrode array

PS: Number of CI-implantees worldwide approx. 200.000 PS: 3 Main companies (Cochlear LtD, Med-El, Advanced Bionics)

© C

ochl

ear L

td



DASP Challenges in hearing instruments (next slides) •  Dynamic range compression •  Noise reduction •  Dereverberation •  Acoustic feedback cancellation •  Active noise control •  Etc.

Technology Challenges in hearing instruments •  Small form factor (cfr. user acceptance) •  Low power: 1…5mW (cfr. battery lifetime ≈ 1 week) •  Low processing delay: 10msec (cfr. synchronization with lip

reading)


6


DASP Challenges: Dynamic range compression Dynamic range & audibility Normal hearing Hearing impaired subjects subjects


Level

100dB

0dB


DASP Challenges: Dynamic range compression Dynamic range & audibility Need `signal dependent amplification’


Level

100dB

0dB Input Level (dB)

Out

put L

evel

(dB

)

0dB 100dB

0dB

100dB

Design: multiband DRC, attack time, release time, …

7


Why multiband DRC? …requires analysis/synthesis filter bank



•  However: Audibility does not imply intelligibility

•  Hearing impaired subjects need 5..10dB larger signal-to-noise ratio (SNR) for speech understanding in noisy environments

•  Need for noise reduction (=speech enhancement) algorithms: •  State-of-the-art: monaural 2-microphone adaptive noise reduction •  Near future: binaural noise reduction (see below) •  Not-so-near future: multi-node noise reduction (see below)


SNR

20dB

0dB 30 50 70 90

Hearing loss (dB, 3-freq-average)

8


DASP Challenges: Noise reduction & beamforming Multimicrophone ‘beamforming’, typically with 2 microphones

e.g. ‘directional’ front microphone and ‘omnidirectional’ back microphone


“filter-and-sum” the

microphone signals


Binaural hearing based on binaural auditory cues •  ITD (interaural time difference) •  ILD (interaural level difference)

•  Binaural cues (ITD: f < 1500Hz, ILD: f > 2000Hz) used for

•  Sound localization •  Noise reduction =`Binaural unmasking’ (‘cocktail party’ effect) 0-5dB


ITD

ILD signal

9


Binaural hearing aids •  Two hearing aids (L&R) with wireless link & cooperation •  Opportunities:

•  More signals (e.g. 2*2 microphones) •  Better sensor spacing (17cm i.o. 1cm)

•  Constraints: power/bandwith/delay of wireless link •  ..10kBit/s: coordinate program settings, parameters,… •  ..300kBits/s: exchange 1 or more (compressed) audio signals

•  Challenges: •  Improved localization through cue preservation •  Improved noise reduction + benefit from binaural unmasking •  Signal selection/filtering, audio coding, synchronisation, …



Future: Multi-node noise reduction in sensor networks/Internet-of-Things


10


DASP Challenges: Dereverberation •  Reverb = filtering effect (‘echo-ing’) of acoustic channel in

between speaker and microphone(s)

•  Reverb has an impact on speech understanding

•  Dereverberation = undo filtering by acoustic channel (e.g. ‘inverse filtering’)



DASP Challenges: Acoustic feedback cancellation •  Problem statement: Loudspeaker signal is fed back into microphone,

then amplified and played back again •  Closed loop system may become unstable (howling) •  Similar to feedback problem in public address systems (for the

musicians amongst you)


Model

F

-

Similar to echo cancellation in GSM handsets, Skype,… but more difficult due to signal correlation

11


DASP Challenges: Active noise control (ANC) ANC (see p.27) to counteract noise leakage & occlusion effect, exploiting additional ‘internal’ reference microphone


© S

.Doc

lo/U

niv.

Old

enbu

rg



DASP Challenges: …piecing things together

12


Overview : Lecture-2

Noise Reduction-I Single-channel noise reduction

`microphone_signal[k] = speech[k] + noise[k]’

•  Spectral subtraction methods (spectral filtering) •  Iterative methods based on speech modeling (Wiener & Kalman Filters) Applications: smartphones, conferencing, hearing aids, …



Noise Reduction-II Microphone Array Processing/Fixed Beamforming Referred to as ‘spatial filtering’ (similar to ‘spectral filtering’)

Applications: smartphones, conferencing, hearing aids, …

),(1 θωY

),(2 θωY

),(1 θωY

)(1 ωF

)(2 ωF

)(ωmF),( θωmY

)(ωMF),( θωMY

md),(1 θωY

)(ωS

θ

Σ),( θωZ

θcosmd

Filter-and-sum beamformer

13



Noise Reduction-III Adaptive Beamforming/Multi-channel noise reduction

•  Adaptive Beamforming

–  Known (fixed) speaker position –  Unknown (time-varying) noise field

•  Multi-channel noise reduction –  Wiener filtering approach : spectral+spatial filtering

Applications: smartphones, conferencing, hearing aids, …



Reverberation & Dereverberation ` microphone_signal[k] = filter*speech[k] (+ noise[k]) ’

•  Reverb = effect of acoustic channel in between speaker and microphone(s)

•  Reverb has an impact on coding, speech recognition, etc.

•  Single-microphone de-reverberation –  Cepstrum techniques

•  Multi-microphone de-reverberation: –  Estimation of acoustic impulse responses –  Inverse-filtering method –  Matched filtering

14


Acoustic Echo- and Feedback Cancellation Adaptive filtering problem: •  Non-stationary/wideband/… speech signals •  Non-stationary/long/… acoustic channels

Adaptive filtering algorithms AEC Control AEC Post-processing Stereo AEC



Overview : Lecture-6 Acoustic Echo- en Feedback Cancellation •  Hearing aids, public address (PA) systems •  Correlation between filter input (`x ’) and near-end signal (‘ n ’) •  Fixes : noise injection, pitch shifting, notch filtering, …

amplifier

15



Active Noise Control •  Solution based on `filtered-X LMS’ •  Application : active headsets/ear defenders


Guest Lecture: Dr. Enzo De Sena, University of Surrey, UK

‘Sound Field Recording and Reproduction’


16


Lectures

Lectures: 1 Intro (counts as half) + 7 Lectures PS: Time budget = 7.5*(2hrs)*4 = 60 hrs

Course Material: Slides

–  Use version 2017-2018 ! –  Download from DASP webpage

homes.esat.kuleuven.be/~dspuser/dasp/


Prerequisites

•  Signals & Systems (JVDW) •  Digital Signal Processing (I) (PW) signal transforms, sampling, multi-rate, DFT, …

• DSP-CIS (MM) filter design, filter banks, optimal & adaptive filters

17


Literature

Literature (general) (available in DSP-CIS library) •  Simon Haykin Àdaptive Filter Theory’ (Prentice Hall 1996) •  P.P. Vaidyanathan `Multirate Systems and Filter Banks’ (Prentice Hall 1993) Literature (specialized) (available in DSP-CIS library) •  S.L. Gay & J. Benesty Àcoustic Signal Processing for Telecommunication’ (Kluwer 2000) •  M. Kahrs & K. Brandenburg (Eds) Àpplications of Digital Signal Processing to Audio and Acoustics’ (Kluwer1998) •  B. Gold & N. Morgan `Speech and Audio Signal Processing’ (Wiley 2000)


Exercise Sessions/Project Acoustic source localization

–  Direction-of-arrival estimation –  Noise reduction –  Synthesis –  Simulated set-up

Direction-of-arrival θ

18


PS: groups of 2

Acoustic Source Localization Project

•  Runs over 4 weeks (non-consecutive) •  Each week

–  1 PC/Matlab session (supervised, 2.5hrs) –  2 ‘Homework’ sesions (unsupervised, 2*2.5hrs)

PS: Time budget = 4*(2.5hrs+5hrs) = 30 hrs •  ‘Deliverables’ after week 2 & 4 •  Grading: based on deliverables, evaluated during sessions

•  TAs: guiliano.bernardi@esat (English+Italian)

randall.ali@esat (English+Spanish)


Work Plan –  Week 1: Matlab acoustic simulation environment –  Week 2: Direction-of-arrival (DoA) estimation based on the

‘MUSIC’ algorithm *deliverable*

–  Week 3: DoA estimation + noise reduction (‘DOA informed beamforming’)

–  Week 4: Binaural synthesis and 3D audio *deliverable*

Acoustic Source Localization Project

..be there !

19


•  Oral exam, with preparation time •  Open book •  Grading

7 for question-1 7 for question-2 +6 for project ___ = 20

Exam


•  Oral exam, with preparation time •  Open book •  Grading

7 for question-1 7 for question-2 +6 for question-3 (related to project work) ___ = 20

September Retake Exam

20


Website

1)  TOLEDO 2)  http://homes.esat.kuleuven.be/~dspuser/dasp/ •  Contact: guiliano.bernardi@esat •  Slides (use `version 2017-2018’ !!) •  Schedule •  DSP-library •  FAQs (send questions to marc.moonen@esat)


Questions?

1)  Ask teaching assistant (during exercises sessions)

2)  E-mail questions to teaching assistant or marc.moonen@esat 3) Make appointment marc.moonen@esat ESAT Room B.00.14

Documents

Digital Audio Signal Processing DASPhomes.esat.kuleuven.be/~dspuser/dasp/material/Slides_2017_2018... · 1 Speech & Audio Processing / Part-I Digital Audio Signal Processing DASP