NIH (NIDCD) initiative to develop real-time portable signal processing tools for advancing research on hearing loss compensation

1

Unless otherwise indicated the information provided is based on the author’s personal analysis and opinions relative to the NIH (NIDCD*1) initiative

*1 The National Institute for Deafness and Other Communication Disorders

Roger Miller & Amy Donahue (2016 IHCON)

“These research tools will

• lower barriers for hardware and software development,

• and facilitate translation of these advances into widespread use with hearing aids, cochlear implants, and consumer electronics devices.“

.

2

Projects funded

Six awards were given starting July 2016, and one July 2017

3

RO1 AwardSmartphone‐Based Open Research Platform for Hearing Improvement Studies Issa Panahi, Nasser Kehtarnavaz, and Linda Thibodeau University of Texas, Dallas, USA

Android and iOS systems, running C shells to:

• Use the phone as a remote microphone

• Wired and wireless version

• Use the phone itself as a hearing aid

• Wired and wireless version

• 4

RO1 Award

A Real-time, Open, Portable, and Extensible Speech Lab

Hari Gurudadri, UC San Diego

•Initial MHA implementation on Os X and Linux Computers

•Portable platform based on Snapdragon 8016 and Debian Unix.

•Single microphone Receiver-in-the-canal BTE delivery system

•http://openspeechplatform.ucsd.edu/index.html 5

SBIR Award: Development of an Open Speech Signal Processing Platform Ray Weber & Ross Snider, Flat Earth, Inc, Bozeman, MT, USA

• Based on the SoC FPGAs which allows for a deterministic low-latency signal

processing

• Applications run on top of Linux, which is running on the dual ARM Cortex A9 CPU

• Applications interact with the FPGA fabric

• I/O daughter cards:

• analog and digitals mics; speaker driver; headphone driver; BT Transceiver; Wi-Fi

Transceiver, etc.

https://flatearthinc.com/open-speech-platform-0

6

SBIR Award: Open Hardware and Software System for Speech and Signal Processing

Rafael Delgado, Intelligent Hearing Systems, Miami, FL

7

• Users can interface directly to the DSP system hardware

using a Dynamically Linked Library (DLL) and Visual

Control Library (VCL).

• Languages that support dlls include most high level

programming languages such as C++, Delphi (Visual

Pascal) and Visual Basic. Other development packages

that also support the use of dlls include MatLab and

LabView.

http://ohsspds.ihsys.info

SBIR Award: Open Source System for Audio Processing

Odile Clavier, Creare Inc., Hanover, NH, USA

8

•1st generation hardware: Teensy 3.6 + custom audio interface board •Relies on Arduino software development environment

•Next generation (Rev D, this spring)): ARM M4

https://tympan.org

Open Portable Platform for Hearing Aid Research

Caslav (Chas) Pavlovic1, Hendrik Kayser2, Paul Maanen3, Tobias Herzke3, Volker Hohmann1,3, Prakash S.R.1,4, and Reza Kassayan11 BatAndCat Corporation 2 University of Oldenburg 3 HörTech gGmbH Oldenburg 4 EarLens Corporation

Research reported in this presentation was supported by the NIDCD of the NIH. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

The BTE Case Design supported by EarLens Corporation

We report on two closely related projects:RO1 Award R01DC015429 Volker Hohmann and Chas Pavlovic July 2016

SBIR Award R44DC016257 Chas Pavlovic July 2017

Hardware Software

Software: Open Master Hearing Aid (open MHA)

11

REQUIREMENTS

• end to end latency (fully loaded)< 10 ms • programmable in C++ • Linux, MacOS, Windows environment • hardware-independence

openMHA - current status

12

Hearing aid processing modules, release 4.8.0: • calibration • multi-band dynamic compressor • adaptive feedback cancellation • single-channel noise reduction • adaptive differential microphone • binaural coherence filter • binaural beamforming algorithms • sound source localization

Lin

calibrationF B

Rin

B F

ADMADM

coherence

DC

Lout Rout

13

Designed for different user groups

Clinicians who

• use “Plug and Play” software modules

• change processing parameters at high level

14

Designed for different user groups

Audiological Researchers and Application Engineers

• set up measurement tools and customize algorithms

• access configuration interface as a text file

nchannels_in = 2 fragsize = 64 srate = 44100

# MHA library name mhalib = transducers

# IO plugin library name iolib = MHAIOFile

mha.plugin_name = overlapadd

mha.calib_in.peaklevel = [90 90] mha.calib_out.peaklevel = [90 90]

mha.overlapadd.fftlen = 256 mha.overlapadd.wnd.len = 128 mha.overlapadd.plugin_name = mhachain

# list of plugins mha.overlapadd.mhachain.algos = [ ... fftfilterbank ... dc ... combinechannels ... ]

# Frequency bands mha.overlapadd.mhachain.fftfilterbank.f = [200 2000]

# gaintable data in dB gains mha.overlapadd.mhachain.dc.gtdata = [[10 -10 -30];[20 -25 -50];[10 -10 -30];[20 -25 -50]]

# input level for first gain entry in dB SPL mha.overlapadd.mhachain.dc.gtmin = [0]

# level step size in dB …

15

Plugin developers who can • Develop and implement new

plugins in the openMHA framework

• For example a new C++ routine for noise reduction

Designed for different user groups

openMHA – structure Basic framework • MHA host application • toolbox with plugins • communication interfaces (I/O) • Matlab/Octave GUI for audiological usage

Documentation • manuals for different usage scenarios • example configuration files for algorithms

16

openMHA

plugins IO

audio backend(Jack, File, TCP)

MHA hostapplication

control applications(e.g., Matlab, Octave)

Hardware platforms 1. Desktop setup - off the shelf; available now

A simple version demonstrated today: • Windows, Linux or MacOs • Headphones that provide a good sound

isolation (e.g. audiometric audio caps) • A lapel microphone

More complex versions may include: • Multichannel audio interface for multiple

microphones • Actual hearing aid hardware for audio

transmission (speakers; multiple microphones etc .)

17

Hardware platforms 2. Single board computer and own codec board (showed today)available now• also serves as a development

board for fully portable version• features three codecs with 6

input and 6 output channels • preamplifiers • supports 2 hearing aids (with

directional microphones) and one binaural headset

18

19

Hardware platforms 3. PORTABLE PLATFORM: Single board computer and own codec board shown today Available; May 2019

L BTE R BTE:

Front and Back Mic Receiver in the canal

Each BTE:

Headphone Mic and Speaker

20

3. Portable platform - May 2019

Hardware platforms

21

Binaural BTE Sound delivery systemAvailable Summer 2019

Hardware platforms

• High end ID and mechanical/acoustical design : Earlens case adapted for MHA purposes

• Front and back MEMS microphones • Differential amplifiers • 8 wire low-noise cable

Preliminary measurements

22

Battery Life: 6 hours

Equivalent Electrical Input Noise: MEMS mike: 26 dB SPLPower supply and circuit: 20 dB SPL (does not include the earpiece cable noise) Expected total (microphone, cable, circuit and power supply) under 28 dB SPL

openMHA software runs successfully with the following latencies: direct routing through JACK server (MHA):3.2 ms with MHA in time domain: 3.9ms with STFT and overlap-add: 5.2 ms with full processing load: under 10 ms

23

24

Libraries and a fitting example available Summer 2019

Device Control and Fitting System

• WiFi at the moment; BLE summer 2019

• Example gui: Earprint format

• Earprint: A self-fitting system (Pavlovic et al. US 8,112,166; now property of HIMPP)

Other related interests - that we may use MHA for and implement in the MHA libraries:

A. Fitting

25

Fitting Process Initial Fitting (formula):

Fine Tuning

Further Specific Environment Adjustments

Audiologist

Audiologist Self

Self

Self

Self Adjustment Difficulties

1. N dimensional space of parameters affecting the preference (N definitely more than 2).

2. Multiple local maxima and minima.

26

27

Device Control and Fitting System

• Each of the three levels require a different parameter space

• The space needs to be arranged so that there is a smooth transition from one cell to another

• Multidimensional EarPrint is possible and needed

All 3 levels: Initial, Fine Tuning, and Environment adjustment

Earprint: Used by Sound ID, SoundHawk, Starkey

28

Device Control and Fitting System

• EarPrit has the same accuracy as any of the other best psychophisical scaling procedures (CS, ME) or the direct speech recognition testing - but in 10% of the time (Ramani, Michael, Pavlovic, JASA, 2011)

• CS, ME, and the Objective Speech Recognition testing for the same accuracy take the same amount of time (Pair Comparisons efficiency decreases rapidly as the number of stimuli increases, Purdy and Pavlovic, 1992)

• FUTURE: Test time and accuracy of all this methods can be substantially increased by the use of DNN.

• It is highly likely that, as data accumulates, the fine tuning to different environments will become, with the assistance of the DNN, totally automatic. • A person walks into a bar • The Environment data sample and/or the location Geo tag

are sent to the Cloud • Adjustments calculated by a Cloud algorithm based on:

• person’s previous adjustments in similar environments • and/or adjustments done by individuals with similar

hearing profile, age, etc. • and/or adjustments typically done in this particular

environment (GPS environment tag)

29

DNN and automated fitting

Other related interest that we will further explore in the MHA B.Connectivity

• BT has become almost ubiquitous in hearing aids • current uses: phone calls; remote mics; TV, etc. • WiFi in larger wearables • possibility to connect to Cloud to access big data

storage and and super computer power

30

THE MERGER OF HEARIG AIDS AND WEARABLES:

• Wearables: devices that feature a number of functions; such as playing music, heart rate monitor, blood pressure, oxygen level, instant language translation, etc.

• Trend: hearing aids are getting many features of wearables and wearables are getting hearing aid functionality (i.e. become “hearables”)

31

Nuheara Bragi

Market Size US (annual sales):

• Hearing Aids: 8 M customers • Headphones: 400 M customers

32

Price

• Hearing aids: $2000 • Headphones: $200

33

What do the following facts suggest?

• Equally sophisticated HW • 50 times bigger headphone volume • 10 time smaller headphone price • and, legal backing for OTC sales at least in the

US

34

Over-the-Counter Hearing Aid Act of 2017

35

Plus the future:

• Wearables becoming smaller and smaller • Backed by huge volume and huge competition • Hearing aid will become a feature of

wearables

36

but “hearing aids should be:

Invisible or hide effectively as parts of desirable objects”

What is a desirable object?

What if it provides extraordinary hearing even for normal hearing people?

In this regard we have evolved a long way:

• 2004: SoundID Device: 1st hearable - low sales

• 2008: SoundID BT headset secret hearable: “pass-through mode” separately sold remote microphone solid sales

In this regard we have evolved a long way:• 2004: SoundID Device: 1st hearable - low sales

• 2008: SoundID BT headset secret hearable: “pass-through mode” separately sold remote microphone high sales

• 2017: super high sales

• High quality stereo earphone

• Connects to phone • Personal binaural sound

amplifier • 3 microphones on each side • Direct wireless link between

multiple devices for voice communications (“Grid”) 40

Bat&Cat headset

Connectivity:person to person

Proprietary and Confidential – BatAndCat Corporation © 2015

Grid - example

41

Scaling speech understandingResults (simulated and actual restaurant)

0 SNR

-10 SNR

Specially Distributed Microphone Grid (BatAndCat: in development):

43

The AI based technologies can utilize the combined signals from distributed microphones to virtually eliminate noise and focus on the target speaker

Spatially Distributed Microphones

44

• microphones and transmitters present already in ubiquitous audio capable devices (Amazon Echo, Apple HomePod, smart phones, future watch phones)

• low cost microphones-transmitters integrated in home automation devices (switches, thermostats, lamps)

• objects specifically designed for this purpose (remote microphones)

For more information:

http://www.openmha.org/https://github.com/HoerTech-gGmbH/openMHAhttps://github.com/HoerTech-gGmbH/Cape4allhttp://batandcat.com/nih-supported-projects.html

45