Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show

Development of an Automated Language Interpreting Tool

for Hispanic Mental Health Patients

The Way Forward:Advancing Mental Health Equity in California

March 30, 2017

Behavioral Health Center for Excellence

Colaborators

• Steven Chan• Alberto Odor• Alvaro Gonzalez• Andrés Sciolla• Glen Xiong• Michelle Burke Parish• Shang Wei• Ana-Maria Iosif• Peter Yellowlees (PI)

Support / Grants

• Blue Shield of California• Virtual Collaborative

Health Network• Behavioral Health Center

for Excellence• AHRQ – 2 R01 Grants


Research• 2006 – 2020

• Transcultural Telepsychiatry Across Language Barriers (AHRQ – R01)

• 2013 – 2020• Automated Translation System for Transcultural Telepsychiatry (AHRQ - RO1)

• 2017 –• Automated Facial Expression Recognition in Telepsychiatry (Pilot)

• 2017 –• Augmented Reality to Show Visual and Auditory Hallucinations to Non-Psychotics

• Need to buy some FTE


The Problem

4.7%of the population(over 5 years of age)had limited English-speaking abilities.

This is nearly the size of Illinois.


The Problem

79%16%

3%2%

21%

English only English OK + Other

English "not well" + Other English "not at all" + Other

2010 U.S. CensusLanguage Use


The Problem

Sacramento County languages

90+non-English languages, like

SpanishMiaoChineseVietnameseTagalog

RussianUkrainianHindiGermanHmong


Old Practice

Patient

Provider

Ad hoc


Current PracticePatient

Provider


Current Practice

Patient

Provider

Live interpreter


Current Practice Problem:

•Unfunded mandate.

•Additional $30 - $50/hour

•Most private insurers do not reimburse interpretation. Except in

California. (Wilson 2013, Guntzel 2012)

•35% difference in interpreter salary

Money, Time Quality


Current Practice Problem

Money, Time, Quality

• Increased encounter time by 57% (live vs phone interpreters) (Locatis 2010).

•Travel time delays (Masland 2010)

•Perceived time constraints (Diamond 2009)


Current Practice Problem

Money, Time, Quality

• Health systems determine what’s “reasonable” for quality of language interpretation (The Joint Commission, 2010)

• Hospitals often don’t test staff competency, utilization, or effectiveness of interpreting services(Tschurtz et al., 2011)

• Only 100 of 3,500 medical interpreters in Oregon have proper qualifications (Foden-Vencil, 2014)


Current Solutions

Live interpretersPhone interpretersVideoconferencing interpreters

• Little information about accuracy of live interpretation

• Individual variabilities

• Hypothesis: Machine translation has similar levels of accuracy in a clinical setting.


Ultimate Goal

• Analyze language as data.

• Create new clinical processes for in-person

care and telemedicine.

• Create a better standard of care with

improved interpreting, more efficiently, and

less expensively.


Asynchronous S&F Telepsychiatry Workflow

History (10 years ago)

Visalia

Tulare

History (10 years ago)


Current State of Automated Translation


SDKs Assessment

Se

rvic

e

Ac

ce

ss

Wo

rks

wit

h a

ny

us

er

wit

ho

ut

pre

tra

inin

g

HIP

AA

Tra

ns

cri

pti

on

Tra

ns

lati

on

Ha

nd

les

str

ea

min

ga

ud

io

Un

de

rsta

nd

s

me

dic

al

term

ino

log

y

Co

ns

um

er

ap

p

ava

ila

ble

to

de

mo

?

Google Translate SaaS API ✓ – – ✓ ✓ ? ✓ ? Android, iOS

Nuance online SaaS API ✓

Nuance Dragon NaturallySpeaking SDK

Client DSClocal software –

✓—

local✓ – – ✓

Nuance Dragon NaturallySpeaking SDK

Server DSSWindows local ✓ ✓— ✓ – ✓ – ?

Nuance Dragon Dictate local software –✓—

local✓ ✓

Nuance 360 SaaS API ✓ ✓ – ✓ ✓ third-party apps

Nuance NDEV SaaS API ✓ – ✓ – ✓ Android, iOS

Mac OS X

Client & Apple-

operated

servers✓ – ✓ – – ✓

Microsoft Bing Voice Recognition &

Microsoft Translator on AzureSaaS API ✓ ✓ ✓ ✓ ✓

Windows Phone, web

translation

IBM BlueMix SaaS API –✓ English

only✓ ✓ – web translation

SRIlocal & client-

server✓ ✓ ✓ ✓ ✓ –

SysTran 7 Business Translatorlocal, server, &

SaaS✓ ✓ – ✓ N/A ✓ iPhone, web


Transcripts

Target

language (HUMAN)

Original

language

Target

language (COMPUTER)


Error Types

omission, in which the interpreter (human or machine) did not interpret a word/phrase uttered;

addition, in which the interpreter added a word/phrase not uttered by the patient;

substitution, in which the interpreter substituted a word/phrase for a different word/phrase; and

false fluency, in which the interpreter used an incorrect word/phrase that does not exist in that particular language


Errors in a PTSD Transcript

Inte

rpre

ter

1

Aggregate counts Transl #Data types J L K M NGrand Totaltotal_added_word_count 2 11 1 0 1 15total_drastic_word_count 214 225 277 189 265 1170total_unusual_word_count 25 12 21 33 31 122total_word_count 1202 1303 1181 1189 1253 6128Grand Total 1443 1551 1480 1411 1550 7435

Aggregate counts Transl #Data types J L K M NGrand Totaltotal_added_word_count 4 3 0 6 2 15total_drastic_word_count 244 343 398 278 477 1740total_unusual_word_count 19 68 74 96 71 328total_word_count 1214 1305 1186 1192 1261 6158Grand Total 1481 1719 1658 1572 1811 8241

Inte

rpre

ter

2


Automated Evaluation of Languages

BLEU (Bilingual Evaluation Understudy), an error-scoring algorithm that has found to have high correlation with human error quality with a score between 0 to 1.

Higher scores indicate a higher degree of similarity between human-translated and computer-translated texts.

The BLEU formula is the most widely used automated metric for testing MT apps.


Implications

• Language apps can be incorporated in both asynchronous and synchronous telemedicine video encounters and workflows.

• Not all translation apps are alike: significant differences in the quality between apps.

• These apps are already being used in commercial industries to replace or augment language interpreters. Expect increased reliability and accuracy in the future.

Thank you

[email protected]

Documents

Development of an Automated Language Interpreting Tool for ... · Language Interpreting Tool for Hispanic Mental Health Patients The Way Forward: ... •Augmented Reality to Show