Treating Translation Quality Metrics as Business Intelligence

PanelistsSteve Richardson, LDS ChurchSamuel Läubli, AutoDeskHyunjoo Han, AutoDeskNancy Anderson, EMC

ModeratorsEduardo D’Antonio, VMwareClove Lynch, VMware

Key Questions

• What can we learn from sharing our data?• Specifically, what kind of business intelligence

could we get (that we don’t have today) from sharing this data?

• How is technology helping you measure it today?

• How could the TAUS DQF and Quality Dashboard contribute?

LDS Church

Translation at the Church of Jesus Christ of Latter-day Saints

• 15M+ members in 168 countries• Approximately 100M translated words produced annually • Materials in 90+ languages in dozens of domains are

translated on a regular basis• MT systems deployed for English to/from 39 languages• Materials in 100+ languages are available at LDS.ORG

Measuring Quality at the Church• Challenges:

– 1,000+ full-time & contracted translators around the world

– From paper to CAT tools w/high speed internet• Until now:

– Policies and human processes focused on individual translators’ work

– Tried SAE J2450; moved to simpler methods– Never been able to measure the overall effect on

quality of published materials

Measuring Quality at the Church:A current example

Measuring Quality at the Church:What we want to know

• What is the quality of our final output?• Does our focus on translators and the

processes we use equate to improved final quality, decreased cost, and timeliness?

• What is the cost/benefit ratio of the various steps in our translation/review processes?

Over the past two years, we have standardized on these systems and tools:

We have used DQF a number of times to measure MT quality, ranking, and PE productivity

DQF Quality Evaluation

DQF Ranking Evaluation

German Swedish Norwegian

Danish Polish

AABB

CC

DDEE

AA BB CC DD EE

DQF Productivity Evaluation

CC

DD

Set up DQF Workflow in SDL WorldServer

Create DQF-enabled Project in WorldServer

Download Project to SDL Trados Studio

After Translation, View DQF Report

Integrating DQF into our Translation Processes

• Continue to assess individual translator performance to target training needs

• Assess quality and productivity across translators, languages, and content types

• Evaluate effect of our process steps on final output quality, cost, and timeliness

• Further analyze productivity and refine pay rates for MT post-editing

• Compare our measurements to industry averages to adjust and/or improve

EMC

Aggregate Quality per Language

Linguistic Quality per vendor

Autodesk