Machine Translation Domain Adaptation Day 19 1. PROJECT #2 2

Machine TranslationDomain Adaptation

Day 19

PROJECT #2

MEMM tools

• Online description of project #2 has been updated with more information

Quick walk throughtraining.txtI/PRP left/VBD ./.John/NNP arrived/VBD ./.

training.featsPRP w0=I:1 w-1=<s>:1VBD w0=left:1 w-1=I:1. w0=.:1 w-1=left:1 NNP w0=John:1 w-1=<s>:1VBD w0=arrived:1 w-1=John:1. w0=.:1 w-1=arrived:1

You write code to convert this to features!

“featurize.pl training.txt training.feats”

trigram.model<binary gobbledegoo>

Run memm_train to train this model

“memm_train --input training.feats --classifier trigram.model --markovOrder 2”

test.txthe/PRP arrived/VBD ./.John/NNP left/VBD ./.

Get some unseen test data…

test.featsPRP w0=he:1 w-1=<s>:1VBD w0=arrived:1 w-1=he:1. w0=.:1 w-1=arrived:1 NNP w0=John:1 w-1=<s>:1VBD w0=left:1 w-1=John:1. w0=.:1 w-1=left:1

Use the same featurization code on test data

“featurize.pl test.txt test.feats”

test.featsPRP w0=he:1 w-1=<s>:1VBD w0=arrived:1 w-1=he:1. w0=.:1 w-1=arrived:1 NNP w0=John:1 w-1=<s>:1VBD w0=left:1 w-1=John:1. w0=.:1 w-1=left:1

test.tagsPRPVBD. NNPVBD.

memm_test predicts tags (memm_test ignores first column; can include true tags)

“memm_test --input test.feats --classifier trigram.model --markovOrder 2 --output test.tags”

MEMM featurestraining.txtI/PRP left/VBD ./.John/NNP arrived/VBD ./.

Actual features used by MEMMPRP w0=I:1 w-1=<s>:1 t[-1]=<s>:1 t[-1]=<s>,t[-2]=<s>:1VBD w0=left:1 w-1=I:1 t[-1]=PRP:1 t[-1]=PRP,t[-2]=<s>:1. w0=.:1 w-1=left:1 t[-1]=VBD:1 t[-1]=VBD,t[-2]=PRP:1<s> t[-1]=.:1 t[-1]=.,t[-2]=VBD:1NNP w0=John:1 w-1=<s>:1 t[-1]=<s>:1 t[-1]=<s>,t[-2]=<s>:1VBD w0=arrived:1 w-1=John:1 t[-1]=NNP:1 t[-1]=NNP,t[-2]=<s>:1. w0=.:1 w-1=arrived:1 t[-1]=VBD:1 t[-1]=VBD,t[-2]=NNP:1<s> t[-1]=.:1 t[-1]=.,t[-2]=VBD:1

You provide these features…

…and add the argument “--markovOrder 2”

The MEMM adds in features about tag

context add training and test time

MACHINE TRANSLATION

Acknowledgments

• Many thanks to (for helpful content and input on content):– Chris Callison-Burch, Matt Post, & Adam Lopez

(JHU)– Philipp Koehn & Barry Haddow (U Edinburgh)– Kevin Knight (ISI)

Translation: global problem and interesting research problem

English32%

Chinese13%

Spanish9%

Japanese7%

French5%

German4%

Arabic4%

Portuguese4%

Other21%

Internet users – 2007• Non-English Internet content and user communities are increasing explosively

• Human translation costs are excessive: major languages range from 10-50 cents per word

• Non-English Internet content and user communities are increasing explosively

• Human translation costs are excessive: major languages range from 10-50 cents per word

Result: the vast majority of published material remains untranslated!

Prevalence of MT on the Web

Estonian

Hungarian

Slovenian

Slovak

Romanian

Latvian

Lithuanian

12.13% 12.93%

25.47%

46.40% 47.40% 50.07% 51.53%

Proportion of MT’d Content by language

From Rarrick et al, 2010

The Goal: (sentence) translation

• Translate source sentences into target sentences– For now, ignore

discourse structure, co-reference, and phenomena across sentence boundaries

滴水之恩當以涌泉相報

A drop of water shall be returned with a burst of

spring.

Types of MT systems

• Source of information– Rule based: People write rules to specify translations of

words, phrases– Data-driven: Use learning techniques to derive translation

“rules” from data sources (e.g., parallel corpora)

• Level of representationInterlingua

Semantic forms

Syntax trees

Phrases

WordsModified Vauquois pyramid

Advantages of data-driven translation

• We can model the genres of documents that we would like to model– Learn contextually appropriate translations for technical

data, chat data, etc.• Very flexible system– Given corpus C = ({x1,y1}, {x2,y2}, …) of sentence pairs– Translate(C, x) = y is a function of the training data and the

input sentence– To build a new system (or optimize our old one) we just

change the data

– But…we need oodles of data to get “good” models

Statistical MT

• Learn word and phrase alignments from “parallel” data

Statistical MT

• Learn word and phrase alignments from “parallel” data– Parallel data? – Parallel documents?

Statistical MT

• Learn word and phrase alignments from “parallel” data– Parallel documents?

Statistical MT

• Learn word and phrase alignments from “parallel” data– Start with parallel documents• Need parallel sentences• Sentence break and sentence align

– Word align and produce word and phrase translation tables (our translation models)

Some Hmong

a house ib lub tsev

a new house ib lub tsev tshiab

my new house kuv lub tsev tshiab

eight new houses yim lub tsev tshiab

my eight new houses kuv yim lub tsev tshiab

Some More Hmong

a house ib lub tsev

a new house ib lub tsev tshiab

my new house kuv lub tsev tshiab

eight new houses yim lub tsev tshiab

my eight new houses kuv yim lub tsev tshiab

the house lub tsev

Even More Hmong

kuv pluag heev I'm very poorib pluag mov a meal ib taig mov a bowl of riceib taig zaub a bowl of vegetables

Statistical MT

• Learn word and phrase alignments from “parallel” data– Start with parallel documents• Need parallel sentences• Sentence break and sentence align

Statistical MT

• Learn word and phrase alignments from “parallel” data– Start with parallel documents

• Need parallel sentences• Sentence break and sentence align

• Use monolingual data to– Build language models

• Inform ordering• Choose best translation from n-best list

Statistical MT Recipe

Start With• Parallel sentences

– Align words & phrases, & generate counts

Build These Components• Translation Model

– Probs associated with aligned words & phrases – P (E|F)

• Monolingual data

• Language Model – P(E)

• Monolingual data• Decoding Algorithm

• Language Model – P(E)• Decoder

– Maximizes P(F|E)*P(E)

Statistical Machine Translation

• Given foreign f, find best English translation e*e* = argmaxe P(e | f)

• Use Bayes’ rule to get “noisy channel” modelP(e | f) = P(f | e) P(∙ e) / P(f)argmaxe P(e | f) = argmax P(f | e) P(∙ e)

• P(f | e) is the channel or translation model• P(e) is the language model

Centauri/Arcturan [Knight, 1997]Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

Slides 38-74 adapted from Kevin Knight and CCB’s JHU crew

Centauri/Arcturan [Knight, 1997]

1a. ok-voon ororok sprok .

1b. at-voon bichat dat .

7a. lalok farok ororok lalok sprok izok enemok .

7b. wat jjat bichat wat dat vat eneat .

2a. ok-drubel ok-voon anok plok sprok .

2b. at-drubel at-voon pippat rrat dat .

8a. lalok brok anok plok nok .

8b. iat lat pippat rrat nnat .

3a. erok sprok izok hihok ghirok .

3b. totat dat arrat vat hilat .

9a. wiwok nok izok kantok ok-yurp .

9b. totat nnat quat oloat at-yurp .4a. ok-voon anok drok brok jok .

4b. at-voon krat pippat sat lat .

10a. lalok mok nok yorok ghirok clok .

10b. wat nnat gat mat bat hilat .5a. wiwok farok izok stok .

5b. totat jjat quat cat .

11a. lalok nok crrrok hihok yorok zanzanok .

11b. wat nnat arrat mat zanzanat .6a. lalok sprok izok jok stok .

6b. wat dat krat quat cat .

12a. lalok rarok nok izok hihok mok .

12b. wat nnat forat arrat vat gat .

Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

process ofelimination

cognate?

zerofertility

Clients do not sell pharmaceuticals in Europe => Clientes no venden medicinas en Europa

It’s Really Spanish/English

1a. Garcia and associates .1b. Garcia y asociados .

7a. the clients and the associates are enemies .7b. los clients y los asociados son enemigos .

2a. Carlos Garcia has three associates .2b. Carlos Garcia tiene tres asociados .

8a. the company has three groups .8b. la empresa tiene tres grupos .

3a. his associates are not strong .3b. sus asociados no son fuertes .

9a. its groups are in Europe .9b. sus grupos estan en Europa .

4a. Garcia has a company also .4b. Garcia tambien tiene una empresa .

10a. the modern groups sell strong pharmaceuticals .10b. los grupos modernos venden medicinas fuertes .

5a. its clients are angry .5b. sus clientes estan enfadados .

11a. the groups do not sell zenzanine .11b. los grupos no venden zanzanina .

6a. the associates are also angry .6b. los asociados tambien estan enfadados .

12a. the small groups are not modern .12b. los grupos pequenos no son modernos .