Unsupervised Constraint Driven Learning for Transliteration Discovery

M. Chang, D. Goldwasser, D. Roth, and Y. Tu

What I am going to do today…

Goal 1 : Present the transliteration work Get feedback!

Goal 2: Think about this work with CCM Tutorial …. I will try to present this work in a slightly

different way Some of them are my personal comment Different than our yesterday discussion

Please give us comment about this Make this work more general (not only

transliteration)

Wait a sec! What is CCM?

),(maxarg xyfwTy

Cc

cT

y xycxyfw ),,(),(maxarg

Constraints Driven Learning

Why Constraints? The Goal: Building a good system easily We have prior knowledge at our hand

Why not inject knowledge directly ?

How useful are constraints? Useful for supervised learning [Yih and Roth 04]

[many others]

Useful for semi-supervised learning [Chang et.al. ACL 2007]

Some times more efficient than labeling data directly

Unsupervised Constraint Driven Learning

In this work We do not use any label instance Achieve to good performance that competitive

several supervised model

Compared to [Chang et.al. ACL 2007]

In ACL 07, they use a small amount of dataset (5-20)

Reason: Bad Models can not benefit from constraints!

For some applications, we have very good resource We do not need labeled instances at all!

6

In a nutshell:

Traditional semi-supervised learning.

• Model can drift from the correct one. Model

Unlabeled Data

Prediction

Label unlabeled data

Feedback

Learn from labeled data

Unsupervised Learning

Resource

7

In a nutshell:

CODLUse constraints to generate better training samples in unsupervised learning.

Prediction+ Constraints

Model

Unlabeled Data

PredictionFeedback

More accurate labeling

Better Model

CODLImproves “Simple” Model

Using Expressive Constraints

Outline

Constraint Driven Learning (CoDL)

Transliteration Discovery

Algorithm

Experimental Results

Transliteration Generation (Not our focus)

Given a Source Transliteration; What is the target transliteration? Bush

布希 Sushi

壽司 Issues

Ambiguity : For the same source word, many different

transliteration Think about Chinese

What we want: find the most widely used transliteration

Transliteration Discovery (Our focus)

Problem Settings Give you two list of words, map them!

Advantages A relatively easy problem Can find the most widely used transliteration

Assumption: Source: English Each source entities has a transliteration in the

target candidates Target candidates might not be named entities

Outline

Constraint Driven Learning (CoDL)

Transliteration Discovery

Algorithm

Experimental Results

Algorithm Outline

Prediction Model

How to use existing resource to construct the Model?

Constraints?

Learning Algorithm

The Prediction Model

How do we make prediction? Given a source word, how to predict the best target

?

Model 1 : Vs, Vt Yes or No Issue: Not many obvious constraints can be added Not a structure prediction problem

Model 2: Vs, Vt Hidden variables Yes or No Predicting F is a structure prediction algorithm We can add constraints more easily

The Prediction Model

Score for a pair

A CCM formulation

A slightly different scoring function

Ff

tsts vvfWfvvWFscore ),|(]1[),,|(

Cc

tststs cvvFvvWFscorevvFg ),,|(),,|(),,(

More on this point in the next few slides

),(maxarg*ts vvscorev

t

||

),,(maxarg),(

t

tsFts v

vvFgvvscore

Hidden Variables

Violation

Prediction Model: Another View

The scoring function looks like weight times features!

If there is a bad feature, score - ∞

Our Hidden variable (Feature Vectors): Character Mapping

),(

),|(]1[),,|(

tsT

Fftsts

vvFW

vvfWfvvWFscore

Cc

tstsT

ts cvvFvvFWvvFg ),,|(),(),,(

),( ts vvF

Everything

(a,a), (o,O), (w,_),……

Algorithm Outline

Prediction Model

How to use existing resource to construct the Model?

Constraints?

Learning Algorithm

Resource: Romanization Table

Hebrew, Russian How can you type Hebrew or Russian?

Use English Keyboard, C maps to A similar character “C” or “S” in Hebrew or Russian

Very easy to get Ambiguous

Special Case: Chinese (Pin Yin)壽司 shòu sī (Low ambiguity) Map Pin-Yin to English (sushi) Romanization Table? a a

Initialize the Table

Every character pair in the Romanization Table Weight = 0 Everything else, -1 Could have better way to do initialization

Note: All (v_s,v_t) will get zero without constraints

Cc

tstsT

ts cvvFvvFWvvFg ),,|(),(),,(

Algorithm Outline

Prediction Model

How to use existing resource to construct the Model?

Constraints?

Learning Algorithm

Constraints

General Constraints Coverage: all character need to be mapped at

least once No crossing: character mappings can not cross

each other

Language Specific Constraints General Restricted Mapping Initial Restricted Mapping Length Restriction

Constraints

Pin-Yin to EnglishMany other works use similar information as well!

Algorithm Outline

Prediction Model

How to use existing resource to construct the Model?

Constraints?

Learning Algorithm

High-Level Overview

Model Resource While Converge

Use Model + Constraints to get Labels (for both F, y)

Update Model with newly labeled F and y (without Constraints) (details in the next slide)

Similar to ACL 07 Update the model without Constraints

Difference from ACL 07 We get feedback from the labels of both hidden

variables and output

Training

Predict hidden variables and the labels

Update

Algorithm

Outline

Constraint Driven Learning (CoDL)

Transliteration Discovery

Algorithm

Experimental Results

Experimental Setting

Evaluation

ACC: Top candidate is (one of) the right answer

Learning Algorithm Linear SVM with C = 0.5

Dataset English-Hebrew 300: 300 English-Chinese 581:681 English-Russian 727:50648 (Target includes all

words)

Results - Hebrew

Results - Russian

Analysis

A small Russian subset was used here

1) Without Constraints (on features), Romanization Table is useless!

2) General Constraints are more important!

4) Better Constraints Lead to Better Final Results

3) Learning has great impact here!But constraints are very important, too!

Related Works (Need more work here)

Learning the score for Edit Distance

Previous transliteration works

Machine translation?

Conclusion

ML: unsupervised constraint driven algorithm Use hidden variable to find more constraints (e.g. co-ref) Use constraints to find “cleaner” feature representation

Transliteration: Usage of Normalization Table as the starting point

We can get good results without training data Right constraints (modeling) is the key

Future Work Transliteration Model: Better Model, Quicker Inference CoDL: Other applications for unsupervised CoDL

33

Constraint - Driven Learning (CODL)

=learn(Tr)For N iterations do

T= For each x in unlabeled dataset

y Inference(x, )T=T {(x, y)}

= +(1- )learn(T)

Any supervised learning algorithm parametrized by

Learn from new training data.Weight supervised and unsupervised model(Nigam2000*).

Augmenting the training set (feedback). Any inference algorithm (with constraints).

Inference(x,C, )

34

Unsupervised Constraint - Driven Learning

=Construct(Resource)For N iterations do

T= For each x in unlabeled dataset

y Inference(x, )T=T {(x, y)}

= +(1- )learn(T)

Construct the model with Resources

Learn from new training data. = 0 in this work

Augmenting the training set (feedback). Any inference algorithm (with constraints).

Inference(x,C, )