Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Semi-supervised Learning for Neural Machine Translation
Yong Cheng
joint work with Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu
1
Machine Translation
2
Automated translation using computer software
Machine Translation
3
Rule-based Machine Translation 1970s
Example-based Machine Translation 1984
Statical Machine Translation (SMT) 1993
Neural Machine Translation NMT 2014
Trends: learning to translate from DATA
Machine Translation
4
Parallel Corpora
Monolingual Corpora
Parallel corpora are usually limited in
quantity quality coverage& &
Monolingual Corpora Used in SMT and NMT
N-gram language model in SMT Koehn et al., [2007]
Monolingual corpora as decipherment Ravi and Knight [2011]
Integrate a neural language model into NMT. Gulccehre et al. [2015]
Additional pseudo parallel corpus. Sennrich et al. [2016]
5
Supervised Training
Parallel Corpus
Objective
6
Unsupervised Training
Monolingual Corpus
7
cc Our Approach — Autoencoders
8
bushi yu shalong juxing le huitan x
cc Our Approach — Autoencoders
9
bushi yu shalong juxing le huitan xP(y | x;
!θ )
cc Our Approach — Autoencoders
10
bushi yu shalong juxing le huitan x
Bush held a talk with sharon y
P(y | x;!θ )
latent
cc Our Approach — Autoencoders
11
bushi yu shalong juxing le huitan x
Bush held a talk with sharon y
P(y | x;!θ )
P(x | y;!θ )
latent
cc Our Approach — Autoencoders
12
bushi yu shalong juxing le huitan x
Bush held a talk with sharon
′xbushi yu shalong juxing le huitan
y
P(y | x;!θ )
P(x | y;!θ )
latent
cc Our Approach — Autoencoders
13
source autoencoder target autoencoder
Unsupervised Training (Autoencoders)
Monolingual Corpus
14
target autoencoder
Semi-supervised Training
15
Training Objective
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
16
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
17
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
18
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
19
Translation Results
Compared with Moses (SMT) and RNNSearch (NMT)
20
Translation Results
Compared with Sennrich et al. [2015a]
21
Example Translation of Monolingual Corpus
22
ConclusionMonolingual corpora is an important resource for neural machine translation.
We have proposed a semi-supervised approach to training bidirectional neural machine translation models for exploiting monolingual corpora.
As our method is sensitive to the OOVs present in monolingual corpora, we plan to integrate Jean et al. (2015)’s technique on using very large vocabulary into our approach.
23
Thank You !
24
Effect of Sample Size
ZH-EN EN-ZH
25
Effect of OOV ratio
ZH-EN EN-ZH
26