Embed Size (px)
Citation preview
Proceedings of the 3rd Workshop on Asian Translation,pages 1–46, Osaka, Japan, December 11-17 2016.
Overview of the 3rd Workshop on Asian Translation
Toshiaki NakazawaJapan Science and
Technology [email protected]
Chenchen Ding and Hideya MinoNational Institute of
Information andCommunications Technology
{chenchen.ding, hideya.mino}@nict.go.jp
Isao GotoNHK
Graham NeubigCarnegie Mellon [email protected]
Sadao KurohashiKyoto University
Abstract
This paper presents the results of the shared tasks from the 3rd workshop on Asian translation(WAT2016) including J↔E, J↔C scientific paper translation subtasks, C↔J, K↔J, E↔J patenttranslation subtasks, I↔E newswire subtasks and H↔E, H↔J mixed domain subtasks. For theWAT2016, 15 institutions participated in the shared tasks. About 500 translation results havebeen submitted to the automatic evaluation server, and selected submissions were manually eval-uated.
1 Introduction
The Workshop on Asian Translation (WAT) is a new open evaluation campaign focusing on Asianlanguages. Following the success of the previous workshops WAT2014 (Nakazawa et al., 2014) andWAT2015 (Nakazawa et al., 2015), WAT2016 brings together machine translation researchers and usersto try, evaluate, share and discuss brand-new ideas of machine translation. We are working toward thepractical use of machine translation among all Asian countries.
For the 3rd WAT, we adopt new translation subtasks with English-Japanese patent description,Indonesian-English news description and Hindi-English and Hindi-Japanese mixed domain corpus inaddition to the subtasks that were conducted in WAT2015. Furthermore, we invited research papers ontopics related to the machine translation, especially for Asian languages. The submissions of the re-search papers were peer reviewed by at least 2 program committee members and the program committeeaccepted 7 papers that cover wide variety of topics such as neural machine translation, simultaneousinterpretation, southeast Asian languages and so on.
WAT is unique for the following reasons:
• Open innovation platformThe test data is fixed and open, so evaluations can be repeated on the same data set to confirmchanges in translation accuracy over time. WAT has no deadline for automatic translation qualityevaluation (continuous evaluation), so translation results can be submitted at any time.
• Domain and language pairsWAT is the world’s first workshop that uses scientific papers as the domain, and Chinese ↔Japanese, Korean ↔ Japanese and Indonesian ↔ English as language pairs. In the future, wewill add more Asian languages, such as Vietnamese, Thai, Burmese and so on.
• Evaluation methodEvaluation is done both automatically and manually. For human evaluation, WAT uses pairwiseevaluation as the first-stage evaluation. Also, JPO adequacy evaluation is conducted for the selectedsubmissions according to the pairwise evaluation results.
This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/
1
LangPair Train Dev DevTest TestASPEC-JE 3,008,500 1,790 1,784 1,812ASPEC-JC 672,315 2,090 2,148 2,107
Table 1: Statistics for ASPEC.
2 Dataset
WAT uses the Asian Scientific Paper Excerpt Corpus (ASPEC) 1, JPO Patent Corpus (JPC) 2, BPPTCorpus 3 and IIT Bombay English-Hindi Corpus (IITB Corpus) 4 as the dataset.
2.1 ASPECASPEC is constructed by the Japan Science and Technology Agency (JST) in collaboration with theNational Institute of Information and Communications Technology (NICT). It consists of a Japanese-English scientific paper abstract corpus (ASPEC-JE), which is used for J↔E subtasks, and a Japanese-Chinese scientific paper excerpt corpus (ASPEC-JC), which is used for J↔C subtasks. The statistics foreach corpus are described in Table1.
2.1.1 ASPEC-JEThe training data for ASPEC-JE was constructed by the NICT from approximately 2 million Japanese-English scientific paper abstracts owned by the JST. Because the abstracts are comparable corpora, thesentence correspondences are found automatically using the method from (Utiyama and Isahara, 2007).Each sentence pair is accompanied by a similarity score and the field symbol. The similarity scores arecalculated by the method from (Utiyama and Isahara, 2007). The field symbols are single letters A-Z andshow the scientific field for each document5. The correspondence between the symbols and field names,along with the frequency and occurrence ratios for the training data, are given in the README file fromASPEC-JE.
The development, development-test and test data were extracted from parallel sentences from theJapanese-English paper abstracts owned by JST that are not contained in the training data. Each dataset contains 400 documents. Furthermore, the data has been selected to contain the same relative fieldcoverage across each data set. The document alignment was conducted automatically and only docu-ments with a 1-to-1 alignment are included. It is therefore possible to restore the original documents.The format is the same as for the training data except that there is no similarity score.
2.1.2 ASPEC-JCASPEC-JC is a parallel corpus consisting of Japanese scientific papers from the literature database andelectronic journal site J-STAGE of JST that have been translated to Chinese with permission from thenecessary academic associations. The parts selected were abstracts and paragraph units from the bodytext, as these contain the highest overall vocabulary coverage.
The development, development-test and test data are extracted at random from documents containingsingle paragraphs across the entire corpus. Each set contains 400 paragraphs (documents). Therefore,there are no documents sharing the same data across the training, development, development-test andtest sets.
2.2 JPCJPC was constructed by the Japan Patent Office (JPO). It consists of a Chinese-Japanese patent de-scription corpus (JPC-CJ), Korean-Japanese patent description corpus (JPC-KJ) and English-Japanesepatent description corpus (JPC-EJ) with four sections, which are Chemistry, Electricity, Mechanical en-gineering, and Physics, based on International Patent Classification (IPC). Each corpus is separated into
1http://lotus.kuee.kyoto-u.ac.jp/ASPEC/2http://lotus.kuee.kyoto-u.ac.jp/WAT/patent/index.html3http://orchid.kuee.kyoto-u.ac.jp/WAT/bppt-corpus/index.html4http://www.cfilt.iitb.ac.in/iitb parallel/index.html5http://opac.jst.go.jp/bunrui/index.html
2
LangPair Train Dev DevTest TestJPC-CJ 1,000,000 2,000 2,000 2,000JPC-KJ 1,000,000 2,000 2,000 2,000JPC-EJ 1,000,000 2,000 2,000 2,000
Table 2: Statistics for JPC.
LangPair Train Dev DevTest TestBPPT-IE 50,000 400 400 400
Table 3: Statistics for BPPT Corpus.
training, development, development-test and test data, which are sentence pairs. This corpus was usedfor patent subtasks C↔J, K↔J and E↔J. The statistics for each corpus are described in Table2.
The Sentence pairs in each data were randomly extracted from a description part of comparable patentdocuments under the condition that a similarity score between sentences is greater than or equal to thethreshold value 0.05. The similarity score was calculated by the method from (Utiyama and Isahara,2007) as with ASPEC. Document pairs which were used to extract sentence pairs for each data werenot used for the other data. Furthermore, the sentence pairs were extracted to be same number amongthe four sections. The maximize number of sentence pairs which are extracted from one document pairwas limited to 60 for training data and 20 for the development, development-test and test data. Thetraining data for JPC-CJ was made with sentence pairs of Chinese-Japanese patent documents publishedin 2012. For JPC-KJ and JPC-EJ, the training data was extracted from sentence pairs of Korean-Japaneseand English-Japanese patent documents published in 2011 and 2012. The development, development-test and test data for JPC-CJ, JPC-KJ and JPC-EJ were respectively made with 100 patent documentspublished in 2013.
2.3 BPPT Corpus
BPPT Corpus was constructed by Badan Pengkajian dan Penerapan Teknologi (BPPT). This corpus con-sists of a Indonesian-English news corpus (BPPT-IE) with five sections, which are Finance, International,Science and Technology, National, and Sports. These data come from Antara News Agency. This corpuswas used for newswire subtasks I↔E. The statistics for each corpus are described in Table3.
2.4 IITB Corpus
IIT Bombay English-Hindi corpus contains English-Hindi parallel corpus (IITB-EH) as well as mono-lingual Hindi corpus collected from a variety of existing sources and corpora developed at the Centerfor Indian Language Technology, IIT Bombay over the years. This corpus was used for mixed domainsubtasks H↔E. Furthermore, mixed domain subtasks H↔J were added as a pivot language task with aparallel corpus created using openly available corpora (IITB-JH) 6. Most sentence pairs in IITB-JH comefrom the Bible corpus. The statistics for each corpus are described in Table4.
3 Baseline Systems
Human evaluations were conducted as pairwise comparisons between the translation results for a specificbaseline system and translation results for each participant’s system. That is, the specific baseline systemwas the standard for human evaluation. A phrase-based statistical machine translation (SMT) systemwas adopted as the specific baseline system at WAT 2016, which is the same system as that at WAT 2014and WAT 2015.
In addition to the results for the baseline phrase-based SMT system, we produced results for the base-line systems that consisted of a hierarchical phrase-based SMT system, a string-to-tree syntax-based
6http://lotus.kuee.kyoto-u.ac.jp/WAT/Hindi-corpus/WAT2016-Ja-Hi.zip
3
LangPair Train Dev Test Monolingual Corpus (Hindi)IITB-EH 1,492,827 520 2,507 45,075,279IITB-JH 152,692 1,566 2,000 -
Table 4: Statistics for IITB Corpus.
SMT system, a tree-to-string syntax-based SMT system, seven commercial rule-based machine trans-lation (RBMT) systems, and two online translation systems. The SMT baseline systems consisted ofpublicly available software, and the procedures for building the systems and for translating using thesystems were published on the WAT web page7. We used Moses (Koehn et al., 2007; Hoang et al., 2009)as the implementation of the baseline SMT systems. The Berkeley parser (Petrov et al., 2006) was usedto obtain syntactic annotations. The baseline systems are shown in Table 5.
The commercial RBMT systems and the online translation systems were operated by the organizers.We note that these RBMT companies and online translation companies did not submit themselves. Be-cause our objective is not to compare commercial RBMT systems or online translation systems fromcompanies that did not themselves participate, the system IDs of these systems are anonymous in thispaper.
7http://lotus.kuee.kyoto-u.ac.jp/WAT/
4
ASP
EC
JPC
IIT
BB
PPT
pivo
tSy
stem
IDSy
stem
Type
JEE
JJC
CJ
JEE
JJC
CJ
JKK
JH
EE
HIE
EI
HJ
JHSM
TPh
rase
Mos
es’P
hras
e-ba
sed
SMT
SMT
✓✓
✓✓
✓✓
✓✓
✓✓
✓✓
✓✓
✓✓
SMT
Hie
roM
oses
’Hie
rarc
hica
lPhr
ase-
base
dSM
TSM
T✓
✓✓
✓✓
✓✓
✓✓
✓✓
✓✓
✓SM
TS2
TM
oses
’Str
ing-
to-T
ree
Synt
ax-b
ased
SMT
and
Ber
kele
ypa
rser
SMT
✓✓
✓✓
✓✓
SMT
T2S
Mos
es’T
ree-
to-S
trin
gSy
ntax
-bas
edSM
Tan
dB
erke
ley
pars
erSM
T✓
✓✓
✓✓
✓R
BM
TX
The
Hon
yaku
V15
(Com
mer
cial
syst
em)
RB
MT
✓✓
✓✓
RB
MT
XA
TL
AS
V14
(Com
mer
cial
syst
em)
RB
MT
✓✓
✓✓
RB
MT
XPA
T-Tr
anse
r200
9(C
omm
erci
alsy
stem
)R
BM
T✓
✓✓
✓R
BM
TX
J-B
eijin
g7
(Com
mer
cial
syst
em)
RB
MT
✓✓
✓✓
RB
MT
XH
ohra
i201
1(C
omm
erci
alsy
stem
)R
BM
T✓
✓✓
RB
MT
XJ
Soul
9(C
omm
erci
alsy
stem
)R
BM
T✓
✓R
BM
TX
Kor
ai20
11(C
omm
erci
alsy
stem
)R
BM
T✓
✓O
nlin
eX
Goo
gle
tran
slat
e(J
uly
and
Aug
ust,
2016
orA
ugus
t,20
15)
(SM
T)
✓✓
✓✓
✓✓
✓✓
✓✓
✓✓
✓✓
✓✓
Onl
ine
XB
ing
tran
slat
or(J
uly
and
Aug
ust,
2016
orA
ugus
tand
Sept
embe
r,20
15)
(SM
T)
✓✓
✓✓
✓✓
✓✓
✓✓
✓✓
✓✓
✓✓
Tabl
e5:
Bas
elin
eSy
stem
s
5
3.1 Training DataWe used the following data for training the SMT baseline systems.
• Training data for the language model: All of the target language sentences in the parallel corpus.• Training data for the translation model: Sentences that were 40 words or less in length. (For ASPEC
Japanese–English training data, we only used train-1.txt, which consists of one million parallelsentence pairs with high similarity scores.)
• Development data for tuning: All of the development data.
3.2 Common Settings for Baseline SMTWe used the following tools for tokenization.
• Juman version 7.08 for Japanese segmentation.• Stanford Word Segmenter version 2014-01-049 (Chinese Penn Treebank (CTB) model) for Chinese
segmentation.• The Moses toolkit for English and Indonesian tokenization.• Mecab-ko10 for Korean segmentation.• Indic NLP Library11 for Hindi segmentation.
To obtain word alignments, GIZA++ and grow-diag-final-and heuristics were used. We used 5-gramlanguage models with modified Kneser-Ney smoothing, which were built using a tool in the Mosestoolkit (Heafield et al., 2013).
3.3 Phrase-based SMTWe used the following Moses configuration for the phrase-based SMT system.
• distortion-limit– 20 for JE, EJ, JC, and CJ– 0 for JK, KJ, HE, and EH– 6 for IE and EI
• msd-bidirectional-fe lexicalized reordering• Phrase score option: GoodTuring
The default values were used for the other system parameters.
3.4 Hierarchical Phrase-based SMTWe used the following Moses configuration for the hierarchical phrase-based SMT system.
• max-chart-span = 1000• Phrase score option: GoodTuring
The default values were used for the other system parameters.
3.5 String-to-Tree Syntax-based SMTWe used the Berkeley parser to obtain target language syntax. We used the following Moses configurationfor the string-to-tree syntax-based SMT system.
• max-chart-span = 1000• Phrase score option: GoodTuring• Phrase extraction options: MaxSpan = 1000, MinHoleSource = 1, and NonTermConsecSource.
The default values were used for the other system parameters.8http://nlp.ist.i.kyoto-u.ac.jp/EN/index.php?JUMAN9http://nlp.stanford.edu/software/segmenter.shtml
10https://bitbucket.org/eunjeon/mecab-ko/11https://bitbucket.org/anoopk/indic nlp library
6
3.6 Tree-to-String Syntax-based SMTWe used the Berkeley parser to obtain source language syntax. We used the following Moses configura-tion for the baseline tree-to-string syntax-based SMT system.
• max-chart-span = 1000• Phrase score option: GoodTuring• Phrase extraction options: MaxSpan = 1000, MinHoleSource = 1, MinWords = 0, NonTermCon-
secSource, and AllowOnlyUnalignedWords.
The default values were used for the other system parameters.
4 Automatic Evaluation
4.1 Procedure for Calculating Automatic Evaluation ScoreWe calculated automatic evaluation scores for the translation results by applying three metrics: BLEU(Papineni et al., 2002), RIBES (Isozaki et al., 2010) and AMFM (Banchs et al., 2015). BLEU scores werecalculated using multi-bleu.perl distributed with the Moses toolkit (Koehn et al., 2007); RIBES scoreswere calculated using RIBES.py version 1.02.4 12; AMFM scores were calculated using scripts createdby technical collaborators of WAT2016. All scores for each task were calculated using one reference.Before the calculation of the automatic evaluation scores, the translation results were tokenized withword segmentation tools for each language.
For Japanese segmentation, we used three different tools: Juman version 7.0 (Kurohashi et al., 1994),KyTea 0.4.6 (Neubig et al., 2011) with Full SVM model 13 and MeCab 0.996 (Kudo, 2005) with IPAdictionary 2.7.0 14. For Chinese segmentation we used two different tools: KyTea 0.4.6 with Full SVMModel in MSR model and Stanford Word Segmenter version 2014-06-16 with Chinese Penn Treebank(CTB) and Peking University (PKU) model 15 (Tseng, 2005). For Korean segmentation we used mecab-ko 16. For English and Indonesian segmentations we used tokenizer.perl 17 in the Moses toolkit. ForHindi segmentation we used Indic NLP Library 18.
Detailed procedures for the automatic evaluation are shown on the WAT2016 evaluation web page 19.
4.2 Automatic Evaluation SystemThe participants submit translation results via an automatic evaluation system deployed on the WAT2016web page, which automatically gives evaluation scores for the uploaded results. Figure 1 shows the sub-mission interface for participants. The system requires participants to provide the following informationwhen they upload translation results:
• Subtask:
– Scientific papers subtask (J ↔ E, J ↔ C);– Patents subtask (C ↔ J , K ↔ J , E ↔ J);– Newswire subtask (I ↔ E)– Mixed domain subtask (H ↔ E, H ↔ J)
• Method (SMT, RBMT, SMT and RBMT, EBMT, NMT, Other);
12http://www.kecl.ntt.co.jp/icl/lirg/ribes/index.html13http://www.phontron.com/kytea/model.html14http://code.google.com/p/mecab/downloads/detail?
name=mecab-ipadic-2.7.0-20070801.tar.gz15http://nlp.stanford.edu/software/segmenter.shtml16https://bitbucket.org/eunjeon/mecab-ko/17https://github.com/moses-smt/mosesdecoder/tree/
RELEASE-2.1.1/scripts/tokenizer/tokenizer.perl18https://bitbucket.org/anoopk/indic nlp library19http://lotus.kuee.kyoto-u.ac.jp/WAT/evaluation/index.html
7
Figu
re1:
The
subm
issi
onw
ebpa
gefo
rpar
ticip
ants
8
• Use of other resources in addition to ASPEC / JPC / BPPT Corpus / IITB Corpus;
• Permission to publish the automatic evaluation scores on the WAT2016 web page.
The server for the system stores all submitted information, including translation results and scores, al-though participants can confirm only the information that they uploaded. Information about translationresults that participants permit to be published is disclosed on the web page. In addition to submittingtranslation results for automatic evaluation, participants submit the results for human evaluation usingthe same web interface. This automatic evaluation system will remain available even after WAT2016.Anybody can register to use the system on the registration web page 20.
5 Human Evaluation
In WAT2016, we conducted 2 kinds of human evaluations: pairwise evaluation and JPO adequacy eval-uation.
5.1 Pairwise Evaluation
The pairwise evaluation is the same as the last year, but not using the crowdsourcing this year. We askedprofessional translation company to do pairwise evaluation. The cost of pairwise evaluation per sentenceis almost the same to that of last year.
We randomly chose 400 sentences from the Test set for the pairwise evaluation. We used the samesentences as the last year for the continuous subtasks. Each submission is compared with the baselinetranslation (Phrase-based SMT, described in Section 3) and given a Pairwise score21.
5.1.1 Pairwise Evaluation of SentencesWe conducted pairwise evaluation of each of the 400 test sentences. The input sentence and two transla-tions (the baseline and a submission) are shown to the annotators, and the annotators are asked to judgewhich of the translation is better, or if they are of the same quality. The order of the two translations areat random.
5.1.2 VotingTo guarantee the quality of the evaluations, each sentence is evaluated by 5 different annotators and thefinal decision is made depending on the 5 judgements. We define each judgement ji(i = 1, · · · , 5) as:
ji =
1 if better than the baseline−1 if worse than the baseline0 if the quality is the same
The final decision D is defined as follows using S =∑
ji:
D =
win (S ≥ 2)loss (S ≤ −2)tie (otherwise)
5.1.3 Pairwise Score CalculationSuppose that W is the number of wins compared to the baseline, L is the number of losses and T is thenumber of ties. The Pairwise score can be calculated by the following formula:
Pairwise = 100× W − L
W + L + T
From the definition, the Pairwise score ranges between -100 and 100.20http://lotus.kuee.kyoto-u.ac.jp/WAT/registration/index.html21It was called HUMAN score in WAT2014 and Crowd score in WAT2015.
9
5 All important information is transmitted correctly.(100%)
4 Almost all important information is transmitted cor-rectly. (80%–)
3 More than half of important information is trans-mitted correctly. (50%–)
2 Some of important information is transmitted cor-rectly. (20%–)
1 Almost all important information is NOT transmit-ted correctly. (–20%)
Table 6: The JPO adequacy criterion
5.1.4 Confidence Interval EstimationThere are several ways to estimate a confidence interval. We chose to use bootstrap resampling (Koehn,2004) to estimate the 95% confidence interval. The procedure is as follows:
1. randomly select 300 sentences from the 400 human evaluation sentences, and calculate the Pairwisescore of the selected sentences
2. iterate the previous step 1000 times and get 1000 Pairwise scores
3. sort the 1000 scores and estimate the 95% confidence interval by discarding the top 25 scores andthe bottom 25 scores
5.2 JPO Adequacy EvaluationThe participants’ systems, which achieved the top 3 highest scores among the pairwise evaluation resultsof each subtask22, were also evaluated with the JPO adequacy evaluation. The JPO adequacy evaluationwas carried out by translation experts with a quality evaluation criterion for translated patent documentswhich the Japanese Patent Office (JPO) decided. For each system, two annotators evaluate the testsentences to guarantee the quality.
5.2.1 Evaluation of SentencesThe number of test sentences for the JPO adequacy evaluation is 200. The 200 test sentences wererandomly selected from the 400 test sentences of the pairwise evaluation. The test sentence include theinput sentence, the submitted system’s translation and the reference translation.
5.2.2 Evaluation CriterionTable 6 shows the JPO adequacy criterion from 5 to 1. The evaluation is performed subjectively. “Im-portant information” represents the technical factors and their relationships. The degree of importanceof each element is also considered to evaluate. The percentages in each grade are rough indications forthe transmission degree of the source sentence meanings. The detailed criterion can be found on the JPOdocument (in Japanese) 23.
6 Participants List
Table 7 shows the list of participants for WAT2016. This includes not only Japanese organizations, butalso some organizations from outside Japan. 15 teams submitted one or more translation results to theautomatic evaluation server or human evaluation.
22The number of systems varies depending on the subtasks.23http://www.jpo.go.jp/shiryou/toushin/chousa/tokkyohonyaku hyouka.htm
10
ASP
EC
JPC
BPP
TII
TB
Cpi
vot
Team
IDO
rgan
izat
ion
JEE
JJC
CJ
JEE
JJC
CJ
JKK
JIE
EI
HE
EH
HJ
JHN
AIS
T(N
eubi
g,20
16)
Nar
aIn
stitu
teof
Scie
nce
and
Tech
nolo
gy✓
Kyo
to-U
(Cro
mie
res
etal
.,20
16)
Kyo
toU
nive
rsity
✓✓
✓✓
TM
U(Y
amag
ishi
etal
.,20
16)
Toky
oM
etro
polit
anU
nive
rsity
✓bj
tunl
p(L
ieta
l.,20
16)
Bei
jing
Jiao
tong
Uni
vers
ity✓
✓✓
✓✓
✓✓
✓Se
nse
(Tan
,201
6)Sa
arla
ndU
nive
rsity
✓✓
NIC
T-2
(Im
amur
aan
dSu
mita
,201
6)N
atio
nalI
nstit
ute
ofIn
form
atio
nan
dC
omm
unic
atio
nTe
chno
logy
✓✓
✓✓
✓✓
✓✓
WA
SUIP
S(Y
ang
and
Lep
age,
2016
)W
ased
aU
nive
rsity
✓E
HR
(Eha
ra,2
016)
Eha
raN
LP
Res
earc
hL
abor
ator
y✓
✓✓
✓✓
✓nt
t(Su
doh
and
Nag
ata,
2016
)N
TT
Com
mun
icat
ion
Scie
nce
Lab
orat
orie
s✓
TOK
YO
MT
(Shu
and
Miu
ra,2
016)
Web
lio,I
nc.
✓II
TB
-EN
-ID
(Sin
ghet
al.,
2016
)In
dian
Inst
itute
ofTe
chno
logy
Bom
bay
✓✓
JAPI
O(K
inos
hita
etal
.,20
16)
Japa
nPa
tent
Info
rmat
ion
Org
aniz
atio
n✓
✓✓
✓✓
IIT
P-M
T(S
enet
al.,
2016
)In
dian
Inst
itute
ofTe
chno
logy
Patn
a✓
UT-
KA
Y(H
ashi
mot
oet
al.,
2016
)U
nive
rsity
ofTo
kyo
✓U
T-A
KY
(Eri
guch
ieta
l.,20
16)
Uni
vers
ityof
Toky
o✓
Tabl
e7:
Lis
tofp
artic
ipan
tsw
hosu
bmitt
edtr
ansl
atio
nre
sults
toW
AT
2016
and
thei
rpar
ticip
atio
nin
each
subt
asks
.
11
7 Evaluation Results
In this section, the evaluation results for WAT2016 are reported from several perspectives. Some of theresults for both automatic and human evaluations are also accessible at the WAT2016 website24.
7.1 Official Evaluation ResultsFigures 2, 3, 4 and 5 show the official evaluation results of ASPEC subtasks, Figures 6, 7, 8, 9 and 10show those of JPC subtasks, Figures 11 and 12 show those of BPPT subtasks and Figures 13 and 14 showthose of IITB subtasks. Each figure contains automatic evaluation results (BLEU, RIBES, AM-FM), thepairwise evaluation results with confidence intervals, correlation between automatic evaluations and thepairwise evaluation, the JPO adequacy evaluation result and evaluation summary of top systems.
The detailed automatic evaluation results for all the submissions are shown in Appendix A. The de-tailed JPO adequacy evaluation results for the selected submissions are shown in Table 8. The weightsfor the weighted κ (Cohen, 1968) is defined as |Evaluation1− Evaluation2|/4.
From the evaluation results, the following can be observed:
• Neural network based translation models work very well also for Asian languages.
• None of the automatic evaluation measures perfectly correlate to the human evaluation result (JPOadequacy).
• The JPO adequacy evaluation result of IITB E→H shows an interesting tendency: the system whichachieved the best average score has the lowest ratio of the perfect translations and vice versa.
7.2 Statistical Significance Testing of Pairwise Evaluation between SubmissionsTables 9, 10, 11 and 12 show the results of statistical significance testing of ASPEC subtasks, Tables 13,14, 15, 16 and 17 show those of JPC subtasks, 18 shows those of BPPT subtasks and 19 shows those ofJPC subtasks. ≫, ≫ and > mean that the system in the row is better than the system in the column at asignificance level of p < 0.01, 0.05 and 0.1 respectively. Testing is also done by the bootstrap resamplingas follows:
1. randomly select 300 sentences from the 400 pairwise evaluation sentences, and calculate the Pair-wise scores on the selected sentences for both systems
2. iterate the previous step 1000 times and count the number of wins (W ), losses (L) and ties (T )
3. calculate p = LW+L
Inter-annotator AgreementTo assess the reliability of agreement between the workers, we calculated the Fleiss’ κ (Fleiss and others,1971) values. The results are shown in Table 20. We can see that the κ values are larger for X → Jtranslations than for J → X translations. This may be because the majority of the workers are Japanese,and the evaluation of one’s mother tongue is much easier than for other languages in general.
7.3 Chronological EvaluationFigure 15 shows the chronological evaluation results of 4 subtasks of ASPEC and 2 subtasks of JPC. TheKyoto-U (2016) (Cromieres et al., 2016), ntt (2016) (Sudoh and Nagata, 2016) and naver (2015) (Lee etal., 2015) are NMT systems, the NAIST (2015) (Neubig et al., 2015) is a forest-to-string SMT system,Kyoto-U (2015) (Richardson et al., 2015) is a dependency tree-to-tree EBMT system and JAPIO (2016)(Kinoshita et al., 2016) system is a phrase-based SMT system.
What we can see is that in ASPEC-JE and EJ, the overall quality is improved from the last year, butthe ratio of grade 5 is decreased. This is because the NMT systems can output much fluent translations
24http://lotus.kuee.kyoto-u.ac.jp/WAT/evaluation/index.html
12
but the adequacy is worse. As for ASPEC-JC and CJ, the quality is very much improved. Literatures(Junczys-Dowmunt et al., 2016) say that Chinese receives the biggest benefits from NMT.
The translation quality of JPC-CJ does not so much varied from the last year, but that of JPC-KJ ismuch worse. Unfortunately, the best systems participated last year did not participate this year, so it isnot directly comparable.
8 Submitted Data
The number of published automatic evaluation results for the 15 teams exceeded 400 before the start ofWAT2016, and 63 translation results for pairwise evaluation were submitted by 14 teams. Furthermore,we selected maximum 3 translation results from each subtask and evaluated them for JPO adequacyevaluation. We will organize the all of the submitted data for human evaluation and make this public.
9 Conclusion and Future Perspective
This paper summarizes the shared tasks of WAT2016. We had 15 participants worldwide, and collected alarge number of useful submissions for improving the current machine translation systems by analyzingthe submissions and identifying the issues.
For the next WAT workshop, we plan to include newspaper translation tasks for Japanese, Chineseand English where the context information is important to achieve high translation quality, so it is achallenging task.
We would also be very happy to include other languages if the resources are available.
Appendix A Submissions
Tables 21 to 36 summarize all the submissions listed in the automatic evaluation server at the time ofthe WAT2016 workshop (12th, December, 2016). The OTHER RESOURCES column shows the use ofresources such as parallel corpora, monolingual corpora and parallel dictionaries in addition to ASPEC,JPC, BPPT Corpus, IITB Corpus.
13
Figure 2: Official evaluation results of ASPEC-JE.
14
Figure 3: Official evaluation results of ASPEC-EJ.
15
Figure 4: Official evaluation results of ASPEC-JC.
16
Figure 5: Official evaluation results of ASPEC-CJ.
17
Figure 6: Official evaluation results of JPC-JE.
18
Figure 7: Official evaluation results of JPC-EJ.
19
Figure 8: Official evaluation results of JPC-JC.
20
Figure 9: Official evaluation results of JPC-CJ.
21
Figure 10: Official evaluation results of JPC-KJ.
22
Figure 11: Official evaluation results of BPPT-IE.
23
Figure 12: Official evaluation results of BPPT-EI.
24
Figure 13: Official evaluation results of IITB-EH.
25
Figure 14: Official evaluation results of IITB-HJ.
26
Annotator A Annotator B all weightedSYSTEM ID average variance average variance average κ κASPEC-JEKyoto-U 1 3.760 0.682 4.010 0.670 3.885 0.205 0.313NAIST 1 3.705 0.728 3.950 0.628 3.828 0.257 0.356NICT-2 3.025 0.914 3.360 0.740 3.193 0.199 0.369ASPEC-EJKyoto-U 1 3.970 0.759 4.065 0.851 4.018 0.346 0.494bjtu nlp 3.800 0.980 3.625 1.364 3.713 0.299 0.509NICT-2 3.745 0.820 3.670 0.931 3.708 0.299 0.486Online A 3.600 0.770 3.590 0.862 3.595 0.273 0.450ASPEC-JCKyoto-U 1 3.995 1.095 3.755 1.145 3.875 0.203 0.362bjtu nlp 3.920 1.054 3.340 1.244 3.630 0.154 0.290NICT-2 2.940 1.846 2.850 1.368 2.895 0.237 0.477ASPEC-CJKyoto-U 1 4.245 1.045 3.635 1.232 3.940 0.234 0.341UT-KAY 1 3.995 1.355 3.370 1.143 3.683 0.152 0.348bjtu nlp 3.950 1.278 3.330 1.221 3.640 0.179 0.401JPC-JEbjtu nlp 4.085 0.798 4.505 0.580 4.295 0.254 0.393Online A 3.910 0.652 4.300 0.830 4.105 0.166 0.336NICT-2 1 3.705 1.118 4.155 1.011 3.930 0.277 0.458JPC-EJNICT-2 1 4.025 0.914 4.510 0.570 4.268 0.234 0.412bjtu nlp 3.920 0.924 4.470 0.749 4.195 0.151 0.340JAPIO 1 4.055 0.932 4.250 0.808 4.153 0.407 0.562JPC-JCbjtu nlp 3.485 1.720 3.015 1.755 3.250 0.274 0.507NICT-2 1 3.230 1.867 2.935 1.791 3.083 0.307 0.492S2T 2.745 2.000 2.680 1.838 2.713 0.305 0.534JPC-CJntt 1 3.605 1.889 3.265 1.765 3.435 0.263 0.519JAPIO 1 3.385 1.947 3.085 2.088 3.235 0.365 0.592NICT-2 1 3.410 1.732 3.045 1.883 3.228 0.322 0.518JPC-KJJAPIO 1 4.580 0.324 4.660 0.304 4.620 0.328 0.357EHR 1 4.510 0.380 4.615 0.337 4.563 0.424 0.478Online A 4.380 0.466 4.475 0.409 4.428 0.517 0.574BPPT-IEOnline A 2.675 0.489 3.375 1.564 3.025 0.048 0.187Sense 1 2.685 0.826 2.420 1.294 2.553 0.242 0.408IITB-EN-ID 2.485 0.870 2.345 1.216 2.415 0.139 0.324BPPT-EIOnline A 2.890 1.778 3.375 1.874 3.133 0.163 0.446Sense 1 2.395 1.059 2.450 1.328 2.423 0.305 0.494IITB-EN-ID 2.185 1.241 2.360 1.130 2.273 0.246 0.477IITB-EHOnline A 3.200 1.330 3.525 1.189 3.363 0.103 0.155EHR 2.590 1.372 1.900 0.520 2.245 0.136 0.263IITP-MT 2.350 1.198 1.780 0.362 2.065 0.066 0.164IITB-HJOnline A 1.955 1.563 2.310 0.664 2.133 0.120 0.287EHR 1 1.530 1.049 2.475 0.739 2.003 0.055 0.194
Table 8: JPO adequacy evaluation results in detail.
27
NA
IST
1
NA
IST
2
Kyo
to-U
1
Kyo
to-U
2
Kyo
to-U
(201
5)
TOSH
IBA
(201
5)
NIC
T-2
Onl
ine
D
TM
U1
bjtu
nlp
TM
U2
NAIST (2015) ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫NAIST 1 - - ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫NAIST 2 - ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫Kyoto-U 1 > ≫ ≫ ≫ ≫ ≫ ≫ ≫Kyoto-U 2 ≫ ≫ ≫ ≫ ≫ ≫ ≫Kyoto-U (2015) ≫ ≫ ≫ ≫ ≫ ≫TOSHIBA (2015) > ≫ ≫ ≫ ≫NICT-2 ≫ ≫ ≫ ≫Online D - ≫ ≫TMU 1 ≫ ≫bjtu nlp -
Table 9: Statistical significance testing of the ASPEC-JE Pairwise scores.
Kyo
to-U
nave
r(20
15)
Onl
ine
A
WE
BL
IOM
T(2
015)
NIC
T-2
bjtu
nlp
EH
R
UT-
AK
Y1
TOK
YO
MT
1
TOK
YO
MT
2
UT-
AK
Y2
JAPI
O
NAIST (2015) ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫Kyoto-U - ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫naver (2015) ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫Online A > ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫WEBLIO MT (2015) ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫NICT-2 - - ≫ ≫ ≫ ≫ ≫bjtu nlp - > ≫ ≫ ≫ ≫EHR - ≫ ≫ ≫ ≫UT-AKY 1 ≫ ≫ ≫ ≫TOKYOMT 1 - ≫ ≫TOKYOMT 2 ≫ ≫UT-AKY 2 ≫
Table 10: Statistical significance testing of the ASPEC-EJ Pairwise scores.
NA
IST
(201
5)
bjtu
nlp
Kyo
to-U
(201
5)
Kyo
to-U
2
NIC
T-2
TOSH
IBA
(201
5)
Onl
ine
D
Kyoto-U 1 ≫ ≫ ≫ ≫ ≫ ≫ ≫NAIST (2015) ≫ ≫ ≫ ≫ ≫ ≫bjtu nlp ≫ ≫ ≫ ≫ ≫Kyoto-U (2015) - ≫ ≫ ≫Kyoto-U 2 ≫ ≫ ≫NICT-2 ≫ ≫TOSHIBA (2015) ≫
Table 11: Statistical significance testing of the ASPEC-JC Pairwise scores.
28
Kyo
to-U
2
bjtu
nlp
UT-
KA
Y1
UT-
KA
Y2
NA
IST
(201
5)
NIC
T-2
Kyo
to-U
(201
5)
EH
R
EH
R(2
015)
JAPI
O
Onl
ine
A
Kyoto-U 1 ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫Kyoto-U 2 ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫bjtu nlp - ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫UT-KAY 1 ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫UT-KAY 2 - ≫ ≫ ≫ ≫ ≫ ≫NAIST (2015) > ≫ ≫ ≫ ≫ ≫NICT-2 - ≫ ≫ ≫ ≫Kyoto-U (2015) - ≫ ≫ ≫EHR ≫ ≫ ≫EHR (2015) ≫ ≫JAPIO ≫
Table 12: Statistical significance testing of the ASPEC-CJ Pairwise scores.O
nlin
eA
NIC
T-2
1
NIC
T-2
2
RB
MT
A
S2T
SMT
Hie
ro
bjtu nlp ≫ ≫ ≫ ≫ ≫ ≫Online A ≫ ≫ ≫ ≫ ≫NICT-2 1 - - - ≫NICT-2 2 - - ≫RBMT A - ≫SMT S2T ≫
Table 13: Statistical significance testing of the JPC-JE Pairwise scores.
NIC
T-2
1
SMT
T2S
NIC
T-2
2
JAPI
O1
SMT
Hie
ro
Onl
ine
A
JAPI
O2
RB
MT
F
bjtu nlp - ≫ ≫ ≫ ≫ ≫ ≫ ≫NICT-2 1 ≫ ≫ ≫ ≫ ≫ ≫ ≫SMT T2S - > ≫ ≫ ≫ ≫NICT-2 2 > ≫ ≫ ≫ ≫JAPIO 1 ≫ ≫ ≫ ≫SMT Hiero - > ≫Online A - ≫JAPIO 2 ≫
Table 14: Statistical significance testing of the JPC-EJ Pairwise scores.
SMT
Hie
ro
SMT
S2T
bjtu
nlp
NIC
T-2
2
Onl
ine
A
RB
MT
C
NICT-2 1 ≫ ≫ ≫ ≫ ≫ ≫SMT Hiero - ≫ ≫ ≫ ≫SMT S2T ≫ ≫ ≫ ≫bjtu nlp ≫ ≫ ≫NICT-2 2 ≫ ≫Online A ≫
Table 15: Statistical significance testing of the JPC-JC Pairwise scores.
29
JAPI
O1
JAPI
O2
NIC
T-2
1
EH
R(2
015)
ntt2
EH
R1
NIC
T-2
2
EH
R2
bjtu
nlp
Kyo
to-U
(201
5)
TOSH
IBA
(201
5)
Onl
ine
A
ntt 1 - > > ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫JAPIO 1 ≫ > ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫JAPIO 2 - - ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫NICT-2 1 - ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫EHR (2015) - - ≫ ≫ ≫ ≫ ≫ ≫ntt 2 - - > ≫ ≫ ≫ ≫EHR 1 - ≫ ≫ ≫ ≫ ≫NICT-2 2 - ≫ ≫ ≫ ≫EHR 2 > > ≫ ≫bjtu nlp - - ≫Kyoto-U (2015) - ≫TOSHIBA (2015) ≫
Table 16: Statistical significance testing of the JPC-CJ Pairwise scores.
TOSH
IBA
(201
5)1
JAPI
O1
TOSH
IBA
(201
5)2
NIC
T(2
015)
1
nave
r(20
15)1
NIC
T(2
015)
2
Onl
ine
A
nave
r(20
15)2
Sens
e(2
015)
1
EH
R(2
015)
1
EH
R2
EH
R(2
015)
2
JAPI
O2
Sens
e(2
015)
2
EHR 1 ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫TOSHIBA (2015) 1 - - ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫JAPIO 1 - ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫TOSHIBA (2015) 2 ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫NICT (2015) 1 - - - - ≫ ≫ ≫ ≫ ≫ ≫naver (2015) 1 - - - ≫ ≫ ≫ ≫ ≫ ≫NICT (2015) 2 - - ≫ ≫ ≫ ≫ ≫ ≫Online A - ≫ ≫ ≫ ≫ ≫ ≫naver (2015) 2 ≫ ≫ ≫ ≫ ≫ ≫Sense (2015) 1 ≫ ≫ ≫ ≫ ≫EHR (2015) 1 - - ≫ ≫EHR 2 - ≫ ≫EHR (2015) 2 ≫ ≫JAPIO 2 ≫
Table 17: Statistical significance testing of the JPC-KJ Pairwise scores.
Onl
ine
B
SMT
S2T
Sens
e1
SMT
Hie
ro
Sens
e2
IIT
B-E
N-I
D
Online A ≫ ≫ ≫ ≫ ≫ ≫Online B ≫ ≫ ≫ ≫ ≫SMT S2T - ≫ ≫ ≫Sense 1 > > ≫SMT Hiero - ≫Sense 2 ≫
Onl
ine
B
Sens
e1
Sens
e2
SMT
T2S
IIT
B-E
N-I
D
SMT
Hie
ro
Online A ≫ ≫ ≫ ≫ ≫ ≫Online B ≫ ≫ ≫ ≫ ≫Sense 1 > ≫ ≫ ≫Sense 2 ≫ ≫ ≫SMT T2S - ≫IITB-EN-ID ≫
Table 18: Statistical significance testing of the BPPT-IE (left) and BPPT-EI (right) Pairwise scores.
30
Onl
ine
B
IIT
P-M
T
EH
R
Online A ≫ ≫ ≫Online B ≫ ≫IITP-MT ≫
Onl
ine
B
EH
R1
EH
R2
Online A ≫ ≫ ≫Online B ≫ ≫EHR 1 ≫
Table 19: Statistical significance testing of the IITB-EH (left) and IITB-HJ (right) Pairwise scores.
ASPEC-JESYSTEM ID κNAIST (2015) 0.078NAIST 1 0.081NAIST 2 0.091Kyoto-U 1 0.106Kyoto-U 2 0.148Kyoto-U (2015) 0.066TOSHIBA (2015) 0.068NICT-2 0.106Online D 0.081TMU 1 0.060bjtu nlp 0.146TMU 2 0.072ave. 0.092
ASPEC-EJSYSTEM ID κNAIST (2015) 0.239Kyoto-U 0.215naver (2015) 0.187Online A 0.181WEBLIO MT (2015) 0.193NICT-2 0.177bjtu nlp 0.247EHR 0.195UT-AKY 1 0.204TOKYOMT 1 0.189TOKYOMT 2 0.200UT-AKY 2 0.201JAPIO 0.183ave 0.201
ASPEC-JCSYSTEM ID κKyoto-U 1 0.177NAIST (2015) 0.221bjtu nlp 0.187Kyoto-U (2015) 0.197Kyoto-U 2 0.251NICT-2 0.190TOSHIBA (2015) 0.214Online D 0.180ave. 0.202
ASPEC-CJSYSTEM ID κKyoto-U 1 0.195Kyoto-U 2 0.151bjtu nlp 0.168UT-KAY 1 0.172UT-KAY 2 0.156NAIST (2015) 0.089NICT-2 0.168Kyoto-U (2015) 0.144EHR 0.152EHR (2015) 0.190JAPIO 0.185Online A 0.207ave. 0.165
JPC-JESYSTEM ID κbjtu nlp 0.256Online A 0.242NICT-2 1 0.280NICT-2 2 0.293RBMT A 0.179S2T 0.296Hiero 0.324ave. 0.267
JPC-EJSYSTEM ID κbjtu nlp 0.339NICT-2 1 0.367T2S 0.378NICT-2 2 0.346JAPIO 1 0.323Hiero 0.383Online A 0.403JAPIO 2 0.336RBMT F 0.323ave. 0.355
JPC-JCSYSTEM ID κNICT-2 1 0.076Hiero 0.127S2T 0.133bjtu nlp 0.085NICT-2 2 0.068Online A 0.055RBMT C 0.116ave. 0.094
JPC-CJSYSTEM ID κntt 1 0.169JAPIO 1 0.121JAPIO 2 0.160NICT-2 1 0.150EHR (2015) 0.123ntt 2 0.114EHR 1 0.155NICT-2 2 0.151EHR 2 0.150bjtu nlp 0.200Kyoto-U (2015) 0.096TOSHIBA (2015) 0.131Online A 0.116ave. 0.141
JPC-KJSYSTEM ID κEHR 1 0.256TOSHIBA (2015) 1 0.221JAPIO 1 0.228TOSHIBA (2015) 2 0.176NICT (2015) 1 0.351naver (2015) 1 0.469NICT (2015) 2 0.345Online A 0.232naver (2015) 2 0.299Sense (2015) 1 0.522EHR (2015) 1 0.363EHR 2 0.399EHR (2015) 2 0.373JAPIO 2 0.260Sense (2015) 2 0.329ave. 0.322
BPPT-IESYSTEM ID κOnline A -0.083Online B -0.051S2T 0.025Sense 1 0.145Hiero 0.057Sense 2 0.102IITB-EN-ID 0.063ave. 0.037
BPPT-EISYSTEM ID κOnline A 0.094Online B 0.063Sense 1 0.135Sense 2 0.160T2S 0.089IITB-EN-ID 0.115Hiero 0.165ave. 0.117
IITB-EHSYSTEM ID κOnline A 0.141Online B 0.110IITP-MT 0.215EHR 0.196ave. 0.166
IITB-HJSYSTEM ID κOnline A 0.285Online B 0.488EHR 1 0.452EHR 2 0.510ave. 0.434
Table 20: The Fleiss’ kappa values for the pairwise evaluation results.
31
Figure 15: The chronological evaluation results of JPO adequacy evaluation.
32
SYST
EM
IDID
ME
TH
OD
OT
HE
RR
ESO
UR
CE
SB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
SMT
Hie
ro2
SMT
NO
18.7
20.
6510
660.
5888
80—
–H
iera
rchi
calP
hras
e-ba
sed
SMT
SMT
Phra
se6
SMT
NO
18.4
50.
6451
370.
5909
50—
–Ph
rase
-bas
edSM
TSM
TS2
T87
7SM
TN
O20
.36
0.67
8253
0.59
3410
+7.0
0St
ring
-to-
Tree
SMT
RB
MT
D88
7O
ther
YE
S15
.29
0.68
3378
0.55
1690
+16.
75R
BM
TD
RB
MT
E76
Oth
erY
ES
14.8
20.
6638
510.
5616
20—
–R
BM
TE
RB
MT
F79
Oth
erY
ES
13.8
60.
6613
870.
5568
40—
–R
BM
TF
Onl
ine
C(2
014)
87O
ther
YE
S10
.64
0.62
4827
0.46
6480
—–
Onl
ine
C(2
014)
Onl
ine
D(2
014)
35O
ther
YE
S15
.08
0.64
3588
0.56
4170
—–
Onl
ine
D(2
014)
Onl
ine
D(2
015)
775
Oth
erY
ES
16.8
50.
6766
090.
5622
70+0
.25
Onl
ine
D(2
015)
Onl
ine
D10
42O
ther
YE
S16
.91
0.67
7412
0.56
4270
+28.
00O
nlin
eD
(201
6)N
AIS
T1
1122
SMT
NO
26.3
90.
7627
120.
5874
50+4
8.25
Neu
ralM
Tw
/Lex
icon
and
Min
Ris
kTr
aini
ng4
Ens
embl
eN
AIS
T2
1247
SMT
NO
26.1
20.
7569
560.
5713
60+4
7.50
Neu
ralM
Tw
/Lex
icon
6E
nsem
ble
Kyo
to-U
111
82N
MT
NO
26.2
20.
7566
010.
5585
40+4
4.25
Ens
embl
eof
4si
ngle
-lay
erm
odel
(30k
voc)
Kyo
to-U
212
46N
MT
NO
24.7
10.
7508
020.
5626
50+4
7.00
voc
src:
200k
voc
tgt:
52k
+B
PE2-
laye
rsel
f-en
sem
blin
gT
MU
112
22N
MT
NO
18.2
90.
7106
130.
5652
70+1
6.00
2016
ourp
ropo
sed
met
hod
toco
ntro
lout
putv
oice
TM
U2
1234
NM
TN
O18
.45
0.71
1542
0.54
6880
+25.
006
ense
mbl
eB
JTU
-nlp
111
68N
MT
NO
18.3
40.
6904
550.
5057
30+1
9.25
RN
NE
ncod
er-D
ecod
erw
ithat
tent
ion
mec
hani
sm,s
ingl
em
odel
NIC
T-2
111
04SM
TY
ES
21.5
40.
7088
080.
5959
30—
–Ph
rase
-bas
edSM
Tw
ithPr
eord
erin
g+
Dom
ain
Ada
ptat
ion
(JPC
and
ASP
EC
)+
Goo
gle
5-gr
amL
M
Tabl
e21
:ASP
EC
-JE
subm
issi
ons
33
SYST
EM
IDID
ME
TH
OD
OT
HE
RB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
RE
SOU
RC
ES
jum
anky
tea
mec
abju
man
kyte
am
ecab
jum
anky
tea
mec
abSM
TPh
rase
5SM
TN
O27
.48
29.8
028
.27
0.68
3735
0.69
1926
0.69
5390
0.73
6380
0.73
6380
0.73
6380
—–
Phra
se-b
ased
SMT
SMT
Hie
ro36
7SM
TN
O30
.19
32.5
630
.94
0.73
4705
0.74
6978
0.74
7722
0.74
3900
0.74
3900
0.74
3900
+31.
50H
iera
rchi
calP
hras
e-ba
sed
SMT
SMT
T2S
875
SMT
NO
31.0
533
.44
32.1
00.
7488
830.
7580
310.
7605
160.
7443
700.
7443
700.
7443
70+3
0.00
Tree
-to-
Stri
ngSM
TR
BM
TA
68O
ther
YE
S12
.86
14.4
313
.16
0.67
0167
0.67
6464
0.67
8934
0.62
6940
0.62
6940
0.62
6940
—–
RB
MT
AR
BM
TB
883
Oth
erY
ES
13.1
814
.85
13.4
80.
6719
580.
6807
480.
6826
830.
6229
300.
6229
300.
6229
30+9
.75
RB
MT
BR
BM
TC
95O
ther
YE
S12
.19
13.3
212
.14
0.66
8372
0.67
2645
0.67
6018
0.59
4380
0.59
4380
0.59
4380
—–
RB
MT
CO
nlin
eA
(201
4)34
Oth
erY
ES
19.6
621
.63
20.1
70.
7180
190.
7234
860.
7258
480.
6954
200.
6954
200.
6954
20—
–O
nlin
eA
(201
4)O
nlin
eA
(201
5)77
4O
ther
YE
S18
.22
19.7
718
.46
0.70
5882
0.71
3960
0.71
8150
0.67
7200
0.67
7200
0.67
7200
+34.
25O
nlin
eA
(201
5)O
nlin
eA
(201
6)10
41O
ther
YE
S18
.28
19.8
118
.51
0.70
6639
0.71
5222
0.71
8559
0.67
7020
0.67
7020
0.67
7020
+49.
75O
nlin
eA
(201
6)O
nlin
eB
(201
4)91
Oth
erY
ES
17.0
418
.67
17.3
60.
6877
970.
6933
900.
6981
260.
6430
700.
6430
700.
6430
70—
–O
nlin
eB
(201
4)O
nlin
eB
(201
5)88
9O
ther
YE
S17
.80
19.5
218
.11
0.69
3359
0.70
1966
0.70
3859
0.64
6160
0.64
6160
0.64
6160
—–
Onl
ine
B(2
015)
Kyo
to-U
111
72N
MT
NO
36.1
938
.20
36.7
80.
8198
360.
8238
780.
8289
560.
7387
000.
7387
000.
7387
00+5
5.25
BPE
tgt/s
rc:
52k
2-la
yer
lstm
self
-en
sem
ble
of3
EH
R1
1140
SMT
NO
31.3
233
.58
32.2
80.
7599
140.
7714
270.
7750
230.
7467
200.
7467
200.
7467
20+3
9.00
PBSM
Tw
ithpr
eord
erin
g(D
L=6
)B
JTU
-nlp
111
43N
MT
NO
31.1
833
.47
31.8
00.
7805
100.
7874
970.
7910
880.
7043
400.
7043
400.
7043
40+3
9.50
RN
NE
ncod
er-D
ecod
erw
ithat
ten-
tion
mec
hani
sm,s
ingl
em
odel
TOK
YO
MT
111
31N
MT
NO
30.2
133
.38
31.2
40.
8096
910.
8172
580.
8199
510.
7052
100.
7052
100.
7052
10+2
9.75
char
1,e
ns2
,ver
sion
1TO
KY
OM
T2
1217
NM
TN
O32
.03
34.7
732
.98
0.80
8189
0.81
4452
0.81
8130
0.72
0810
0.72
0810
0.72
0810
+30.
50C
ombi
natio
nof
NM
Tan
dT
2SJA
PIO
111
65SM
TY
ES
20.5
222
.56
21.0
50.
7234
670.
7285
840.
7314
740.
6607
900.
6607
900.
6607
90+4
.25
Phra
se-b
ased
SMT
with
Preo
rder
-in
g+
JAPI
Oco
rpus
+ru
le-b
ased
post
edito
rN
ICT-
21
1097
SMT
YE
S34
.67
36.8
635
.37
0.78
4335
0.79
0993
0.79
3409
0.75
3080
0.75
3080
0.75
3080
+41.
25Ph
rase
-bas
edSM
Tw
ithPr
eord
er-
ing
+D
omai
nA
dapt
atio
n(J
PCan
dA
SPE
C)+
Goo
gle
5-gr
amL
MU
T-A
KY
112
24N
MT
NO
30.1
433
.20
31.0
90.
8060
250.
8144
900.
8158
360.
7081
400.
7081
400.
7081
40+2
1.75
tree
-to-
seq
NM
Tm
odel
(cha
ract
er-
base
dde
code
r)U
T-A
KY
212
28N
MT
NO
33.5
736
.95
34.6
50.
8169
840.
8244
560.
8276
470.
7314
400.
7314
400.
7314
40+3
6.25
tree
-to-
seq
NM
Tm
odel
(wor
d-ba
sed
deco
der)
Tabl
e22
:ASP
EC
-EJ
subm
issi
ons
34
SYST
EM
IDID
ME
TH
OD
OT
HE
RB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
RE
SOU
RC
ES
kyte
ast
anfo
rd(c
tb)
stan
ford
(pku
)ky
tea
stan
ford
(ctb
)st
anfo
rd(p
ku)
kyte
ast
anfo
rd(c
tb)
stan
ford
(pku
)SM
TPh
rase
7SM
TN
O27
.96
28.0
127
.68
0.78
8961
0.79
0263
0.79
0937
0.74
9450
0.74
9450
0.74
9450
—–
Phra
se-b
ased
SMT
SMT
Hie
ro3
SMT
NO
27.7
127
.70
27.3
50.
8091
280.
8095
610.
8113
940.
7451
000.
7451
000.
7451
00—
–H
iera
rchi
calP
hras
e-ba
sed
SMT
SMT
S2T
881
SMT
NO
28.6
528
.65
28.3
50.
8076
060.
8094
570.
8084
170.
7552
300.
7552
300.
7552
30+7
.75
Stri
ng-t
o-Tr
eeSM
TR
BM
TB
886
Oth
erY
ES
17.8
617
.75
17.4
90.
7448
180.
7458
850.
7437
940.
6679
600.
6679
600.
6679
60-1
1.00
RB
MT
BR
BM
TC
244
Oth
erN
O9.
629.
969.
590.
6422
780.
6487
580.
6453
850.
5949
000.
5949
000.
5949
00—
–R
BM
TC
Onl
ine
C(2
014)
216
Oth
erY
ES
7.26
7.01
6.72
0.61
2808
0.61
3075
0.61
1563
0.58
7820
0.58
7820
0.58
7820
—–
Onl
ine
C(2
014)
Onl
ine
C(2
015)
891
Oth
erY
ES
7.44
7.05
6.75
0.61
1964
0.61
5048
0.61
2158
0.56
6060
0.56
6060
0.56
6060
—–
Onl
ine
C(2
015)
Onl
ine
D(2
014)
37O
ther
YE
S9.
378.
938.
840.
6069
050.
6063
280.
6041
490.
6254
300.
6254
300.
6254
30—
–O
nlin
eD
(201
4)O
nlin
eD
(201
5)77
7O
ther
YE
S10
.73
10.3
310
.08
0.66
0484
0.66
0847
0.66
0482
0.63
4090
0.63
4090
0.63
4090
-14.
75O
nlin
eD
(201
5)O
nlin
eD
(201
6)10
45O
ther
YE
S11
.16
10.7
210
.54
0.66
5185
0.66
7382
0.66
6953
0.63
9440
0.63
9440
0.63
9440
-26.
00O
nlin
eD
(201
6)K
yoto
-U1
1071
NM
TN
O31
.98
32.0
831
.72
0.83
7579
0.83
9354
0.83
5932
0.76
3290
0.76
3290
0.76
3290
+58.
752
laye
rls
tmdr
opou
t0.
520
0kso
urce
voc
unk
repl
aced
Kyo
to-U
211
09E
BM
TN
O30
.27
29.9
429
.92
0.81
3114
0.81
3581
0.81
3054
0.76
4230
0.76
4230
0.76
4230
+30.
75K
yoto
EB
MT
2016
w/o
rera
nkin
gB
JTU
-nlp
111
20N
MT
NO
30.5
730
.49
30.3
10.
8296
790.
8291
130.
8276
370.
7546
900.
7546
900.
7546
90+4
6.25
RN
NE
ncod
er-D
ecod
erw
ithat
ten-
tion
mec
hani
sm,s
ingl
em
odel
NIC
T-2
111
05SM
TY
ES
30.0
029
.97
29.7
80.
8208
910.
8200
690.
8210
900.
7596
700.
7596
700.
7596
70+2
4.00
Phra
se-b
ased
SMT
with
Preo
rder
-in
g+
Dom
ain
Ada
ptat
ion
(JPC
and
ASP
EC
)
Tabl
e23
:ASP
EC
-JC
subm
issi
ons
35
SYST
EM
IDID
ME
TH
OD
OT
HE
RB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
RE
SOU
RC
ES
jum
anky
tea
mec
abju
man
kyte
am
ecab
jum
anky
tea
mec
abSM
TPh
rase
8SM
TN
O34
.65
35.1
634
.77
0.77
2498
0.76
6384
0.77
1005
0.75
3010
0.75
3010
0.75
3010
—–
Phra
se-b
ased
SMT
SMT
Hie
ro4
SMT
NO
35.4
335
.91
35.6
40.
8104
060.
7987
260.
8076
650.
7509
500.
7509
500.
7509
50—
–H
iera
rchi
calP
hras
e-ba
sed
SMT
SMT
T2S
879
SMT
NO
36.5
237
.07
36.6
40.
8252
920.
8204
900.
8250
250.
7548
700.
7548
700.
7548
70+1
7.25
Tree
-to-
Stri
ngSM
TR
BM
TA
885
Oth
erY
ES
9.37
9.87
9.35
0.66
6277
0.65
2402
0.66
1730
0.62
6070
0.62
6070
0.62
6070
-28.
00R
BM
TA
RB
MT
D24
2O
ther
NO
8.39
8.70
8.30
0.64
1189
0.62
6400
0.63
3319
0.58
6790
0.58
6790
0.58
6790
—–
RB
MT
DO
nlin
eA
(201
4)36
Oth
erY
ES
11.6
313
.21
11.8
70.
5959
250.
5981
720.
5985
730.
6580
600.
6580
600.
6580
60—
–O
nlin
eA
(201
4)O
nlin
eA
(201
5)77
6O
ther
YE
S11
.53
12.8
211
.68
0.58
8285
0.59
0393
0.59
2887
0.64
9860
0.64
9860
0.64
9860
-19.
00O
nlin
eA
(201
5)O
nlin
eA
(201
6)10
43O
ther
YE
S11
.56
12.8
711
.69
0.58
9802
0.58
9397
0.59
3361
0.65
9540
0.65
9540
0.65
9540
-51.
25O
nlin
eA
(201
6)O
nlin
eB
(201
4)21
5O
ther
YE
S10
.48
11.2
610
.47
0.60
0733
0.59
6006
0.60
0706
0.63
6930
0.63
6930
0.63
6930
—–
Onl
ine
B(2
014)
Onl
ine
B(2
015)
890
Oth
erY
ES
10.4
111
.03
10.3
60.
5973
550.
5928
410.
5972
980.
6282
900.
6282
900.
6282
90—
–O
nlin
eB
(201
5)K
yoto
-U1
1255
NM
TN
O44
.29
45.0
544
.32
0.86
9360
0.86
4748
0.86
9913
0.78
4380
0.78
4380
0.78
4380
+56.
00sr
c:20
0ktg
t:50
k2-
laye
rsse
lf-
ense
mbl
ing
Kyo
to-U
212
56N
MT
NO
46.0
446
.70
46.0
50.
8765
310.
8729
040.
8769
460.
7859
100.
7859
100.
7859
10+6
3.75
voc:
30k
ense
mbl
eof
3in
depe
n-de
ntm
odel
+re
vers
ere
scor
ing
EH
R1
1063
SMT
YE
S39
.75
39.8
539
.40
0.84
3723
0.83
6156
0.84
1952
0.76
9490
0.76
9490
0.76
9490
+32.
50L
M-b
ased
mer
ging
ofou
tput
sof
preo
rder
edw
ord-
base
dPB
-SM
T(D
L=6
)an
dpr
eord
ered
char
acte
r-ba
sed
PBSM
T(D
L=6
).B
JTU
-nlp
111
38N
MT
NO
38.8
339
.25
38.6
80.
8528
180.
8463
010.
8522
980.
7608
400.
7608
400.
7608
40+4
9.00
RN
NE
ncod
er-D
ecod
erw
ithat
ten-
tion
mec
hani
sm,s
ingl
em
odel
JAPI
O1
1208
SMT
YE
S26
.24
27.8
726
.37
0.79
0553
0.78
0637
0.78
5917
0.69
6770
0.69
6770
0.69
6770
+16.
50Ph
rase
-bas
edSM
Tw
ithPr
eord
er-
ing
+JA
PIO
corp
us+
rule
-bas
edpo
sted
itor
NIC
T-2
110
99SM
TY
ES
40.0
240
.45
40.2
90.
8439
410.
8377
070.
8425
130.
7685
800.
7685
800.
7685
80+3
6.50
Phra
se-b
ased
SMT
with
Preo
rder
-in
g+
Dom
ain
Ada
ptat
ion
(JPC
and
ASP
EC
)+G
oogl
e5-
gram
LM
UT-
KA
Y1
1220
NM
TN
O37
.63
39.0
737
.82
0.84
7407
0.84
2055
0.84
8040
0.75
3820
0.75
3820
0.75
3820
+41.
00A
nen
d-to
-end
NM
Tw
ith51
2di
men
sion
alsi
ngle
-lay
erL
STM
s,U
NK
repl
acem
ent,
and
dom
ain
adap
tatio
nU
T-K
AY
212
21N
MT
NO
40.5
041
.81
40.6
70.
8602
140.
8546
900.
8604
490.
7655
300.
7655
300.
7655
30+4
7.25
Ens
embl
eof
ourN
MT
mod
els
with
and
with
outd
omai
nad
apta
tion
Tabl
e24
:ASP
EC
-CJ
subm
issi
ons
36
SYST
EM
IDID
ME
TH
OD
OT
HE
RR
ESO
UR
CE
SB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
SMT
Phra
se97
7SM
TN
O30
.80
0.73
0056
0.66
4830
—–
Phra
se-b
ased
SMT
SMT
Hie
ro97
9SM
TN
O32
.23
0.76
3030
0.67
2500
+8.7
5H
iera
rchi
calP
hras
e-ba
sed
SMT
SMT
S2T
980
SMT
NO
34.4
00.
7934
830.
6727
60+2
3.00
Stri
ng-t
o-Tr
eeSM
TR
BM
TA
1090
Oth
erY
ES
21.5
70.
7503
810.
5212
30+2
3.75
RB
MT
AR
BM
TB
1095
Oth
erY
ES
18.3
80.
7109
920.
5181
10—
–R
BM
TB
RB
MT
C10
88O
ther
YE
S21
.00
0.75
5017
0.51
9210
—–
RB
MT
CO
nlin
eA
(201
6)10
35O
ther
YE
S35
.77
0.80
3661
0.67
3950
+32.
25O
nlin
eA
(201
6)O
nlin
eB
(201
6)10
51O
ther
YE
S16
.00
0.68
8004
0.48
6450
—–
Onl
ine
B(2
016)
BJT
U-n
lp1
1149
NM
TN
O41
.62
0.85
1975
0.69
0750
+41.
50R
NN
Enc
oder
-Dec
oder
with
atte
ntio
nm
echa
nism
,sin
gle
mod
elN
ICT-
21
1080
SMT
NO
35.6
80.
8243
980.
6675
40+2
5.00
Phra
se-b
ased
SMT
with
Preo
rder
ing
+D
omai
nA
dapt
atio
nN
ICT-
22
1103
SMT
YE
S36
.06
0.82
5420
0.67
2890
+24.
25Ph
rase
-bas
edSM
Tw
ithPr
eord
erin
g+
Dom
ain
Ada
ptat
ion
(JPC
and
ASP
EC
)+
Goo
gle
5-gr
amL
M
Tabl
e25
:JPC
-JE
subm
issi
ons
SYST
EM
IDID
ME
TH
OD
OT
HE
RB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
RE
SOU
RC
ES
jum
anky
tea
mec
abju
man
kyte
am
ecab
jum
anky
tea
mec
abSM
TPh
rase
973
SMT
NO
32.3
634
.26
32.5
20.
7285
390.
7282
810.
7290
770.
7119
000.
7119
000.
7119
00—
–Ph
rase
-bas
edSM
TSM
TH
iero
974
SMT
NO
34.5
736
.61
34.7
90.
7777
590.
7786
570.
7790
490.
7153
000.
7153
000.
7153
00+2
1.00
Hie
rarc
hica
lPhr
ase-
base
dSM
TSM
TT
2S97
5SM
TN
O35
.60
37.6
535
.82
0.79
7353
0.79
6783
0.79
8025
0.71
7030
0.71
7030
0.71
7030
+30.
75Tr
ee-t
o-St
ring
SMT
RB
MT
D10
85O
ther
YE
S23
.02
24.9
023
.45
0.76
1224
0.75
7341
0.76
0325
0.64
7730
0.64
7730
0.64
7730
—–
RB
MT
DR
BM
TE
1087
Oth
erY
ES
21.3
523
.17
21.5
30.
7434
840.
7419
850.
7423
000.
6469
300.
6469
300.
6469
30—
–R
BM
TE
RB
MT
F10
86O
ther
YE
S26
.64
28.4
826
.84
0.77
3673
0.76
9244
0.77
3344
0.67
5470
0.67
5470
0.67
5470
+12.
75R
BM
TF
Onl
ine
A(2
016)
1036
Oth
erY
ES
36.8
837
.89
36.8
30.
7981
680.
7924
710.
7963
080.
7191
100.
7191
100.
7191
10+2
0.00
Onl
ine
A(2
016)
Onl
ine
B(2
016)
1073
Oth
erY
ES
21.5
722
.62
21.6
50.
7430
830.
7352
030.
7409
620.
6599
500.
6599
500.
6599
50—
–O
nlin
eB
(201
6)B
JTU
-nlp
111
12N
MT
NO
39.4
641
.16
39.4
50.
8427
620.
8401
480.
8426
690.
7225
600.
7225
600.
7225
60+3
9.50
RN
NE
ncod
er-D
ecod
erw
ithat
ten-
tion
mec
hani
sm,s
ingl
em
odel
JAPI
O1
1141
SMT
YE
S45
.57
46.4
045
.74
0.85
1376
0.84
8580
0.84
9513
0.74
7910
0.74
7910
0.74
7910
+17.
75Ph
rase
-bas
edSM
Tw
ithPr
eord
er-
ing
+JA
PIO
corp
usJA
PIO
211
56SM
TY
ES
47.7
948
.57
47.9
20.
8591
390.
8563
920.
8574
220.
7628
500.
7628
500.
7628
50+2
6.75
Phra
se-b
ased
SMT
with
Preo
rder
-in
g+
JPC
/JA
PIO
corp
ora
NIC
T-2
110
78SM
TN
O39
.03
40.7
438
.98
0.82
6228
0.82
3582
0.82
4428
0.72
5540
0.72
5540
0.72
5540
+30.
75Ph
rase
-bas
edSM
Tw
ithPr
eord
er-
ing
+D
omai
nA
dapt
atio
nN
ICT-
22
1098
SMT
YE
S40
.90
42.5
140
.66
0.83
6556
0.83
2401
0.83
2622
0.73
8630
0.73
8630
0.73
8630
+37.
75Ph
rase
-bas
edSM
Tw
ithPr
eord
er-
ing
+D
omai
nA
dapt
atio
n(J
PCan
dA
SPE
C)+
Goo
gle
5-gr
amL
M
Tabl
e26
:JPC
-EJ
subm
issi
ons
37
SYST
EM
IDID
ME
TH
OD
OT
HE
RB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
RE
SOU
RC
ES
kyte
ast
anfo
rd(c
tb)
stan
ford
(pku
)ky
tea
stan
ford
(ctb
)st
anfo
rd(p
ku)
kyte
ast
anfo
rd(c
tb)
stan
ford
(pku
)SM
TPh
rase
966
SMT
NO
30.6
032
.03
31.2
50.
7873
210.
7978
880.
7943
880.
7109
400.
7109
400.
7109
40—
–Ph
rase
-bas
edSM
TSM
TH
iero
967
SMT
NO
30.2
631
.57
30.9
10.
7884
150.
7991
180.
7966
850.
7183
600.
7183
600.
7183
60+4
.75
Hie
rarc
hica
lPhr
ase-
base
dSM
TSM
TS2
T96
8SM
TN
O31
.05
32.3
531
.70
0.79
3846
0.80
2805
0.80
0848
0.72
0030
0.72
0030
0.72
0030
+4.2
5St
ring
-to-
Tree
SMT
RB
MT
C11
18O
ther
YE
S12
.35
13.7
213
.17
0.68
8240
0.70
8681
0.70
0210
0.47
5430
0.47
5430
0.47
5430
-41.
25R
BM
TC
Onl
ine
A10
38O
ther
YE
S23
.02
23.5
723
.29
0.75
4241
0.76
0672
0.76
0148
0.70
2350
0.70
2350
0.70
2350
-23.
00O
nlin
eA
(201
6)O
nlin
eB
1069
Oth
erY
ES
9.42
9.59
8.79
0.64
2026
0.65
1070
0.64
3520
0.52
7180
0.52
7180
0.52
7180
—–
Onl
ine
B(2
016)
BJT
U-n
lp1
1150
NM
TN
O31
.49
32.7
932
.51
0.81
6577
0.82
2978
0.82
0820
0.70
1490
0.70
1490
0.70
1490
-1.0
0R
NN
Enc
oder
-Dec
oder
with
atte
n-tio
nm
echa
nism
,sin
gle
mod
elN
ICT-
21
1081
SMT
NO
33.3
534
.64
33.8
10.
8085
130.
8179
960.
8153
220.
7232
700.
7232
700.
7232
70-1
1.00
Phra
se-b
ased
SMT
with
Preo
rder
-in
g+
Dom
ain
Ada
ptat
ion
NIC
T-2
211
06SM
TY
ES
33.4
034
.64
33.8
30.
8117
880.
8203
200.
8187
010.
7315
200.
7315
200.
7315
20+1
4.00
Phra
se-b
ased
SMT
with
Preo
rder
-in
g+
Dom
ain
Ada
ptat
ion
(JPC
and
ASP
EC
)
Tabl
e27
:JPC
-JC
subm
issi
ons
38
SYST
EM
IDID
ME
TH
OD
OT
HE
RB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
RE
SOU
RC
ES
jum
anky
tea
mec
abju
man
kyte
am
ecab
jum
anky
tea
mec
abSM
TPh
rase
431
SMT
NO
38.3
438
.51
38.2
20.
7820
190.
7789
210.
7814
560.
7231
100.
7231
100.
7231
10—
–Ph
rase
-bas
edSM
TSM
TH
iero
430
SMT
NO
39.2
239
.52
39.1
40.
8060
580.
8020
590.
8045
230.
7293
700.
7293
700.
7293
70—
–H
iera
rchi
calP
hras
e-ba
sed
SMT
SMT
T2S
432
SMT
NO
39.3
939
.90
39.3
90.
8149
190.
8113
500.
8135
950.
7259
200.
7259
200.
7259
20+2
0.75
Tree
-to-
Stri
ngSM
TR
BM
TA
759
Oth
erN
O10
.49
10.7
210
.35
0.67
4060
0.66
4098
0.66
7349
0.55
7130
0.55
7130
0.55
7130
-39.
25R
BM
TA
RB
MT
B76
0O
ther
NO
7.94
8.07
7.73
0.59
6200
0.58
1837
0.58
6941
0.50
2100
0.50
2100
0.50
2100
—–
RB
MT
BO
nlin
eA
(201
5)64
7O
ther
YE
S26
.80
27.8
126
.89
0.71
2242
0.70
7264
0.71
1273
0.69
3840
0.69
3840
0.69
3840
-7.0
0O
nlin
eA
(201
5)O
nlin
eA
(201
6)10
40O
ther
YE
S26
.99
27.9
127
.02
0.70
7739
0.70
2718
0.70
6707
0.69
3720
0.69
3720
0.69
3720
-19.
75O
nlin
eA
(201
6)O
nlin
eB
(201
5)64
8O
ther
YE
S12
.33
12.7
212
.44
0.64
8996
0.64
1255
0.64
8742
0.58
8380
0.58
8380
0.58
8380
—–
Onl
ine
B(2
015)
EH
R1
1007
SMT
YE
S40
.95
41.2
040
.51
0.82
8040
0.82
4502
0.82
6864
0.74
5080
0.74
5080
0.74
5080
+39.
00C
ombi
natio
nof
wor
d-ba
sed
PB-
SMT
and
char
acte
r-ba
sed
PBSM
Tw
ithD
L=6
.E
HR
210
09SM
Tan
dR
BM
TY
ES
41.0
541
.05
40.5
20.
8270
480.
8219
400.
8248
520.
7350
100.
7350
100.
7350
10+3
5.50
Com
bina
tion
ofw
ord-
base
dPB
-SM
T,ch
arac
ter-
base
dPB
SMT
and
RB
MT
+PB
SPE
with
DL
=6.
ntt1
1193
SMT
NO
40.7
541
.05
40.6
80.
8259
850.
8221
250.
8248
400.
7301
900.
7301
900.
7301
90+3
9.25
PBM
Tw
ithpr
e-or
deri
ngon
depe
n-de
ncy
stru
ctur
esnt
t212
00N
MT
NO
43.4
744
.27
43.5
30.
8452
710.
8431
050.
8449
680.
7492
700.
7492
700.
7492
70+4
6.50
NM
Tw
ithpr
e-or
deri
ngan
dat
ten-
tion
over
bidi
rect
iona
lLST
Ms
(pre
-or
deri
ngm
odul
eis
the
sam
eas
the
PBM
Tsu
bmis
sion
)B
JTU
-nlp
111
28N
MT
NO
39.3
439
.72
39.3
00.
8353
140.
8305
050.
8332
160.
7214
600.
7214
600.
7214
60+3
2.25
RN
NE
ncod
er-D
ecod
erw
ithat
ten-
tion
mec
hani
sm,s
ingl
em
odel
JAPI
O1
1180
SMT
YE
S43
.87
44.4
743
.66
0.83
3586
0.82
9360
0.83
1534
0.74
8330
0.74
8330
0.74
8330
+43.
50Ph
rase
-bas
edSM
Tw
ithPr
eord
er-
ing
+JA
PIO
corp
usJA
PIO
211
92SM
TY
ES
44.3
245
.12
44.0
90.
8349
590.
8301
640.
8329
550.
7512
000.
7512
000.
7512
00+4
6.25
Phra
se-b
ased
SMT
with
Preo
rder
-in
g+
JAPI
Oco
rpus
NIC
T-2
110
79SM
TN
O41
.09
41.2
741
.24
0.82
7009
0.82
2664
0.82
5323
0.73
3020
0.73
3020
0.73
3020
+36.
75Ph
rase
-bas
edSM
Tw
ithPr
eord
er-
ing
+D
omai
nA
dapt
atio
nN
ICT-
22
1100
SMT
YE
S41
.87
42.3
942
.13
0.82
9640
0.82
6744
0.82
8107
0.73
9890
0.73
9890
0.73
9890
+43.
25Ph
rase
-bas
edSM
Tw
ithPr
eord
er-
ing
+D
omai
nA
dapt
atio
n(J
PCan
dA
SPE
C)+
Goo
gle
5-gr
amL
M
Tabl
e28
:JPC
-CJ
subm
issi
ons
39
SYST
EM
IDID
ME
TH
OD
OT
HE
RR
ESO
UR
CE
SB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
SMT
Phra
se10
20SM
TN
O67
.09
0.93
3825
0.84
4950
—–
Phra
se-b
ased
SMT
SMT
Hie
ro10
21SM
TN
O66
.52
0.93
2391
0.84
4550
-3.5
0H
iera
rchi
calP
hras
e-ba
sed
SMT
RB
MT
C10
83O
ther
YE
S43
.26
0.87
2746
0.76
6520
—–
RB
MT
CR
BM
TD
1089
Oth
erY
ES
45.5
90.
8774
110.
7655
30-5
3.25
RB
MT
DO
nlin
eA
1037
Oth
erY
ES
48.7
50.
8989
760.
7913
20-2
1.00
Onl
ine
A(2
016)
Onl
ine
B10
68O
ther
YE
S28
.21
0.82
7843
0.69
2980
—–
Onl
ine
B(2
016)
Tabl
e29
:JPC
-JK
subm
issi
ons
SYST
EM
IDID
ME
TH
OD
OT
HE
RB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
RE
SOU
RC
ES
jum
anky
tea
mec
abju
man
kyte
am
ecab
jum
anky
tea
mec
abSM
TPh
rase
438
SMT
NO
69.2
270
.36
69.7
30.
9413
020.
9397
290.
9407
560.
8562
200.
8562
200.
8562
20—
–Ph
rase
-bas
edSM
TSM
TH
iero
439
SMT
NO
67.4
168
.65
68.0
00.
9371
620.
9359
030.
9365
700.
8505
600.
8505
600.
8505
60+2
.75
Hie
rarc
hica
lPhr
ase-
base
dSM
TR
BM
TA
653
Oth
erY
ES
42.0
043
.97
42.4
50.
8763
960.
8737
340.
8751
460.
7120
200.
7120
200.
7120
20-7
.25
RB
MT
AR
BM
TB
654
Oth
erY
ES
34.7
437
.51
35.5
40.
8457
120.
8490
140.
8462
280.
6431
500.
6431
500.
6431
50—
–R
BM
TB
Onl
ine
A(2
015)
652
Oth
erY
ES
55.0
556
.84
55.4
60.
9091
520.
9093
850.
9088
380.
8004
600.
8004
600.
8004
60+3
8.75
Onl
ine
A(2
015)
Onl
ine
A(2
016)
1039
Oth
erY
ES
54.7
856
.68
55.1
40.
9073
200.
9076
520.
9067
430.
7987
500.
7987
500.
7987
50+8
.00
Onl
ine
A(2
016)
Onl
ine
B(2
015)
651
Oth
erY
ES
36.4
138
.72
37.0
10.
8517
450.
8522
630.
8519
450.
7287
500.
7287
500.
7287
50—
–O
nlin
eB
(201
5)E
HR
110
05SM
TY
ES
71.5
172
.32
71.7
70.
9446
510.
9435
140.
9446
060.
8663
700.
8663
700.
8663
70-3
.00
Com
bina
tion
ofw
ord-
base
dPB
-SM
Tan
dch
arac
ter-
base
dPB
SMT
with
DL
=0.
Pare
nthe
ses
surr
ound
-in
gnu
mbe
rin
Kor
ean
sent
ence
sar
ede
lete
d.E
HR
210
06SM
TY
ES
62.3
364
.17
62.7
50.
9270
650.
9272
150.
9270
170.
8180
300.
8180
300.
8180
30+2
1.75
Com
bina
tion
ofw
ord-
base
dPB
-SM
Tan
dch
arac
ter-
base
dPB
SMT
with
DL
=0.
Pare
nthe
ses
inK
orea
nsi
dean
dno
tin
Japa
nese
side
are
adde
dto
Japa
nese
for
trai
ning
and
dev
sets
.JA
PIO
112
06SM
TY
ES
68.6
269
.49
68.9
00.
9384
740.
9370
660.
9382
300.
8581
900.
8581
900.
8581
90-9
.00
Phra
se-b
ased
SMT
+JA
PIO
corp
us+
rule
-bas
edpo
sted
itor
JAPI
O2
1209
SMT
YE
S70
.32
71.0
770
.52
0.94
2137
0.94
0544
0.94
1746
0.86
3660
0.86
3660
0.86
3660
+17.
50Ph
rase
-bas
edSM
T+
JPC
/JA
PIO
corp
ora
+ru
le-b
ased
post
edito
r
Tabl
e30
:JPC
-KJ
subm
issi
ons
40
SYST
EM
IDID
ME
TH
OD
OT
HE
RR
ESO
UR
CE
SB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
SMT
Phra
se97
1SM
TN
O24
.57
0.77
9545
0.57
8310
0.00
Phra
se-b
ased
SMT
SMT
Hie
ro98
1SM
TN
O23
.62
0.77
6309
0.57
5450
-8.2
5H
iera
rchi
calP
hras
e-ba
sed
SMT
SMT
S2T
982
SMT
NO
22.9
00.
7804
360.
5772
10-3
.25
Stri
ng-t
o-Tr
eeSM
TO
nlin
eA
1033
Oth
erY
ES
28.1
10.
7978
520.
6072
90+4
9.25
Onl
ine
AO
nlin
eB
1052
Oth
erY
ES
19.6
90.
7706
900.
5789
20+3
4.50
Onl
ine
BSe
nse
111
71SM
TN
O25
.62
0.78
2761
0.56
4500
-5.0
0B
asel
ine-
C50
-PB
MT
Sens
e2
1173
SMT
NO
25.9
70.
7877
680.
5707
10-8
.25
Clu
ster
cat-
PBM
T
Tabl
e31
:BPP
T-IE
subm
issi
ons
SYST
EM
IDID
ME
TH
OD
OT
HE
RR
ESO
UR
CE
SB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
SMT
Phra
se97
2SM
TN
O23
.95
0.80
8362
0.55
9800
0.00
Phra
se-b
ased
SMT
SMT
Hie
ro98
3SM
TN
O22
.64
0.79
6701
0.56
8660
-17.
00H
iera
rchi
calP
hras
e-ba
sed
SMT
SMT
T2S
984
SMT
NO
23.6
50.
7923
460.
5725
20-7
.75
Tree
-to-
Stri
ngSM
TO
nlin
eA
1034
Oth
erY
ES
24.2
00.
8195
040.
5547
20+3
5.75
Onl
ine
AO
nlin
eB
1050
Oth
erY
ES
18.0
90.
7894
990.
5144
30+1
0.50
Onl
ine
BSe
nse
111
70SM
TN
O25
.16
0.80
7097
0.56
8780
+1.2
5B
asel
ine-
C50
-PB
MT
Sens
e2
1174
SMT
NO
25.3
10.
8084
840.
5718
90-2
.75
Clu
ster
cat-
C50
-PB
MT
ITT
B-E
N-I
D1
1239
SMT
NO
22.3
50.
8089
430.
5559
70-9
.25
BL
NL
M
Tabl
e32
:BPP
T-E
Isub
mis
sion
s
41
SYST
EM
IDID
ME
TH
OD
OT
HE
RR
ESO
UR
CE
SB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
Onl
ine
A10
31O
ther
YE
S21
.37
0.71
4537
0.62
1100
+44.
75O
nlin
eA
(201
6)O
nlin
eB
1048
Oth
erY
ES
15.5
80.
6832
140.
5905
20+1
4.00
Onl
ine
B(2
016)
SMT
Phra
se10
54SM
TN
O10
.32
0.63
8090
0.57
4850
0.00
Phra
se-b
ased
SMT
Tabl
e33
:IIT
B-H
Esu
bmis
sion
s
SYST
EM
IDID
ME
TH
OD
OT
HE
RR
ESO
UR
CE
SB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
SMT
Phra
se12
52SM
TN
O10
.790
000
0.65
1166
0.66
0860
—–
Phra
se-b
ased
SMT
Onl
ine
A10
32O
ther
YE
S18
.720
000
0.71
6788
0.67
0660
+57.
25O
nlin
eA
(201
6)O
nlin
eB
1047
Oth
erY
ES
16.9
7000
00.
6912
980.
6684
50+4
2.50
Onl
ine
B(2
016)
EH
R1
1166
SMT
NO
11.7
5000
00.
6718
660.
6507
500.
00PB
SMT
with
preo
rder
ing
(DL
=6)
IIT
P-M
T1
1185
SMT
YE
S13
.710
000
0.68
8913
0.65
7330
+4.7
5II
TP-
MT
Syst
em1
Tabl
e34
:IIT
B-E
Hsu
bmis
sion
s
42
SYST
EM
IDID
ME
TH
OD
OT
HE
RB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
RE
SOU
RC
ES
jum
anky
tea
mec
abju
man
kyte
am
ecab
jum
anky
tea
mec
abSM
TPh
rase
1251
SMT
NO
2.05
4.17
2.42
0.44
0122
0.49
6402
0.46
1763
0.36
0910
0.36
0910
0.36
0910
—–
Phra
se-b
ased
SMT
Onl
ine
A10
64O
ther
YE
S6.
6010
.42
7.47
0.56
5109
0.59
7863
0.57
6725
0.49
5270
0.49
5270
0.49
5270
+39.
75O
nlin
eA
(201
6)O
nlin
eB
1065
Oth
erY
ES
5.70
8.91
6.38
0.56
0486
0.58
9558
0.57
1670
0.47
1450
0.47
1450
0.47
1450
+17.
75O
nlin
eB
(201
6)E
HR
111
67SM
TY
ES
7.81
10.1
28.
110.
5792
850.
6170
980.
5887
230.
4681
400.
4681
400.
4681
40+1
3.75
PBSM
Tw
ithph
rase
tabl
epi
votin
gan
dpi
votl
angu
age
(en)
reor
deri
ng.
Use
rdic
tiona
ryan
dT
ED
base
dL
Mar
eus
ed.
EH
R2
1179
SMT
YE
S7.
669.
807.
950.
5859
530.
6181
060.
5974
900.
4731
200.
4731
200.
4731
20+1
0.00
PBSM
Tw
ithse
nten
cele
vel
pivo
t-in
gan
dpi
votl
angu
age
(en)
reor
der-
ing.
Use
rdic
tiona
ryan
dT
ED
base
dL
Mar
eus
ed.
Tabl
e35
:IIT
B-H
Jsu
bmis
sion
s
SYST
EM
IDID
ME
TH
OD
OT
HE
RR
ESO
UR
CE
SB
LE
UR
IBE
SA
MFM
Pair
SYST
EM
DE
SCR
IPT
ION
SMT
Phra
se12
53SM
TN
O1.
5900
000.
3994
480.
4679
80—
–Ph
rase
-bas
edSM
TO
nlin
eB
1066
Oth
erY
ES
4.21
0000
0.48
8631
0.52
8220
+51.
75O
nlin
eB
(201
6)O
nlin
eA
1067
Oth
erY
ES
4.43
0000
0.49
5349
0.52
5690
+54.
50O
nlin
eA
(201
6)
Tabl
e36
:IIT
B-J
Hsu
bmis
sion
s
43
ReferencesRafael E. Banchs, Luis F. D’Haro, and Haizhou Li. 2015. Adequacy-fluency metrics: Evaluating mt in the
continuous space model framework. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 23(3):472–482, March.
Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partialcredit. Psychological Bulletin, 70(4):213 – 220.
Fabien Cromieres, Chenhui Chu, Toshiaki Nakazawa, and Sadao Kurohashi. 2016. Kyoto university participationto wat 2016. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 166–174, Osaka,Japan, December. The COLING 2016 Organizing Committee.
Terumasa Ehara. 2016. Translation systems and experimental results of the ehr group for wat2016 tasks. InProceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 111–118, Osaka, Japan, December.The COLING 2016 Organizing Committee.
Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Character-based decoding in tree-to-sequence attention-based neural machine translation. In Proceedings of the 3rd Workshop on Asian Translation(WAT2016), pages 175–183, Osaka, Japan, December. The COLING 2016 Organizing Committee.
J.L. Fleiss et al. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin,76(5):378–382.
Kazuma Hashimoto, Akiko Eriguchi, and Yoshimasa Tsuruoka. 2016. Domain adaptation and attention-basedunknown word replacement in chinese-to-japanese neural machine translation. In Proceedings of the 3rd Work-shop on Asian Translation (WAT2016), pages 75–83, Osaka, Japan, December. The COLING 2016 OrganizingCommittee.
Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified kneser-neylanguage model estimation. In Proceedings of the 51st Annual Meeting of the Association for ComputationalLinguistics (Volume 2: Short Papers), pages 690–696, Sofia, Bulgaria, August. Association for ComputationalLinguistics.
Hieu Hoang, Philipp Koehn, and Adam Lopez. 2009. A unified framework for phrase-based, hierarchical, andsyntax-based statistical machine translation. In Proceedings of the International Workshop on Spoken LanguageTranslation, pages 152–159.
Kenji Imamura and Eiichiro Sumita. 2016. Nict-2 translation system for wat2016: Applying domain adapta-tion to phrase-based statistical machine translation. In Proceedings of the 3rd Workshop on Asian Translation(WAT2016), pages 126–132, Osaka, Japan, December. The COLING 2016 Organizing Committee.
Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Automatic evaluationof translation quality for distant language pairs. In Proceedings of the 2010 Conference on Empirical Meth-ods in Natural Language Processing, EMNLP ’10, pages 944–952, Stroudsburg, PA, USA. Association forComputational Linguistics.
Marcin Junczys-Dowmunt, Tomasz Dwojak, and Hieu Hoang. 2016. Is neural machine translation ready fordeployment? a case study on 30 translation directions.
Satoshi Kinoshita, Tadaaki Oshio, Tomoharu Mitsuhashi, and Terumasa Ehara. 2016. Translation using japiopatent corpora: Japio at wat2016. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages133–138, Osaka, Japan, December. The COLING 2016 Organizing Committee.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, BrookeCowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, andEvan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Annual Meeting of theAssociation for Computational Linguistics (ACL), demonstration session.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Dekang Lin and DekaiWu, editors, Proceedings of EMNLP 2004, pages 388–395, Barcelona, Spain, July. Association for Computa-tional Linguistics.
T. Kudo. 2005. Mecab : Yet another part-of-speech and morphological analyzer. http://mecab.sourceforge.net/.
Sadao Kurohashi, Toshihisa Nakamura, Yuji Matsumoto, and Makoto Nagao. 1994. Improvements of Japanesemorphological analyzer JUMAN. In Proceedings of The International Workshop on Sharable Natural Lan-guage, pages 22–28.
44
Hyoung-Gyu Lee, JaeSong Lee, Jun-Seok Kim, and Chang-Ki Lee. 2015. NAVER Machine Translation Systemfor WAT 2015. In Proceedings of the 2nd Workshop on Asian Translation (WAT2015), pages 69–73, Kyoto,Japan, October.
Shaotong Li, JinAn Xu, Yufeng Chen, and Yujie Zhang. 2016. System description of bjtu nlp neural machinetranslation system. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 104–110,Osaka, Japan, December. The COLING 2016 Organizing Committee.
Toshiaki Nakazawa, Hideya Mino, Isao Goto, Sadao Kurohashi, and Eiichiro Sumita. 2014. Overview of the 1stWorkshop on Asian Translation. In Proceedings of the 1st Workshop on Asian Translation (WAT2014), pages1–19, Tokyo, Japan, October.
Toshiaki Nakazawa, Hideya Mino, Isao Goto, Graham Neubig, Sadao Kurohashi, and Eiichiro Sumita. 2015.Overview of the 2nd Workshop on Asian Translation. In Proceedings of the 2nd Workshop on Asian Translation(WAT2015), pages 1–28, Kyoto, Japan, October.
Graham Neubig, Yosuke Nakata, and Shinsuke Mori. 2011. Pointwise prediction for robust, adaptable japanesemorphological analysis. In Proceedings of the 49th Annual Meeting of the Association for ComputationalLinguistics: Human Language Technologies: Short Papers - Volume 2, HLT ’11, pages 529–533, Stroudsburg,PA, USA. Association for Computational Linguistics.
Graham Neubig, Makoto Morishita, and Satoshi Nakamura, 2015. Proceedings of the 2nd Workshop on AsianTranslation (WAT2015), chapter Neural Reranking Improves Subjective Quality of Machine Translation: NAISTat WAT2015, pages 35–41. Workshop on Asian Translation.
Graham Neubig. 2016. Lexicons and minimum risk training for neural machine translation: Naist-cmu at wat2016.In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 119–125, Osaka, Japan, December.The COLING 2016 Organizing Committee.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluationof machine translation. In ACL, pages 311–318.
Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretabletree annotation. In Proceedings of the 21st International Conference on Computational Linguistics and 44thAnnual Meeting of the Association for Computational Linguistics, pages 433–440, Sydney, Australia, July.Association for Computational Linguistics.
John Richardson, Raj Dabre, Fabien Cromieres, Toshiaki Nakazawa, and Sadao Kurohashi. 2015. KyotoEBMTSystem Description for the 2nd Workshop on Asian Translation. In Proceedings of the 2nd Workshop on AsianTranslation (WAT2015), pages 54–60, Kyoto, Japan, October.
Sukanta Sen, Debajyoty Banik, Asif Ekbal, and Pushpak Bhattacharyya. 2016. Iitp english-hindi machine transla-tion system at wat 2016. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 216–222,Osaka, Japan, December. The COLING 2016 Organizing Committee.
Raphael Shu and Akiva Miura. 2016. Residual stacking of rnns for neural machine translation. In Proceedingsof the 3rd Workshop on Asian Translation (WAT2016), pages 223–229, Osaka, Japan, December. The COLING2016 Organizing Committee.
Sandhya Singh, Anoop Kunchukuttan, and Pushpak Bhattacharyya. 2016. Iit bombay ’s english-indonesiansubmission at wat: Integrating neural language models with smt. In Proceedings of the 3rd Workshop on AsianTranslation (WAT2016), pages 68–74, Osaka, Japan, December. The COLING 2016 Organizing Committee.
Katsuhito Sudoh and Masaaki Nagata. 2016. Chinese-to-japanese patent machine translation based on syntacticpre-ordering for wat 2016. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 211–215, Osaka, Japan, December. The COLING 2016 Organizing Committee.
Liling Tan. 2016. Faster and lighter phrase-based machine translation baseline. In Proceedings of the 3rd Work-shop on Asian Translation (WAT2016), pages 184–193, Osaka, Japan, December. The COLING 2016 Organiz-ing Committee.
Huihsin Tseng. 2005. A conditional random field word segmenter. In In Fourth SIGHAN Workshop on ChineseLanguage Processing.
Masao Utiyama and Hitoshi Isahara. 2007. A japanese-english patent parallel corpus. In MT summit XI, pages475–482.
45
Hayahide Yamagishi, Shin Kanouchi, Takayuki Sato, and Mamoru Komachi. 2016. Controlling the voice ofa sentence in japanese-to-english neural machine translation. In Proceedings of the 3rd Workshop on AsianTranslation (WAT2016), pages 203–210, Osaka, Japan, December. The COLING 2016 Organizing Committee.
Wei Yang and Yves Lepage. 2016. Improving patent translation using bilingual term extraction and re-tokenizationfor chinese–japanese. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 194–202,Osaka, Japan, December. The COLING 2016 Organizing Committee.
46
