Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1213–1223Florence, Italy, July 28 - August 2, 2019. c©2019 Association for Computational Linguistics

1213

A Compact and Language-Sensitive Multilingual Translation Method

Yining Wang1,2, Long Zhou1,2, Jiajun Zhang1,2∗, Feifei Zhai4,Jingfang Xu4 and Chengqing Zong1,2,3

1National Laboratory of Pattern Recognition, CASIA, Beijing, China2University of Chinese Academy of Sciences, Beijing, China

3CAS Center for Excellence in Brain Science and Intelligence Technology, Beijing, China4Sogou Inc., Beijing, China

{yining.wang, long.zhou, jjzhang, cqzong}@nlpr.ia.ac.cn{zhaifeifei,xujingfang}@sogou-inc.com

Abstract

Multilingual neural machine translation(Multi-NMT) with one encoder-decodermodel has made remarkable progress due toits simple deployment. However, this multi-lingual translation paradigm does not makefull use of language commonality and param-eter sharing between encoder and decoder.Furthermore, this kind of paradigm cannotoutperform the individual models trained onbilingual corpus in most cases. In this paper,we propose a compact and language-sensitivemethod for multilingual translation. Tomaximize parameter sharing, we first presenta universal representor to replace both encoderand decoder models. To make the representorsensitive for specific languages, we furtherintroduce language-sensitive embedding,attention, and discriminator with the abilityto enhance model performance. We verifyour methods on various translation scenarios,including one-to-many, many-to-many andzero-shot. Extensive experiments demon-strate that our proposed methods remarkablyoutperform strong standard multilingualtranslation systems on WMT and IWSLTdatasets. Moreover, we find that our modelis especially helpful in low-resource andzero-shot translation scenarios.

1 Introduction

Encoder-decoder based sequence-to-sequence ar-chitecture (Kalchbrenner and Blunsom, 2013;Sutskever et al., 2014; Zhang and Zong, 2015;Vaswani et al., 2017; Gehring et al., 2017) fa-cilitates the development of multilingual neuralmachine translation (Multi-NMT) (Dong et al.,2015; Luong et al., 2016; Firat et al., 2016; John-son et al., 2017; Gu et al., 2018). The domi-

∗ Jiajun Zhang is the corresponding author and the workis done while Yining Wang is doing research intern at SogouInc.

N

Lang-Sensi Embedding

Lang-Sensi attention

Discrimnator

M Src N Tgt

EncoderDecoderSharing

Representor

N

Lang-Sensi Embedding

Lang-Sensi Attention

Lang-Sensi Discrimnator

M Src N Tgt

Encoder DecoderSharing

Representor

Figure 1: Our proposed compact representor, replacingencoder and decoder, can perform multilingual trans-lation from M source languages to N target languages.We also introduce three specific modules consisting oflanguage-sensitive embedding, language-sensitive at-tention, and language-sensitive discriminator.

nant paradigm of Multi-NMT contains one en-coder to represent multiple languages and one de-coder to generate output tokens of separate lan-guages (Johnson et al., 2017; Ha et al., 2016). Thisparadigm is widely used in Multi-NMT systemsdue to simple implementation and convenient de-ployment.

However, this paradigm has two drawbacks. Forone hand, using single encoder-decoder frame-work for all language pairs usually yields infe-rior performance compared to individually trainedsingle-pair models in most cases (Lu et al., 2018;Platanios et al., 2018; Wang et al., 2018). Forthe other hand, although this paradigm saves lotsof parameters compared to another Multi-NMTframework which employs separate encoders anddecoders to handle different languages (Donget al., 2015; Zoph and Knight, 2016; Luong et al.,2016; Firat et al., 2016), parameter sharing be-tween encoder and decoder are not fully explored.Since both encoder and decoder have similar

1214

structures but use different parameters, the com-monality of languages cannot be fully exploited inthis paradigm. A natural question arises that whynot share the parameters between encoder and de-coder on multilingual translation scenario?

To address these issues, we present a compactand language-sensitive method in this work, asshown in Figure 1. We first propose a unifiedrepresentor by tying encoder and decoder weightsin Multi-NMT model, which can not only re-duce parameters but also make full use of lan-guage commonality and universal representation.To enhance the model ability to distinguish dif-ferent languages, we further introduce language-sensitive embedding, attention, and discriminator.

We conduct extensive experiments to verifythe effectiveness of our proposed model on vari-ous Multi-NMT tasks including one-to-many andmany-to-many which is further divided into bal-anced, unbalanced and zero-shot. Experimen-tal results demonstrate that our model can sig-nificantly outperform the strong standard base-line multilingual systems and achieve even betterperformance than individually trained models onmost of the language pairs.

Specifically, our contributions are three-fold inthis work:

(1) We present a universal representor to replaceencoder and decoder, leading to a compact transla-tion model, which fully explores the commonalitybetween languages.

(2) We introduce language-sensitive embed-ding, attention, and discriminator which augmentthe ability of Multi-NMT model in distinguishingdifferent languages.

(3) Extensive experiments demonstrate the su-periority of our proposed method on various trans-lation tasks including one-to-many, many-to-manyand zero-shot scenarios. Moreover, for many-to-many using unbalance translation pairs, we canachieve the new state-of-the-art results on IWSLT-15 English-Vietnamese. For zero-shot translation,our methods can achieve even better results thanindividually trained models with the parallel cor-pus.

2 Background

In this section, we will introduce the backgroundof the encoder-decoder (Sutskever et al., 2014;Cho et al., 2014) framework and self-attention-based Transformer (Vaswani et al., 2017).

2.1 Encoder-Decoder Framework

Given a set of sentence pairs D = {(x, y)},the encoder fenc with parameters θenc maps aninput sequence x = (x1, x2, · · · , xn) to a se-quence of continuous representations henc =

(henc1 , henc

2 , · · · , hencn ) whose size varies concern-

ing the source sentence length. The decoderfdec with θdec generates an output sequence y =(y1, y2, · · · , ym) by computing P(yt |y<t ) as fol-lows:

P(yt |y<t ) = softmax( f (hdec, ct )) (1)

where hdec is a sequence of continuous represen-tations for the decoder and ct is the context vectorwhich can be calculated as follows:

ct =n∑i=1

ai,t henci (2)

where ai,t is attention weight:

ai,t = softmax(ei,t ) =exp ei,t∑nj=1 exp ej,t

(3)

where ei,t is a similarity score between the sourceand target representations. The parameters of cal-culating cross-attention weight ai,t are denoted asθattn.

The encoder and decoder are trained to maxi-mize the conditional probability of target sequencegiven a source sequence:

Lt (D;θ) =|D |∑d=1

M∑t=1

log P(yt |y<t, x; θenc, θdec, θattn) (4)

where M is target sentence length. For simplicity,we do not specify d in this formula.

Both the encoder and decoder can be imple-mented by the different basic neural models struc-tures, such as RNN (LSTM/GRU) (Sutskeveret al., 2014; Cho et al., 2014), CNN (Gehringet al., 2017), and self-attention (Vaswani et al.,2017). Our proposed method can be applied to anyencoder-decoder architecture. Considering the ex-cellent translation performance of self-attentionbased Transformer (Vaswani et al., 2017), we im-plement our method based on this architecture.

2.2 Transformer Network

Transformer is a stacked network with several lay-ers containing two or three basic blocks in eachlayer. For a single layer in the encoder, it consistsof a multi-head self-attention and a position-wise

1215

feed-forward network. For the decoder model, be-sides the above two basic blocks, a multi-headcross-attention follows multi-head self-attention.In this block, the calculation method of similar-ity score et in Equation 3 is a little different fromLuong et al. (2015) and Bahdanau et al. (2015):

ei,t =1√

dmWkhenc

i ∗Wqhdect (5)

where dm is the dimension of hidden units, Wk andWq are parameters of this cross-attention block,which are denoted as θattn in Equation 4.

All the basic blocks are associated with resid-ual connections, followed by layer normalization(Ba et al., 2016). Since the Transformer networkcontains no recurrence, positional embeddings areused in the model to make use of sequence or-der. More details regarding the architecture canbe found in Vaswani et al. (2017).

2.3 Multilingual TranslationIn contrast to NMT models, multilingual modelsperform the multi-task paradigm with some de-gree of parameter sharing, in which models arejointly trained on multiple language pairs. Wemainly focus on mainstream multilingual transla-tion method proposed by Johnson et al. (2017),which has a unified encoder-decoder frameworkwith a shared attention module for multiple lan-guage pairs. They decompose the probability ofthe target sequences into the products of per tokenprobabilities in all translation forms:

Lm−t (D;θ) =L∑l=1

|Dl |∑d=1

M∑t=1

log P(ylt |xl, yl<t ; θenc, θdec, θattn)

(6)

where L is the number of translation pairs andP(ylt |x

l, yl<t ;θ) denotes the translation probabilityof t-th word of the d-th sentence in l-th translationpair. Note that the translation process for all targetlanguages uses the same parameter set θ.

3 Our Method

In this section, we introduce our compact andlanguage-sensitive method for multilingual trans-lation, which can compress the model by a repre-sentor and improve model ability with language-sensitive modules.

3.1 A Compact RepresentorIn Multi-NMT model, the encoder and decoderare two key components, which play analogous

Embeddings

Multi-head Self-Attention

Residual&Norm

N

Softmax

Linear

Output Probabilities

Language-Sensitive Discriminator

+Position Embdding

lang-1 lang-2 ... lang-n lang-1 lang-2 ... lang-n

lang-1 lang-2 ... lang-n

src tgt

Language-Sensitive Embedding

+ + + + + +Representor

Residual&Norm Feed

Forward

Residual&NormMulti-head Language-Sensitive

Cross-Attention

Figure 2: The framework of Multi-NMT using ourcompact and language-sensitive method.

roles and have a similar structure in each layer.We argue that encoder and decoder can share thesame parameters if necessary. Thus, we introducea representor to replace both encoder and decoderby sharing weight parameters of the self-attentionblock, feed-forward block and the normalizationblock, as shown in Figure 2. The representor pa-rameters are denoted θrep. Therefore, the objectivefunction (Equation 6) becomes:

Lm−t (D;θ) =L∑l=1

|Dl |∑d=1

M∑t=1

log P(ylt |xl, yl<t ; θrep, θattn) (7)

This representor (θrep) coordinates the semanticpresentation of multiple languages in a closely re-lated universal level, which also increases the uti-lization of commonality for different languages.

3.2 Language-Sensitive Modules

The compact representor maximizes the sharing ofparameters and makes full use of language com-monality. However, it lacks the ability to discrimi-nate different languages. In our method, we intro-duce three language-sensitive modules to enhanceour model as follows:

1) Language-Sensitive Embedding: Previ-ously, Press and Wolf (2017) conduct the weighttying of input and output embedding in NMTmodel. Generally, a shared vocabulary is built

1216

upon subword units like BPE (Sennrich et al.,2016b) and wordpiece (Wu et al., 2016; Schus-ter and Nakajima, 2012). However, it remainsunder-exploited which kind of embedding shar-ing is best for Multi-NMT. We divide the sharingmanners into four categories including language-based manner (LB, different languages have sep-arate input embeddings), direction-based manner(DB, languages in source side and target sidehave different input embeddings), representor-based manner (RB, shared input embeddings forall languages) and three-way weight tying man-ner (TWWT) proposed in Press and Wolf (2017),in which the output embedding of the target sideis also shared besides representor-based sharing.We compare these four sharing manners for Multi-NMT in our experiments, and we will discuss theresults in Section 5.

Considering the last three sharing manners can-not model a sense of which language a token be-longs to, we propose a new language-sensitive em-bedding in our method to specify different lan-guages explicitly. Similar to the position embed-dings described in Section 2, this kind of embed-ding is added to the embedding of each tokenfor corresponding language, which can indicatethe translation direction on the source side andguide the generation process for target languages.This embedding is denoted as Elang ∈ R

|K |∗dmodel ,where |K | is the number of languages involved,and dmodel is the dimension of hidden states in ourmodel. Note that this embedding can be learnedduring training.

2) Language-Sensitive Attention: In NMT ar-chitecture, cross-attention only appearing in thedecoder network locates the most-relevant sourcepart when generating each token in target lan-guage. For Multi-NMT, we introduce three dif-ferent ways to design the cross-attention mecha-nism, consisting of i) shared-attention, ii) hybrid-attention, and iii)) language-sensitive attentionutilized in our method.

i): In our proposed compact representor, weshare self-attention block between encoder and de-coder. For the shared-attention, we make a fur-ther step to share parameters of cross-attention andself-attention, which can be regarded as coordina-tion of information from both the source side andtarget side.

ii): Different from the above attention mech-anism, the hybrid-attention utilizes independent

cross-attention modules but it is shared for alltranslation tasks.

iii): In the language-sensitive attention, it al-lows the model to select the cross-attention param-eters associated with specific translation tasks dy-namically.

In our paper, we investigate these three atten-tion mechanisms. We argue that both the sharedand hybrid mechanisms tend to be confused to ex-tract information from different source languageswhen decoding multiple source languages withdifferent word orders. Thus, we mainly focuson languages-sensitive attention in our method.To this end, we use multiple sets of parametersθattn to represent cross-attention modules of differ-ent translation tasks. However, language-sensitiveattention does not support zero-shot translationbecause there is no explicit training set for thisspecific translation task. Therefore, we employhybrid-attention mechanism in our zero-shot ex-periments.

3) Language-Sensitive Discriminator: In ourmethod, the representor which shares encoder anddecoder makes full use of language common-ality, but it weakens the model ability to dis-tinguish different languages. Hence we intro-duce a new language-sensitive discriminator tostrengthen model representation.

In NMT framework, the hidden states on thetop layer can be viewed as a fine-grained abstrac-tion (Anastasopoulos and Chiang, 2018). For thislanguage-sensitive module, we first employ a neu-ral model fdis on the top layer of reprensentor hrep

top,and the output of this model is a language judg-ment score Plang.

hdis = fdis(hreptop)

Plang(d) = softmax(Wdis ∗ hdisd + bdis)

(8)

where Plang(d) is language judgment score for sen-tence pair d, Wdis, bdis are parameters, which aredenoted as θdis. We test two different types of neu-ral models for fdis, including convolutional net-work with max pooling layer and two-layer feed-forward network.

And then, we obtain an discriminant objectivefunction as follows:

Ldis (θdis) =∑k∈K

|D |∑d=1I {gd = k} ∗ logPlang (d) (9)

where I {·} is indicator function, and gd belongs tolanguage k.

1217

Finally, we incorporate the language-sensitivediscriminator into our Multi-NMT model, and itcan be optimized through an end-to-end mannerfor all translation language pairs D with the fol-lowing objective function.

L(D;θ) =L(D; θrep, θattn, θdis)

=(1 − λ)Lm−t (θrep, θattn) + λLdis(θdis)(10)

where λ is learned or pre-defined weight to bal-ance the translation task and language judgmenttask.

4 Experimental Settings

4.1 Data

In this section, we describe the datasets usingin our experiments on one-to-many and many-to-many multilingual translation scenarios.

One-to-Many: For this translation scenario,we perform one-to-two, one-to-three, and one-to-four multilingual translation on the combi-nation of WMT-141 (English-to-German, brieflyEn→De), WMT-172 datasets (English-to-Latvian,briefly En→Lv) and WMT-183 (English-to-Finnish, English-to-Chinese without UN part4,briefly En→Fi and En→Zh) datasets.

Many-to-Many: For many-to-many transla-tion, we test our methods on IWSLT-175 trans-lation datasets, including English, Italian, Roma-nian, Dutch (briefly, En, It, Ro, Nl). In orderto perform zero-shot translation, we discard someparticular language pairs. We also evaluate ourmethod on the unbalanced training corpus. Tothis end, we construct the training corpus usingresource-rich En-De, En-Fi in WMT datasets andlow-resource English-Vietnamese (briefly, En-Vi)in IWSLT-156.

The statistical information of all the datasets isdetailed in Table 1.

4.2 Training Details

We implement our compact and language-sensitive method for Multi-NMT based on the ten-sor2tensor7 library. We use wordpiece method(Wu et al., 2016; Schuster and Nakajima, 2012) to

1http://www.statmt.org/wmt14/translation-task.html2http://www.statmt.org/wmt17/translation-task.html3http://www.statmt.org/wmt18/translation-task.html4https://cms.unov.org/UNCorpus/5https://sites.google.com/site/iwsltevaluation20176https://sites.google.com/site/iwsltevaluation20157https://github.com/tensorflow/tensor2tensor

Datasets Language pair Train Dev Test

WMT

En-De 4.50M 6003 3003En-Lv 4.50M 2003 2001En-Fi 3.25M 3000 3000En-Zh 9.02M 2002 2001

IWSLT

En-It 231.6k 929 1566En-Ro 220.5k 914 1678En-Nl 237.2k 1003 1777Ro-It 217.5k 914 1643En-Vi 130.9k 768 1268

Table 1: The statistics of all the datasets includingWMT and IWSLT tasks.

encode the combination of both source side sen-tences and target side sentences. The vocabularysize is 37,000 for both sides. We train our mod-els using configuration transformer base adoptedby Vaswani et al. (2017), which contains a 6-layer encoder and a 6-layer decoder with 512-dimensional hidden representations. Each mini-batch contains roughly 3,072 source and 3,072 tar-get tokens, which belongs to one translation di-rection. We use Adam optimizer (Kingma andBa, 2014) with β1=0.9, β2=0.98, and ε=10−9. Forevaluation, we use beam search with a beam sizeof k = 4 and length penalty α = 0.6. All our meth-ods are trained and tested on a single Nvidia P40GPU.

5 Results and Analysis

In this section, we discuss the results of our exper-iments about our compact and language-sensitivemethod on Multi-NMT. The translation perfor-mance is evaluated by character-level BLEU5for En→Zh translation and case-sensitive BLEU4(Papineni et al., 2002) for other translation tasks.In our experiments, the models trained on individ-ual language pair are denoted by NMT Baselines,and the baseline Multi-NMT models are denotedby Multi-NMT Baselines.

5.1 One-to-Many Translation

5.1.1 Main Results

The main results on the one-to-many translationscenario, including one-to-two, one-to-three andone-to-four translation tasks are reported in Ta-ble 2. We present a typical Multi-NMT adopt-ing Johnson et al. (2017) method on Transformeras our Multi-NMT baselines model. Obviously,Multi-NMT Baselines cannot outperform NMTBaselines in all cases, among which four direc-tions are comparable and twelve are worse.

1218

Task Tgt NMTBaselines

Multi-NMT BaselinesJohnson et al. (2017)

Three-StgyWang et al. (2018) Rep+Emb Rep+Emb

+AttnRep+Emb+Attn+Dis

One-to-Two

De 27.50 27.26 27.35 26.60 26.96 27.74Lv 16.28 16.32 16.38 15.37 15.87 16.79De 27.50 27.88 27.89 26.96 27.32 27.96Fi 16.83 16.47 16.70 15.78 16.58 16.89De 27.50 26.80 26.99 26.08 26.68 27.45Zh 26.04 25.54 25.78 24.48 25.33 26.17

One-to-Three

De 27.50 25.44 25.55 24.82 25.45 26.06Zh 26.04 24.87 25.63 24.12 24.93 26.12Fi 16.83 16.86 16.97 16.06 16.78 17.12De 27.50 25.98 26.12 24.88 25.80 26.42Lv 16.28 14.88 15.44 14.51 15.58 16.31Fi 16.83 16.94 17.05 16.15 16.79 17.22

One-to-Four

De 27.50 23.59 22.88 22.88 23.58 24.08Lv 16.28 15.57 16.02 15.00 16.21 16.57Zh 26.04 25.24 25.83 24.15 25.27 26.29Fi 16.83 13.45 14.12 12.99 14.11 15.03

Table 2: Translation performance on one-to-two, one-to-three and one-to-four translation tasks. Rep denotes ourproposed representor. Emb, Attn, and Dis represent our proposed language-sensitive methods to address multilin-gual translation. Note that the source language of all our experiments is English.

100 100 100

4127.3

20.529.8

19.2 14.5

35.222.5 18.7

35.422.6 18.8

0

20

40

60

80

100

120

one-to-two one-to-three one-to-four

Indiv Baselines Rep+Emb Rep+Emb+Attn Rep+Emb+Attn+Dis

Number of Parameters per Language Pair (M)

Tasks

Figure 3: The comparison of model scale among indi-vidually trained system, baselines Multi-NMT systemand our methods. Y-axis represents the model parame-ters per language pair, which is calculated by averagingmodel parameters on all translation tasks involved.

With respect to our proposed method, it is clearthat our compact method consistently outperformsthe baseline systems. Compared with anotherstrong one-to-many translation model Three-Stgyproposed by Wang et al. (2018), our compactmethod can achieve better results as well. More-over, our method can perform even better than in-dividually trained systems in most cases (elevenout of sixteen cases). The results demonstrate theeffectiveness of our method.

5.1.2 Model Size

Besides improving the translation results, we alsocompress the model size by introducing the rep-resentor. We investigate the scale of parametersused on average in each translation direction. Wecompare three models, including NMT Baselinesmodel, Multi-NMT Baselines model, and our com-pact Multi-NMT model. As shown in Figure 3, all

Src→Tgt EmbManners Size Tgt-1 Tgt-2

En→De/Lv

LB 139M 26.58 15.76DB 100M 27.22 16.26RB 82M 27.26 16.32

TWWT 63M 26.82 16.02

En→De/Zh

LB 139M 27.34 25.61DB 100M 27.15 25.22RB 82M 27.22 25.38

TWWT 63M 26.91 24.99

Table 3: Size (number of parameters) and BLEU scoresof various embedding sharing manners. LB, DB, RB,TWWT denote language-based manner, direction-based manner, representor-based manner, and three-way weight tying manner separately, as mentioned inSection 3.2. Tgt-1 and Tgt-2 mean the results of thefirst (De) and the second (Lv/Zh) target language.

the multilingual translation models reduce the pa-rameters. Compared with Multi-NMT Baselines,we can observe that our method further reducesthe model size of Multi-NMT. Considering Ta-ble 2 and Figure 3 together, we note that eventhough our proposed method in one-to-four trans-lation task only uses 18.8% parameters of NMTBaselines, we can achieve better performance onEn→Zh and En→Lv.

5.1.3 Discussion of Language-SensitiveModules

Table 2 shows that our proposed language-sensitive modules are complementary with eachother. In this subsection, we will analyze eachmodule in detail.

Language-Sensitive Embedding: As men-tioned in section 3.2, embedding sharing man-

1219

0.01 0.03 0.05 0.07 0.0922.4

22.6

22.8

23.2

23.4

BLEU

Sco

re

22.57

22.76

23.19

23.07

22.6722.63

22.91

23.35

23.19

22.74

CNN on En-De FFN on En-De

23.0

0.01 0.03 0.05 0.07 0.09

22.2

22.4

22.6

22.8

23.0

BLEU

Sco

re

22.21

22.37

22.97

22.78

22.43

22.15

22.28

22.72

22.56

22.24

CNN on En-LvFFN on En-Lv

Figure 4: The comparison of two neural models with different hyper-parameter λ. CNN and FFN denote convolu-tion network and feed-forward network, respectively.

ners for Multi-NMT are divided into four cat-egories. We show the results of these sharingmanners in Table 3. To make a fair compari-son, we sample 4.5M sentence pairs from En-Zhdataset. As shown in this table, our representor-based sharing manner consistently outperformsboth the direction-based manner and three-wayweight tying manner. Furthermore, even thoughthe representor-based manner has about 40%fewer parameters than the language-based man-ner, it achieves comparable or even better per-formance. We find that language-based sharingmanner is unstable because it achieves the highestBLEU score on Multi-NMT of similar languages(En→De/Zh), but the worst quality on dissimilarlanguages (En→De/Lv). Taking into account oftranslation quality and stability, we choose to userepresentor-based sharing manner in our method.

As described in Section 3.2, our proposedlanguage-sensitive embedding is added to the in-put embedding of each token, which is unlikeconvention Multi-NMT method adding a specialtoken into source side sentences or vocabularies(Johnson et al., 2017; Ha et al., 2016). Thereexists a question, is this kind of embeddings es-sential in our representor? To make a verifica-tion, we do the ablation study without this module.We observe that Multi-NMT model does not con-verge during training, which demonstrates theselanguage-sensitive embeddings play a significantrole in our model.

Language-Sensitive Attention: We presentthree types of cross-attention mechanisms inSection 3.2. We adopt shared-attention andlanguage-sensitive attention for Rep+Emb and

Rep+Emb+Attn separately. Comparing these twomethods in Table 2, Rep+Emb+Attn method out-performs Rep+Emb method in all cases, whichdemonstrates the language-sensitive is useful formultiple language pairs with different word order.We also conduct the experiment of our representorwith the hybrid-attention mechanism. Since thismethod has similar performance with Rep+Embbut is larger in size, we ignore its results here.

Language-Sensitive Discriminator: In sec-tion 3.2, we employ two different types of theneural model as a language-sensitive discrimina-tor, and there is a hyper-parameter λ in Equa-tion 10. We present the effect of convolutionalnetwork and feed-forward network with differenthyper-parameters on development datasets in Fig-ure 4. Considering that distinguishing betweenlanguages is only an auxiliary task in Multi-NMT,we set the maximum of λ to be 0.1. As shown inFigure 4, when we adopt the convolution networkas our discriminator with λ = 0.05, our language-sensitive method performs best. We also conductthe experiments in which the hyper-parameter λis learnable. The experiment results are similar tothe best settings mentioned above both on En→De(23.35 vs. 23.19) and En→Lv (22.97 vs. 22.72).For simplicity, all our experiments listed in Table 2and 4 adopt convolution network as the language-sensitive discriminator with λ = 0.05.

5.2 Many-to-Many Translation

Table 4 reports the detailed results of differentmethods under the many-to-many translation sce-nario. We will analyze the performance below.

1220

Task Src→Tgt NMTBaselines

Multi-NMT BaselinesJohnson et al. (2017) Rep+Emb Rep+Emb

+AttnRep+Emb+Attn+Dis

Many-to-Many for Balanced Corpus

ISupervised

Four-to-Four

En→It 28.41 29.53 29.47 29.98 30.23It→En 30.66 31.70 31.76 32.23 32.75

En→Ro 21.41 22.23 22.16 22.87 23.53Ro→En 26.09 27.69 27.58 27.98 28.32En→Nl 25.88 27.88 26.96 27.32 27.96Nl→En 27.48 28.67 28.58 28.86 29.32It→Ro 12.77 13.86 13.89 14.35 14.89Ro→It 13.54 14.78 14.66 14.87 15.22

IIZero-Shot

Nl→Ro 14.15 13.70 13.98 15.12 15.54Ro→Nl 14.33 13.91 14.17 14.86 15.41It→Nl 18.24 17.97 18.02 18.98 19.74Nl→It 18.11 17.59 18.16 19.18 19.87

Many-to-Many for Unbalanced Corpus

IIISupervised

Three-to-Three

En→De 27.60 24.39 23.78 25.45 26.06De→En 32.23 28.85 28.14 28.98 30.37En→Fi 16.83 14.58 13.82 14.26 14.77Fi→En 22.37 19.60 19.15 19.96 21.03En→Vi 26.78 28.89 28.84 30.49 32.01Vi→En 25.72 27.19 27.27 29.14 31.71

Table 4: Translation performance under the many-to-many scenario, consisting of supervised four-to-four andzero-shot translation on the balanced corpus, and supervised three-to-three on the unbalanced corpus. Note thatwe do not use the Nl-Ro and It-Nl language pairs in our many-to-many translation task for the balanced corpus.

5.2.1 Results of Balanced Corpus

In part I of Table 4, our compact and language-sensitive method (Rep+Emb+Attn+Dis) performsconsistently better than corresponding Multi-NMTBaselines, and it can achieve the improvementsup to 1.30 BELU points (23.53 vs. 22.23 onEn→Ro). Although Rep+Emb method dramati-cally reduces the model parameters, it performs onpar with Multi-NMT Baselines. Compared withNMT Baselines model, our method also achievesbetter results, which is nearly 2 BLEU pointson average. Experimental results on our bal-anced corpus demonstrate that our method is ro-bust and valid under the many-to-many translationscenario.

5.2.2 Results of Unbalanced Corpus

For unbalanced corpus, our method can achievebetter results than Multi-NMT Baselines as well,as shown in part III of Table 4. Moreover, fromthe last two lines of this part, we can observethat compared with NMT Baselines, the transla-tion quality of En↔Vi can achieve the improve-ments up to 5.23/5.99 BLEU points (32.01/31.71vs. 26.78/25.72), both of which are new state-of-the-art on these translation tasks to the best of ourknowledge. The results show that our method ismore effective in low-resource language pairs, es-pecially for the unbalanced corpus.

5.2.3 Zero-Shot ResultsPart II in Table 4 shows the performance of zero-shot translation. Note that we conduct experi-ments of this translation scenario using hybrid-attention mechanism. Compared with Multi-NMTBaselines, our compact and language-sensitivemethod performs significantly better with the im-provement as large as 2.28 BLEU points onNl→It. Note that the training datasets do not con-tain parallel data for Nl-Ro and It-Nl.

It is interesting to figure out the translationperformance of Nl↔Ro and It↔Nl when bilin-gual training corpus is available. We conduct ex-periments of NMT Baselines on Nl-Ro and It-Nlwith all sentence pairs in IWSLT-17 (about 200k),which is similar to other training pairs in our bal-anced corpus. As shown in part II, Multi-NMTBaselines underperform the NMT Baselines on allcases. However, our method performs better thanNMT Baselines, and it achieves the improvementup to 1.76 BLEU points on Nl→It translation task.

6 Related Work

Our work is related to two lines of research, andwe describe each of them as follows:

Model Compactness and Multi-NMT: To re-duce the model size in NMT, weight pruning,knowledge distillation, quantization, and weightsharing (Kim and Rush, 2016; See et al., 2016;He et al., 2018; Zhou et al., 2018) have been ex-

1221

plored. Due to the benefit of compactness, multi-lingual translation has been extensively studied inDong et al. (2015), Luong et al. (2016) and John-son et al. (2017). Owing to excellent translationperformance and ease of use, many researchers(Blackwood et al., 2018; Lakew et al., 2018) haveconducted translation based on the framework ofJohnson et al. (2017) and Ha et al. (2016). Zhouet al. (2019) propose to perform decoding in twotranslation directions synchronously, which can beapplied on different target languages and is a newresearch area for Multi-NMT. In our method, wepresent a compact method for Multi-NMT, whichcan not only compress the model but also yield su-perior performance.

Low-Resource and Zero-Shot NMT: Manyresearchers have explored low-resource NMT us-ing transfer learning (Zoph et al., 2016; Neu-big and Hu, 2018) and data augmenting (Sen-nrich et al., 2016a; Zhang and Zong, 2016) ap-proaches. For zero-shot translation, Cheng et al.(2017) and Chen et al. (2017) utilize a pivot-basedmethod, which bridges the gap between source-to-pivot and pivot-to-target two steps. Multilin-gual translation is another direction to deal withboth low-resource and zero-shot translation. Guet al. (2018) enable sharing of lexical and sen-tence representation across multiple languages, es-pecially for extremely low-resource Multi-NMT.Firat et al. (2016), Lakew et al. (2017), and John-son et al. (2017) propose to make use of multi-linguality in Multi-NMT to address the zero-shotproblem. In this work, we propose a method forMulti-NMT to boost the accuracy of the multi-lingual translation, which better fits on both low-resource scenario and zero-shot scenario.

7 Conclusion

In this paper, we have proposed a compact andlanguage-sensitive method for multilingual trans-lation. We first introduce a representor for replac-ing both encoder and decoder so as to fully ex-plore the commonality among languages. Basedon the representor architecture, we then proposethree language-specific modules dealing with em-bedding, attention and language discrimination re-spectively, in order to enhance the multilanguagetranslation model with the ability of distinguish-ing among different languages. The empirical ex-periments demonstrate that our proposed methodscan outperform strong standard multilingual trans-

lation systems on one-to-many and many-to-manytranslation tasks. Moreover, our method is provedto be especially helpful in the low-resource andzero-shot translation scenarios.

Acknowledgments

The research work descried in this paper has beensupported by the National Key Research and De-velopment Program of China under Grant No.2016QY02D0303 and the Natural Science Foun-dation of China under Grant No. U1836221 and61673380. The research work in this paper hasalso been supported by Beijing Advanced Inno-vation Center for Language Resources and So-gou Inc. We would like to thank Yang Zhao andYuchen Liu for their invaluable discussions on thispaper.

ReferencesAntonios Anastasopoulos and David Chiang. 2018.

Tied multitask learning for neural speech translation.In Proceedings of NAACL 2018, pages 82–91.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin-ton. 2016. Layer normalization. arXiv preprintarXiv:1607.06450.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-gio. 2015. Neural machine translation by jointlylearning to align and translate. In Proceedings ofICLR 2015.

Graeme Blackwood, Miguel Ballesteros, and ToddWard. 2018. Multilingual neural machine transla-tion with task-specific attention. In Proceedings ofCOLING 2018, pages 3112–3122.

Yun Chen, Yong Cheng, Yang Liu, and Li Victor,O.K. 2017. A teacher-student framework for zero-resource neural machine translation. In Proceedingsof ACL 2017, pages 1925–1935.

Yong Cheng, Qian Yang, Yang Liu, Maosong Sun, andWei Xu. 2017. Joint training for pivot-based neu-ral machine translation. Proceedings of IJCAI 2017,pages 3974–3980.

Kyunghyun Cho, Bart van Merrienboer Caglar Gul-cehre, Dzmitry Bahdanau, Fethi Bougares HolgerSchwenk, and Yoshua Bengio. 2014. Learningphrase representations using rnn encoder–decoderfor statistical machine translation. In Proceedingsof EMNLP 2014, pages 1724–1734.

Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, andHaifeng Wang. 2015. Multi-task learning for mul-tiple language translation. In Proceedings of ACL2015, pages 1723–1732.

1222

Orhan Firat, Kyunghyun Cho, and Yoshua Bengio.2016. Multi-way, multilingual neural machinetranslation with a shared attention mechanism. InProceedings of NAACL 2016, pages 866–875.

Jonas Gehring, Michael Auli, David Grangier, De-nis Yarats, and Yann N. Dauphin. 2017. Convolu-tional sequence to sequence learning. arXiv preprintarXiv:1601.03317.

Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor OKLi. 2018. Universal neural machine translation forextremely low resource languages. In Proceedingsof NAACL 2018, pages 344–354.

Thanh-Le Ha, Jan Niehues, and Alexander Waibel.2016. Toward multilingual neural machine trans-lation with universal encoder and decoder. In Pro-ceedings of IWSLT 2016.

Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, ZhiboChen, and Tie-Yan Liu. 2018. Layer-wise coordi-nation between encoder and decoder for neural ma-chine translation. In Proceedings of NIPS 2018,pages 7944–7954.

Melvin Johnson, Mike Schuster, Quoc V. Le, MaximKrikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat,Fernanda Viegas, Martin Wattenberg, Greg Corrado,Macduff Hughes, and Jeffrey Dean. 2017. Google’smultilingual neural machine translation system: En-abling zero-shot translation. Transactions of the As-sociation for Computational Linguistics, 5:339–351.

Nal Kalchbrenner and Phil Blunsom. 2013. Recurrentcontinuous translation models. In Proceedings ofEMNLP 2013, pages 1700–1709.

Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings ofEMNLP 2016, pages 1317–1327.

Diederik P Kingma and Jimmy Ba. 2014. Adam: Amethod for stochastic optimization. arXiv preprintarXiv:1412.6980.

Surafel M Lakew, ADG Mattia, and F Marcello. 2017.Multilingual neural machine translation for low re-source languages. CLiC-it.

Surafel Melaku Lakew, Mauro Cettolo, and MarcelloFederico. 2018. A comparison of transformer andrecurrent neural networks on multilingual neuralmachine translation. In Proceedings of COLING2018, pages 641–652.

Yichao Lu, Phillip Keung, Faisal Ladhak, Vikas Bhard-waj, Shaonan Zhang, and Jason Sun. 2018. A neuralinterlingua for multilingual machine translation. InProceedings of WMT 2018, pages 84–92.

Minh-Thang Luong, Quoc V Le, Ilya Sutskever, OriolVinyals, and Lukasz Kaiser. 2016. Multi-task se-quence to sequence learning. In Proceedings ofICLR 2016.

Minh-Thang Luong, Hieu Pham, and Christopher DManning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings ofEMNLP, pages 1412–1421.

Graham Neubig and Junjie Hu. 2018. Rapid adaptationof neural machine translation to new languages. InProceedings of EMNLP 2018, pages 875–880.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic eval-uation of machine translation. In Proceedings ofACL, pages 311–318.

Emmanouil Antonios Platanios, Mrinmaya Sachan,Graham Neubig, and Tom Mitchell. 2018. Contex-tual parameter generation for universal neural ma-chine translation. In Proceedings of EMNLP 2018,pages 425–435.

Ofir Press and Lior Wolf. 2017. Using the output em-bedding to improve language models. In Proceed-ings of EACL 2017, pages 157–163.

Mike Schuster and Kaisuke Nakajima. 2012. Japaneseand korean voice search. In Proceedings of ICASSP2012.

Abigail See, Minh-Thang Luong, and Christopher D.Manning. 2016. Compression of neural machinetranslation models via pruning. In Proceedings ofSIGNLL 2016, pages 291–301.

Rico Sennrich, Barry Haddow, and Alexandra Birch.2016a. Improving neural machine translation mod-els with monolingual data. In Proceedings of ACL2016, pages 86–96.

Rico Sennrich, Barry Haddow, and Alexandra Birch.2016b. Neural machine translation of rare wordswith subword units. In Proceedings of ACL 2016,pages 1715–1725.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.Sequence to sequence learning with neural net-works. In Proceedings of NIPS, pages 3104–3112.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, andŁukasz Kaiser. 2017. Attention is all you need. InProceedings of NIPS, pages 30–34.

Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu,and Chengqing Zong. 2018. Three strategies to im-prove one-to-many multilingual translation. In Pro-ceedings of EMNLP 2018, pages 2955–2960.

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc VLe, Mohammad Norouzi, Wolfgang Macherey,Maxim Krikun, Yuan Cao, Qin Gao, KlausMacherey, et al. 2016. Google’s neural ma-chine translation system: Bridging the gap betweenhuman and machine translation. arXiv preprintarXiv:1609.08144.

1223

Jiajun Zhang and Chengqing Zong. 2015. Deep neu-ral networks in machine translation: An overview.IEEE Intelligent Systems, 30(5):16–25.

Jiajun Zhang and Chengqing Zong. 2016. Exploit-ing source-side monolingual data in neural machinetranslation. In Proceedings of EMNLP, pages 1535–1545.

Long Zhou, Yuchen Liu, Jiajun Zhang, ChengqingZong, and Guoping Huang. 2018. Language-independent representor for neural machine transla-tion. arXiv preprint arXiv:1811.00258.

Long Zhou, Jiajun Zhang, and Chengqing Zong. 2019.Synchronous bidirectional neural machine transla-tion. Transactions of the Association for Compu-tational Linguistics, 7:91–105.

Barret Zoph and Kevin Knight. 2016. Multi-sourceneural translation. In Proceedings of NAACL 2016,pages 30–34.

Barret Zoph, Deniz Yuret, Jonathan May, and KevinKnight. 2016. Transfer learning for low-resourceneural machine translation. In Proceedings ofEMNLP 2016, pages 1568–1575.