Conditional Generation and Snapshot Learning inNeural Dialogue Systems

Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina M. Rojas-Barahona,Pei-Hao Su, Stefan Ultes, David Vandyke, Steve Young

Cambridge University Engineering Department,Trumpington Street, Cambridge, CB2 1PZ, UK

{thw28,mg436,nm480,lmr46,phs26,su259,djv27,sjy11}@cam.ac.uk

Abstract

Recently a variety of LSTM-based condi-tional language models (LM) have been ap-plied across a range of language generationtasks. In this work we study various model ar-chitectures and different ways to represent andaggregate the source information in an end-to-end neural dialogue system framework. Amethod called snapshot learning is also pro-posed to facilitate learning from supervisedsequential signals by applying a companioncross-entropy objective function to the condi-tioning vector. The experimental and analyt-ical results demonstrate firstly that competi-tion occurs between the conditioning vectorand the LM, and the differing architecturesprovide different trade-offs between the two.Secondly, the discriminative power and trans-parency of the conditioning vector is key toproviding both model interpretability and bet-ter performance. Thirdly, snapshot learningleads to consistent performance improvementsindependent of which architecture is used.

1 Introduction

Recurrent Neural Network (RNN)-based condi-tional language models (LM) have been shown tobe very effective in tackling a number of real worldproblems, such as machine translation (MT) (Choet al., 2014) and image caption generation (Karpa-thy and Fei-Fei, 2015). Recently, RNNs were ap-plied to task of generating sentences from an ex-plicit semantic representation (Wen et al., 2015a).Attention-based methods (Mei et al., 2016) andLong Short-term Memory (LSTM)-like (Hochreiter

and Schmidhuber, 1997) gating mechanisms (Wen etal., 2015b) have both been studied to improve gen-eration quality. Although it is now clear that LSTM-based conditional LMs can generate plausible nat-ural language, less effort has been put in compar-ing the different model architectures. Furthermore,conditional generation models are typically testedon relatively straightforward tasks conditioned ona single source (e.g. a sentence or an image) andwhere the goal is to optimise a single metric (e.g.BLEU). In this work, we study the use of condi-tional LSTMs in the generation component of neu-ral network (NN)-based dialogue systems which de-pend on multiple conditioning sources and optimis-ing multiple metrics.

Neural conversational agents (Vinyals and Le,2015; Shang et al., 2015) are direct extensions ofthe sequence-to-sequence model (Sutskever et al.,2014) in which a conversation is cast as a source totarget transduction problem. However, these mod-els are still far from real world applications be-cause they lack any capability for supporting domainspecific tasks, for example, being able to interactwith databases (Sukhbaatar et al., 2015; Yin et al.,2016) and aggregate useful information into their re-sponses. Recent work by Wen et al. (2016a), how-ever, proposed an end-to-end trainable neural dia-logue system that can assist users to complete spe-cific tasks. Their system used both distributed andsymbolic representations to capture user intents, andthese collectively condition a NN language genera-tor to generate system responses. Due to the diver-sity of the conditioning information sources, the bestway to represent and combine them is non-trivial.

In Wen et al. (2016a), the objective function forlearning the dialogue policy and language generatordepends solely on the likelihood of the output sen-tences. However, this sequential supervision signalmay not be informative enough to learn a good con-ditioning vector representation resulting in a gener-ation process which is dominated by the LM. Thiscan often lead to inappropriate system outputs.

In this paper, we therefore also investigate the useof snapshot learning which attempts to mitigate thisproblem by heuristically applying companion super-vision signals to a subset of the conditioning vector.This idea is similar to deeply supervised nets (Leeet al., 2015) in which the final cost from the out-put layer is optimised together with the companionsignals generated from each intermediary layer. Wehave found that snapshot learning offers several ben-efits: (1) it consistently improves performance; (2) itlearns discriminative and robust feature representa-tions and alleviates the vanishing gradient problem;(3) it appears to learn transparent and interpretablesubspaces of the conditioning vector.

2 Related Work

Machine learning approaches to task-oriented di-alogue system design have cast the problem asa partially observable Markov Decision Process(POMDP) (Young et al., 2013) with the aim ofusing reinforcement learning (RL) to train dia-logue policies online through interactions with realusers (Gasic et al., 2013). In order to make RLtractable, the state and action space must be care-fully designed (Young et al., 2010) and the un-derstanding (Henderson et al., 2014; Mrksic et al.,2015) and generation (Wen et al., 2015b; Wen et al.,2016b) modules were assumed available or trainedstandalone on supervised corpora. Due to the under-lying hand-coded semantic representation (Traum,1999), the conversation is far from natural and thecomprehension capability is limited. This motivatesthe use of neural networks to model dialogues fromend to end as a conditional generation problem.

Interest in generating natural language using NNscan be attributed to the success of RNN LMs forlarge vocabulary speech recognition (Mikolov etal., 2010; Mikolov et al., 2011). Sutskever etal. (2011) showed that plausible sentences can be

obtained by sampling characters one by one fromthe output layer of an RNN. By conditioning anLSTM on a sequence of characters, Graves (2013)showed that machines can synthesise handwritingindistinguishable from that of a human. Later on,this idea has been tried in several research fields,for example, generating image captions by condi-tioning an RNN on a convolutional neural network(CNN) output (Karpathy and Fei-Fei, 2015; Xu etal., 2015); translating a source to a target languageby conditioning a decoder LSTM on top of an en-coder LSTM (Cho et al., 2014; Bahdanau et al.,2015); or generating natural language by condition-ing on a symbolic semantic representation (Wen etal., 2015b; Mei et al., 2016). Among all these meth-ods, attention-based mechanisms (Bahdanau et al.,2015; Hermann et al., 2015; Ling et al., 2016) havebeen shown to be very effective improving perfor-mance using a dynamic source aggregation strategy.

To model dialogue as conditional generation,a sequence-to-sequence learning (Sutskever et al.,2014) framework has been adopted. Vinyals and Le(2015) trained the same model on several conversa-tion datasets and showed that the model can gener-ate plausible conversations. However, Serban et al.(2015b) discovered that the majority of the gener-ated responses are generic due to the maximum like-lihood criterion, which was latter addressed by Liet al. (2016a) using a maximum mutual informationdecoding strategy. Furthermore, the lack of a con-sistent system persona was also studied in Li et al.(2016b). Despite its demonstrated potential, a ma-jor barrier for this line of research is data collection.Many works (Lowe et al., 2015; Serban et al., 2015a;Dodge et al., 2016) have investigated conversationdatasets for developing chat bot or QA-like generalpurpose conversation agents. However, collectingdata to develop goal oriented dialogue systems thatcan help users to complete a task in a specific do-main remains difficult. In a recent work by Wen etal. (2016a), this problem was addressed by design-ing an online, parallel version of Wizard-of-Oz datacollection (Kelley, 1984) which allows large scaleand cheap in-domain conversation data to be col-lected using Amazon Mechanical Turk. An NN-based dialogue model was also proposed to learnfrom the collected dataset and was shown to be ableto assist human subjects to complete specific tasks.

Snapshot learning can be viewed as a special formof weak supervision (also known as distant- or selfsupervision) (Craven and Kumlien, 1999; Snow etal., 2004), in which supervision signals are heuristi-cally labelled by matching unlabelled corpora withentities or attributes in a structured database. It hasbeen widely applied to relation extraction (Mintz etal., 2009) and information extraction (Hoffmann etal., 2011) in which facts from a knowledge base (e.g.Freebase) were used as objectives to train classifiers.Recently, self supervision was also used in mem-ory networks (Hill et al., 2016) to improve the dis-criminative power of memory attention. Conceptu-ally, snapshot learning is related to curriculum learn-ing (Bengio et al., 2009). Instead of learning eas-ier examples before difficult ones, snapshot learningcreates an easier target for each example. In prac-tice, snapshot learning is similar to deeply super-vised nets (Lee et al., 2015) in which companion ob-jectives are generated from intermediary layers andoptimised altogether with the output objective.

3 Neural Dialogue SystemThe testbed for this work is a neural network-basedtask-oriented dialogue system proposed by Wen etal. (2016a). The model casts dialogue as a sourceto target sequence transduction problem (modelledby a sequence-to-sequence architecture (Sutskeveret al., 2014)) augmented with the dialogue his-tory (modelled by a belief tracker (Henderson etal., 2014)) and the current database search outcome(modelled by a database operator). The model con-sists of both encoder and decoder modules. The de-tails of each module are given below.

3.1 Encoder ModuleAt each turn t, the goal of the encoder is to producea distributed representation of the system action m

t

,which is then used to condition a decoder to gen-erate the next system response in skeletal form1. Itconsists of four submodules: intent network, belieftracker, database operator, and policy network.Intent Network The intent network takes a se-quence of tokens1 and converts it into a sentence em-bedding representing the user intent using an LSTM

1Delexicalisation: slots and values are replaced by generictokens (e.g. keywords like Chinese food are replaced by[v.food] [s.food] to allow weight sharing.

network. The hidden layer of the LSTM at the lastencoding step z

t

is taken as the representation. Asmentioned in Wen et al. (2016a), this representationcan be viewed as a distributed version of the speechact (Traum, 1999) used in traditional systems.Belief Trackers In addition to the intent network,the neural dialogue system uses a set of slot-basedbelief trackers (Henderson et al., 2014; Mrksic et al.,2015) to track user requests. By taking each user in-put as new evidence, the task of a belief tracker isto maintain a multinomial distribution p over valuesv 2 V

s

for each informable slot2 s, and a binarydistribution for each requestable slot2. These prob-ability distributions ps

t

are called belief states of thesystem. The belief states ps

t

, together with the intentvector z

t

, can be viewed as the system’s comprehen-sion of the user requests up to turn t.Database Operator Based on the belief states ps

t

,a DB query is formed by taking the union of themaximum values of each informable slot. A vectorx

t

representing different degrees of matching in theDB (no match, 1 match, ... or more than 5 matches)is produced by counting the number of matched enti-ties and expressing it as a 6-bin 1-hot encoding. If x

t

is not zero, an associated entity pointer is maintainedwhich identifies one of the matching DB entities se-lected at random. The entity pointer is updated if thecurrent entity no longer matches the search criteria;otherwise it stays the same.Policy Network Based on the vectors z

t

, ps

t

, andx

t

from the above three modules, the policy networkcombines them into a single action vector m

t

by athree-way matrix transformation,

m

t

= tanh(Wzm

z

t

+W

xm

x

t

+P

s2GW

s

pm

p

s

t

) (1)

where matrices Wzm

, Ws

pm

, and W

xm

are param-eters and G is the domain ontology.

3.2 Decoder ModuleConditioned on the system action vector m

t

pro-vided by the encoder module, the decoder mod-ule uses a conditional LSTM LM to generate therequired system output token by token in skeletalform1. The final system response can then be formed

2Informable slots are slots that users can use to constrain thesearch, such as food type or price range; Requestable slots areslots that users can ask a value for, such as phone number. Thisinformation is specified in the domain ontology.

(a) Language model type LSTM (b) Memory type LSTM (c) Hybrid type LSTM

Figure 1: Three different conditional generation architectures.

by substituting the actual values of the database en-tries into the skeletal sentence structure.

3.2.1 Conditional Generation NetworkIn this paper we study and analyse three differentvariants of LSTM-based conditional generation ar-chitectures:Language Model Type The most straightforwardway to condition the LSTM network on additionalsource information is to concatenate the condition-ing vector m

t

together with the input word embed-ding w

j

and previous hidden layer hj�1,

0

BB@

i

j

f

j

o

j

c

j

1

CCA =

0

BB@

sigmoidsigmoidsigmoidtanh

1

CCAW4n,3n

0

@m

t

w

j

h

j�1

1

A

c

j

= f

j

� c

j�1 + i

j

� c

j

h

j

= o

j

� tanh(cj

)

where index j is the generation step, n is the hiddenlayer size, i

j

, fj

,oj

2 [0, 1]n are input, forget, andoutput gates respectively, c

j

and c

j

are proposed cellvalue and true cell value at step j, and W4n,3n arethe model parameters. The model is shown in Fig-ure 1a. Since it does not differ significantly from theoriginal LSTM, we call it the language model type(lm) conditional generation network.Memory Type The memory type (mem) condi-tional generation network was introduced by Wen etal. (2015b), shown in Figure 1b, in which the condi-tioning vector m

t

is governed by a standalone read-ing gate r

j

. This reading gate decides how much in-formation should be read from the conditioning vec-tor and directly writes it into the memory cell c

j

,0

BB@

i

j

f

j

o

j

r

j

1

CCA =

0

BB@

sigmoidsigmoidsigmoidsigmoid

1

CCAW4n,3n

0

@m

t

w

j

h

j�1

1

A

c

j

= tanh�W

c

(wj

� h

j�1)�

c

j

= f

j

� c

j�1 + i

j

� c

j

+ r

j

�m

t

h

j

= o

j

� tanh(cj

)

where W

c

is another weight matrix to learn. Theidea behind this is that the model isolates the con-ditioning vector from the LM so that the model hasmore flexibility to learn to trade off between the two.Hybrid Type Continuing with the same idea as thememory type network, a complete separation of con-ditioning vector and LM (except for the gate con-trolling the signals) is provided by the hybrid typenetwork shown in Figure 1c,

0

BB@

i

j

f

j

o

j

r

j

1

CCA =

0

BB@

sigmoidsigmoidsigmoidsigmoid

1

CCAW4n,3n

0

@m

t

w

j

h

j�1

1

A

c

j

= tanh�W

c

(wj

� h

j�1)�

c

j

= f

j

� c

j�1 + i

j

� c

j

h

j

= o

j

� tanh(cj

) + r

j

�m

t

This model was motivated by the fact that long-termdependency is not needed for the conditioning vec-tor because we apply this information at every step janyway. The decoupling of the conditioning vectorand the LM is attractive because it leads to better in-terpretability of the results and provides the potentialto learn a better conditioning vector and LM.

3.2.2 Attention and Belief RepresentationAttention An attention-based mechanism providesan effective approach for aggregating multiple infor-mation sources for prediction tasks. Like Wen et al.

(2016a), we explore the use of an attention mecha-nism to combine the tracker belief states in whichthe policy network in Equation 1 is modified as

m

j

t

= tanh(Wzm

z

t

+W

xm

x

t

+P

s2G ↵j

s

W

s

pm

p

s

t

)

where the attention weights ↵j

s

are calculated by,

↵j

s

= softmax�r

| tanh�W

r

· (vt

� p

s

t

�w

t

j

� h

t

j�1)��

where vt

= z

t

+x

t

and matrix W

r

and vector r areparameters to learn.Belief Representation The effect of different be-lief state representations on the end performance arealso studied. For user informable slots, the full beliefstate p

s

t

is the original state containing all categori-cal values; the summary belief state contains onlythree components: the summed value of all categor-ical probabilities, the probability that the user saidthey “don’t care” about this slot and the probabil-ity that the slot has not been mentioned. For userrequestable slots, on the other hand, the full beliefstate is the same as the summary belief state becausethe slot values are binary rather than categorical.

3.3 Snapshot LearningLearning conditional generation models from se-quential supervision signals can be difficult, becauseit requires the model to learn both long-term worddependencies and potentially distant source encod-ing functions. To mitigate this difficulty, we in-troduce a novel method called snapshot learningto create a vector of binary labels ⌥

j

t

2 [0, 1]d,d < dim(mj

t

) as the snapshot of the remaining partof the output sentence T

t,j:|Tt| from generation stepj. Each element of the snapshot vector is an indica-tor function of a certain event that will happen in thefuture, which can be obtained either from the sys-tem response or dialogue context at training time. Acompanion cross entropy error is then computed toforce a subset of the conditioning vector mj

t

⇢ m

j

t

to be close to the snapshot vector,

Lss

(·) = �P

t

Pj

E[H(⌥j

t

, mj

t

)] (2)

where H(·) is the cross entropy function, ⌥j

t

and mj

t

are elements of vectors ⌥j

t

and m

j

t

, respectively. Inorder to make the tanh activations of mj

t

compat-ible with the 0-1 snapshot labels, we squeeze each

Figure 2: The idea of snapshot learning. The snap-shot vector was trained with additional supervisionson a set of indicator functions heuristically labelledusing the system response.

value of mj

t

by adding 1 and dividing by 2 beforecomputing the cost.

The indicator functions we use in this work havetwo forms: (1) whether a particular slot value (e.g.,[v.food]1) is going to occur, and (2) whether the sys-tem has offered a venue3, as shown in Figure 2. Theoffer label in the snapshot is produced by checkingthe delexicalised name token ([v.name]) in the en-tire dialogue. If it has occurred, every label in sub-sequent turns is labelled with 1. Otherwise it is la-belled with 0. To create snapshot targets for a partic-ular slot value, the output sentence is matched withthe corresponding delexicalised token turn by turn,per generation step. At each generation step, the tar-get is labelled with 0 if that delexicalised token hasbeen generated; otherwise it is set to 1. However, forthe models without attention, the targets per turn areset to the same because the condition vector will notbe able to learn the dynamically changing behaviourwithout attention.

4 Experiments

Dataset The dataset used in this work was col-lected in the Wizard-of-Oz online data collection de-scribed by Wen et al. (2016a), in which the task ofthe system is to assist users to find a restaurant inCambridge, UK area. There are three informableslots (food, pricerange, area) that users can use toconstrain the search and six requestable slots (ad-dress, phone, postcode plus the three informable

3Details of the specific application used in this study aregiven in Section 4 below.

Architecture Belief Success(%) SlotMatch(%) T5-BLEU T1-BLEU

Belief state representationlm full 72.6 / 74.5 52.1 / 60.3* 0.207 / 0.229* 0.216 / 0.238*

lm summary 74.5 / 76.5 57.4 / 61.2* 0.221 / 0.231* 0.227 / 0.240*

Conditional architecturelm summary 74.5 / 76.5 57.4 / 61.2* 0.221 / 0.231* 0.227 / 0.240*

mem summary 75.5 / 77.5 59.2 / 61.3* 0.222 / 0.232* 0.231 / 0.243*

hybrid summary 76.1 / 79.2 52.4 / 60.6* 0.202 / 0.228* 0.212 / 0.237*

Attention-based modellm summary 79.4 / 78.2 60.6 / 60.2 0.228 / 0.231 0.239 / 0.241mem summary 76.5 / 80.2* 57.4 / 61.0* 0.220 / 0.229 0.228 / 0.239hybrid summary 79.0 / 81.8* 56.2 / 60.5* 0.214 / 0.227* 0.224 / 0.240*

Table 1: Performance comparison of different model architectures, belief state representations, and snapshotlearning. The numbers to the left and right of the / sign are learning without and with snapshot, respectively.The model with the best performance on a particular metric (column) is shown in bold face. The lm models inConditional architecture and Attention-based model are the same models as in Wen et al. (2016a). Statisticalsignificance was computed using two-tailed Wilcoxon Signed-Rank Test (* p <0.05) to compare models w/and w/o snapshot learning.

slots) that the user can ask a value for once a restau-rant has been offered. There are 676 dialogues in thedataset (including both finished and unfinished dia-logues) and approximately 2750 turns in total. Thedatabase contains 99 unique restaurants.Training The training procedure was divided intotwo stages. Firstly, the belief tracker parameters✓b

were pre-trained using cross entropy errors be-tween tracker labels and predictions. Having fixedthe tracker parameters, the remaining parts of themodel ✓\b are trained using the cross entropy errorsfrom the generation network LM,

L(✓\b) = �P

t

Pj

H(yt

j

,pt

j

) + �Lss

(·) (3)

where yt

j

and p

t

j

are output token targets and predic-tions respectively, at turn t of output step j, L

ss

(·)is the snapshot cost from Equation 2, and � is thetradeoff parameter in which we set to 1 for all mod-els trained with snapshot learning. We treated eachdialogue as a batch and used stochastic gradient de-scent with a small l2 regularisation term to train themodel. The collected corpus was partitioned intoa training, validation, and testing sets in the ratio3:1:1. Early stopping was implemented based on thevalidation set considering only LM log-likelihoods.Gradient clipping was set to 1. The hidden layersizes were set to 50, and the weights were randomly

initialised between -0.3 and 0.3 including word em-beddings. The vocabulary size is around 500 forboth input and output, in which rare words andwords that can be delexicalised have been removed.

Decoding In order to compare models trained withdifferent recipes rather than decoding strategies, wedecode all the trained models with the average logprobability of tokens in the sentence. We appliedbeam search with a beamwidth equal to 10, thesearch stops when an end-of-sentence token is gen-erated. In order to consider language variability, weran decoding until 5 candidates were obtained andperformed evaluation on them.

Metrics We compared models trained with differ-ent recipes by performing a corpus-based evaluationin which the model is used to predict each systemresponse in the held-out test set. Three evaluationmetrics were used: BLEU score (on top-1 and top-5 candidates) (Papineni et al., 2002), slot matchingrate and objective task success rate (Su et al., 2015).The dialogue is marked as successful if both: (1)the offered entity matches the task that was speci-fied to the user, and (2) the system answered all theassociated information requests (e.g. what is the ad-dress?) from the user. The slot matching rate is thepercentage of delexicalised tokens (e.g. [s.food] and[v.area]1) appear in the candidate also appear in the

(a) Hybrid LSTM w/o snapshot learning (b) Hybrid LSTM w/ snapshot learning

Figure 3: Learned attention heat maps over trackers. The first three columns in each figure are informableslot trackers and the rest are requestable slot trackers. The generation model is the hybrid type LSTM.

reference. We computed the BLEU scores on theskeletal sentence forms before substituting with theactual entity values. All the results were averagedover 10 random initialised networks.Results Table 1 shows the evaluation results. Thenumbers to the left and right of each table cell are thesame model trained w/o and w/ snapshot learning.The first observation is that snapshot learning con-sistently improves on most metrics regardless of themodel architecture. This is especially true for BLEUscores. We think this may be attributed to the morediscriminative conditioning vector learned throughthe snapshot method, which makes the learning ofthe conditional LM easier.

In the first block belief state representation, wecompare the effect of two different belief represen-tations. As can be seen, using a succinct represen-tation is better (summary>full) because the iden-tity of each categorical value in the belief state doesnot help when the generation decisions are done inskeletal form. In fact, the full belief state representa-tion may encourage the model to learn incorrect co-adaptation among features when the data is scarce.

In the conditional architecture block, we com-pare the three different conditional generation archi-tectures as described in section 3.2.1. This resultshows that the language model type (lm) and mem-ory type (mem) networks perform better in terms ofBLEU score and slot matching rate, while the hybridtype (hybrid) networks achieve higher task success.This is probably due to the degree of separation be-

Model i

j

f

j

r

j

/oj

hybrid, full 0.567 0.502 0.405hybrid, summary 0.539 0.540 0.428+ att. 0.540 0.559 0.459

Table 2: Average activation of gates on test set.

tween the LM and conditioning vector: a couplingapproach (lm, mem) sacrifices the conditioning vec-tor but learns a better LM and higher BLEU; whilea complete separation (hybrid) learns a better condi-tioning vector and offers a higher task success.

Lastly, in the attention-based model block wetrain the three architectures with the attention mech-anism and compare them again. Firstly, the char-acteristics of the three models we observed abovealso hold for attention-based models. Secondly, wefound that the attention mechanism improves allthe three architectures on task success rate but notBLEU scores. This is probably due to the limita-tions of using n-gram based metrics like BLEU toevaluate the generation quality (Stent et al., 2005).

5 Model Analysis

Gate Activations We first studied the average ac-tivation of each individual gate in the models by av-eraging them when running generation on the testset. We analysed the hybrid models because theirreading gate to output gate activation ratio (r

j

/oj

)shows clear tradeoff between the LM and the con-ditioning vector components. As can be seen in Ta-

(a)

(b)

(c)

Figure 4: Three example responses generated from the hybrid model trained with snapshot and attention.Each line represents a neuron that detects a particular snapshot event.

ble 2, we found that the average forget gate activa-tions (f

j

) and the ratio of the reading gate to the out-put gate activation (r

j

/oj

) have strong correlationsto performance: a better performance (row 3>row2>row 1) seems to come from models that can learna longer word dependency (higher forget gate f

t

ac-tivations) and a better conditioning vector (thereforehigher reading to output gate ratio r

j

/oj

).Learned Attention We have visualised thelearned attention heat map of models trained withand without snapshot learning in Figure 3. The at-tention is on both the informable slot trackers (firstthree columns) and the requestable slot trackers (theother columns). We found that the model trainedwith snapshot learning (Figure 3b) seems to pro-duce a more accurate and discriminative attentionheat map comparing to the one trained without it(Figure 3a). This may contribute to the better perfor-

mance achieved by the snapshot learning approach.

Snapshot Neurons As mentioned earlier, snap-shot learning forces a subspace of the condition-ing vector m

j

t

to become discriminative and in-terpretable. Three example generated sentencestogether with the snapshot neuron activations areshown in Figure 4. As can be seen, when generat-ing words one by one, the neuron activations werechanging to detect different events they were as-signed by the snapshot training signals: e.g. in Fig-ure 4b the light blue and orange neurons switchedtheir domination role when the token [v.address]was generated; the offered neuron is in a high ac-tivation state in Figure 4b because the system wasoffering a venue, while in Figure 4a it is not acti-vated because the system was still helping the userto find a venue.

6 Conclusion and Future Work

This paper has investigated different conditionalgeneration architectures and a novel method calledsnapshot learning to improve response generation ina neural dialogue system framework. The resultsshowed three major findings. Firstly, although thehybrid type model did not rank highest on all met-rics, it is nevertheless preferred because it achievedthe highest task success and also it provided more in-terpretable results. Secondly, snapshot learning pro-vided gains on virtually all metrics regardless of thearchitecture used. The analysis suggested that thebenefit of snapshot learning mainly comes from themore discriminative and robust subspace represen-tation learned from the heuristically labelled com-panion signals, which in turn facilitates optimisationof the final target objective. Lastly, the results sug-gested that by making a complex system more inter-pretable at different levels not only helps our under-standing but also leads to the highest success rates.

However, there is still much work left to do. Thiswork focused on conditional generation architec-tures and snapshot learning in the scenario of gen-erating dialogue responses. It would be very help-ful if the same comparison could be conducted inother application domains such as machine transla-tion or image caption generation so that a wider viewof the effectiveness of these approaches can be as-sessed. Furthermore, removing slot-value delexical-isation and learning confirmation behaviour in noisyspeech conditions are also main research problemsfrom the system development prospective.

Acknowledgments

Tsung-Hsien Wen and David Vandyke are supportedby Toshiba Research Europe Ltd, Cambridge Re-search Laboratory.

ReferencesDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-

gio. 2015. Neural machine translation by jointlylearning to align and translate. In ICLR.

Yoshua Bengio, Jerome Louradour, Ronan Collobert, andJason Weston. 2009. Curriculum learning. In ICML.

Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre,Fethi Bougares, Holger Schwenk, and Yoshua Bengio.2014. Learning phrase representations using RNN

encoder-decoder for statistical machine translation. InEMNLP.

Mark Craven and Johan Kumlien. 1999. Constructingbiological knowledge bases by extracting informationfrom text sources. In ISMB.

Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bor-des, Sumit Chopra, Alexander Miller, Arthur Szlam,and Jason Weston. 2016. Evaluating prerequi-site qualities for learning end-to-end dialog systems.ICLR.

Milica Gasic, Catherine Breslin, Matthew Henderson,Dongho Kim, Martin Szummer, Blaise Thomson, Pir-ros Tsiakoulis, and Steve Young. 2013. On-line policyoptimisation of bayesian spoken dialogue systems viahuman interaction. In ICASSP.

Alex Graves. 2013. Generating sequences with recurrentneural networks. arXiv preprint:1308.0850.

Matthew Henderson, Blaise Thomson, and Steve Young.2014. Word-based dialog state tracking with recurrentneural networks. In SIGdial.

Karl Moritz Hermann, Tomas Kocisky, Edward Grefen-stette, Lasse Espeholt, Will Kay, Mustafa Suleyman,and Phil Blunsom. 2015. Teaching machines to readand comprehend. In NIPS.

Felix Hill, Antoine Bordes, Sumit Chopra, and Jason We-ston. 2016. The goldilocks principle: Reading chil-dren’s books with explicit memory representations. InICLR.

Sepp Hochreiter and Jurgen Schmidhuber. 1997. Longshort-term memory. Neural Computation.

Raphael Hoffmann, Congle Zhang, Xiao Ling, LukeZettlemoyer, and Daniel S. Weld. 2011. Knowledge-based weak supervision for information extraction ofoverlapping relations. In ACL.

Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descrip-tions. In CVPR.

John F. Kelley. 1984. An iterative design methodologyfor user-friendly natural language office informationapplications. ACM Transaction on Information Sys-tems.

Chen-Yu Lee, Saining Xie, Patrick Gallagher, ZhengyouZhang, and Zhuowen Tu. 2015. Deeply-supervisednets. In AISTATS.

Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao,and Bill Dolan. 2016a. A diversity-promoting ob-jective function for neural conversation models. InNAACL-HLT.

Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao,and Bill Dolan. 2016b. A persona-based neural con-versation model. arXiv perprint:1603.06155.

Wang Ling, Edward Grefenstette, Karl Moritz Hermann,Tomas Kocisky, Andrew Senior, Fumin Wang, and

Phil Blunsom. 2016. Latent predictor networks forcode generation. arXiv preprint:1603.06744.

Ryan Lowe, Nissan Pow, Iulian Serban, and JoellePineau. 2015. The ubuntu dialogue corpus: A largedataset for research in unstructured multi-turn dia-logue systems. In SIGdial.

Hongyuan Mei, Mohit Bansal, and Matthew R. Walter.2016. What to talk about and how? selective gen-eration using lstms with coarse-to-fine alignment. InNAACL.

Tomas Mikolov, Martin Karafit, Lukas Burget, JanCernocky, and Sanjeev Khudanpur. 2010. Recurrentneural network based language model. In InterSpeech.

Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan H.Cernocky, and Sanjeev Khudanpur. 2011. Exten-sions of recurrent neural network language model. InICASSP.

Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky.2009. Distant supervision for relation extraction with-out labeled data. In ACL.

Nikola Mrksic, Diarmuid O Seaghdha, Blaise Thomson,Milica Gasic, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2015. Multi-domainDialog State Tracking using Recurrent Neural Net-works. In ACL.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evalua-tion of machine translation. In ACL.

Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, andJoelle Pineau. 2015a. A survey of available cor-pora for building data-driven dialogue systems. arXivpreprint:1512.05742.

Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio,Aaron C. Courville, and Joelle Pineau. 2015b. Hier-archical neural network generative models for moviedialogues. In AAAI.

Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neu-ral responding machine for short-text conversation. InACL.

Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2004.Learning syntactic patterns for automatic hypernymdiscovery. In NIPS.

Amanda Stent, Matthew Marge, and Mohit Singhai.2005. Evaluating evaluation methods for generationin the presence of variation. In CICLing 2005.

Pei-Hao Su, David Vandyke, Milica Gasic, Dongho Kim,Nikola Mrksic, Tsung-Hsien Wen, and Steve J. Young.2015. Learning from real users: Rating dialogue suc-cess with neural networks for reinforcement learningin spoken dialogue systems. In Interspeech.

Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, andRob Fergus. 2015. End-to-end memory networks. InNIPS.

Ilya Sutskever, James Martens, and Geoffrey E. Hinton.2011. Generating text with recurrent neural networks.In ICML.

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.Sequence to sequence learning with neural networks.In NIPS.

David R. Traum, 1999. Foundations of Rational Agency,chapter Speech Acts for Dialogue Agents. Springer.

Oriol Vinyals and Quoc V. Le. 2015. A neural conversa-tional model. In ICML Deep Learning Workshop.

Tsung-Hsien Wen, Milica Gasic, Dongho Kim, NikolaMrksic, Pei-Hao Su, David Vandyke, and SteveYoung. 2015a. Stochastic language generation in di-alogue using recurrent neural networks with convolu-tional sentence reranking. In SIGdial.

Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-HaoSu, David Vandyke, and Steve Young. 2015b. Seman-tically conditioned lstm-based natural language gener-ation for spoken dialogue systems. In EMNLP.

Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-HaoSu, Stefan Ultes, David Vandyke, and Steve Young.2016a. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint:1604.04562.

Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-HaoSu, David Vandyke, and Steve Young. 2016b. Multi-domain neural network language generation for spo-ken dialogue systems. In NAACL-HLT.

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho,Aaron C. Courville, Ruslan Salakhutdinov, Richard S.Zemel, and Yoshua Bengio. 2015. Show, attend andtell: Neural image caption generation with visual at-tention. In ICML.

Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao.2016. Neural enquirer: Learning to query tables. InIJCAI.

Steve Young, Milica Gasic, Simon Keizer, FrancoisMairesse, Jost Schatzmann, Blaise Thomson, and KaiYu. 2010. The hidden information state model: Apractical framework for pomdp-based spoken dialoguemanagement. Computer, Speech and Language.

Steve Young, Milica Gasic, Blaise Thomson, and Ja-son D. Williams. 2013. Pomdp-based statistical spo-ken dialog systems: A review. Proceedings of IEEE.