Self-Explaining Structures Improve NLP Models

Zijun Sun♣, Chun Fan♠F, Qinghong Han♣, Xiaofei Sun♣,Yuxian Meng♣, Fei Wu� and Jiwei Li �♣

�Zhejiang University, ♠Computer Center of Peking UniversityFPeng Cheng Laboratory, ♣ Shannon.AI

{zijun_sun, jiwei_li}@shannonai.com, fanchun@pku.edu.cn, wufei@zju.edu.cn

Abstract

Existing approaches to explaining deep learn-ing models in NLP usually suffer from two ma-jor drawbacks: (1) the main model and the ex-plaining model are decoupled: an additionalprobing or surrogate model is used to interpretan existing model, and thus existing explain-ing tools are not self-explainable; (2) the prob-ing model is only able to explain a model’spredictions by operating on low-level featuresby computing saliency scores for individualwords but are clumsy at high-level text unitssuch as phrases, sentences, or paragraphs.

To deal with these two issues, in this paper,we propose a simple yet general and effec-tive self-explaining framework for deep learn-ing models in NLP. The key point of the pro-posed framework is to put an additional layer,as is called by the interpretation layer, on topof any existing NLP model. This layer aggre-gates the information for each text span, whichis then associated with a specific weight, andtheir weighted combination is fed to the soft-max function for the final prediction.

The proposed model comes with the followingmerits: (1) span weights make the model self-explainable and do not require an additionalprobing model for interpretation; (2) the pro-posed model is general and can be adapted toany existing deep learning structures in NLP;(3) the weight associated with each text spanprovides direct importance scores for higher-level text units such as phrases and sentences.We for the first time show that interpretabil-ity does not come at the cost of performance:a neural model of self-explaining features ob-tains better performances than its counterpartwithout the self-explaining nature, achieving anew SOTA performance of 59.1 on SST-5 anda new SOTA performance of 92.3 on SNLI. 1

1Code is available at https://github.com/ShannonAI/Self_Explaining_Structures_Improve_NLP_Models

1 Introduction

A long-term criticism against deep learning mod-els is the lack of interpretability (Simonyan et al.,2013; Bach et al., 2015a; Montavon et al., 2017;Kindermans et al., 2017). The black-box natureof neural models not only significantly limits thescope of applications of deep learning models (e.g.,in biomedical or legal domains), but also hindersmodel behavior analysis and error analysis.

To enhance neural models’ interpretability, vari-ous approaches to rationalize neural models’ pre-dictions have been proposed (details see Section2). Existing interpretation models have two majordrawbacks. Firstly, they are not self-explainableand require a probing model to be additionally builtto interpret the original model. Building the prob-ing model is not only an additional burden, butmore importantly, the intrinsic decoupling natureof the two models makes the probing model onlyable to provide approximate interpretation results.Take the gradient-based saliency model (Simonyanet al., 2013; Li et al., 2015; Selvaraju et al., 2017) asan example, the derivative of the target probability(or the logit) w.r.t. the input dimensions is only anapproximate feature weight in the first-order Taylorexpansion, with all higher order items omitted.

Secondly, interpretation models mostly focus onlearning word-level importance scores assigned bythe the main model and are hard to be adapted tohigher-level text units such as phrases, sentences,or paragraphs. A straightforward way to computethe saliency score for a phrase or a sentence couldbe averaging the scores for its constituent words.Unfortunately, this over-simplified strategy is in-adequate to capture the semantic composition inlanguage, which involves multiple layers of highlynon-linear operations in neural nets, and can thusresult in high risk of misinterpretation. As pointed

arX

iv:2

012.

0178

6v2

[cs

.CL

] 9

Dec

202

0

by Murdoch et al. (2018), interpreting neural netpredictions should go beyond word level.

We raise the following question: what makes agood interpretation method for NLP? Firstly,the method should be self-explainable and noadditional probing model is needed. Secondly,the model should offer precise and clear saliencyscores for any level of text units. Thirdly, the self-explainable nature does not tradeoff performances.

Towards these three purposes, in this paper, we pro-pose a self-explainable framework for deep neuralmodels in the context of NLP. The key point of theproposed framework is to put an additional layer,as is called the interpretation layer on top of any ex-isting NLP model, and this layer aggregates the in-formation for all (i.e., O(n2)) text spans. Each textspan is associated with a specific weight, and theirweighted combination is fed to the softmax func-tion for the final prediction. The proposed struc-ture offers following advantages: (1) the model isself-explainable: the interpretation layer is trainedalong with the objective, with no probing modelneeded. Weights at the interpretation layer can di-rectly be used as saliency scores for correspondingtext spans; (2) since the saliency score for any textspan can be straightforwardly derived from the in-terpretation layer, the model offers direct, preciseand clear saliency scores for any level of text unitsbeyond word level. (3) the interpretation layer col-lects information for each text span in the form ofrepresentations, and forwards the weighted sum tothe final prediction layer. Therefore, no additionalinformation is incorporated, nor any informationis lost at the interpretation layer. This makes themodel not come at the cost of performances.

The proposed framework is general, and can beadapted to any prevalent deep learning structurein NLP. We show that the proposed framework (1)offers clear model interpretations; (2) facilitateserror analysis; (3) can help downstream NLP taskssuch as adversarial example generation; and (4)most importantly, for the first time proves that self-explainable nature does not come at the cost ofperformances, but rather, leads to better results inNLP, achieving a new SOTA performance of 59.1on SST-5 and a new SOTA performance of 92.3 onSNLI.

2 Related Work

Rationalizing model predictions is of growing in-terest (Ribeiro et al., 2016; Shrikumar et al., 2017;Bach et al., 2015b; Kindermans et al., 2017; Mon-tavon et al., 2017; Schwab and Karlen, 2019; Ar-rieta et al., 2020). In NLP, approaches to interpretneural models include extracting pieces of inputtext, called “rationales”, as justifications to modelpredictions (Lei et al., 2016; Chang et al., 2019;DeYoung et al., 2020; Jain et al., 2020), study-ing the efficacy and dynamics of hidden statesin recurrent networks (Karpathy et al., 2015; Shiet al., 2016; Greff et al., 2016; Strobelt et al., 2016),and applying variants of the attention mechanism(Bahdanau et al., 2014) to interpret model behav-iors (Jain and Wallace, 2019; Serrano and Smith,2019; Wiegreffe and Pinter, 2019; Vashishth et al.,2019; Rogers et al., 2020). Lei et al. (2016) pro-posed to extract text snippets as model explanations.Rajani et al. (2019) collected human rationalesfor commonsense reasoning. Other works (Chenet al., 2019; Lehman et al., 2019; Chai et al., 2020)trained independent models to extract supportingsentences as auxiliary guidelines for downstreamtasks. Using attentions as a tool for model inter-pretation, Ghaeini et al. (2018) visualized atten-tion heatmaps to understand how natural languageinference models build interactions between twosentences; Vig and Belinkov (2019); Tenney et al.(2019); Clark et al. (2019); Htut et al. (2019) ana-lyzed the attention structures by plotting heatmaps,and found that meaningful linguistic patterns ex-ist in different heads and layers. Despite the in-terpretability the attention mechanism offers, Ser-rano and Smith (2019); Jain and Wallace (2019)observed the highly inconsistency between atten-tions and predictors, and suggested that attentionsshould not be treated as justification for a decision.

Saliency methods are widely used in computer vi-sion (Simonyan et al., 2013; Zeiler and Fergus,2014; Springenberg et al., 2014; Adler et al., 2018;Datta et al., 2016; Srinivas and Fleuret, 2019) andNLP (Denil et al., 2015; Li et al., 2015, 2016; Arraset al., 2016; Ebrahimi et al., 2017; Feng et al., 2018;Meng et al., 2020) for model interpretation. Thekey idea is to find the salient features responsiblefor a model’s prediction. Simonyan et al. (2013);Srinivas and Fleuret (2019) visualized the contribu-tions of input pixels by compute the derivatives ofthe label logit in the output layer with respect to the

input pixel. Adler et al. (2018); Datta et al. (2016)explained neural models by perturbing differentparts of the input, and compared the performancechange in downstream tasks to measure the impor-tance of perturbed features. In the context of NLP,Denil et al. (2015) used saliency maps to identifyand extract task-specific salient sentences from doc-uments to maximally preserve document topics andsemantics; Li et al. (2015) visualized word-levelsaliency maps to understand how individual wordsaffect model predictions; Ebrahimi et al. (2017)crafted white-box adversarial examples to find themost salient text-editing operations (flip, insertionand deletion) to trick models by computing deriva-tives w.r.t. these editing operations; Meng et al.(2020) combined the saliency method and the influ-ence function (Koh and Liang, 2017) to interpretmodel predictions from both learning history andinputs.

Our work is inspired by Melis and Jaakkola (2018),which achieves “self-explaining” by jointly train-ing a deep learning model and a linear regressionmodel with human-interpretable features. It isworth noting that the model in Melis and Jaakkola(2018) still requires a surrogate model, i.e., thelinear regression model with human-interpretablefeatures. Instead, the proposed framework doesnot require a surrogate model. Our work is alsoinspired by Selvaraju et al. (2017) in computer vi-sion, which computes the gradient w.r.t. featuremaps of the high-level convolutional layer to ob-tain high-level saliency scores.

3 The Self-Explaining Framework

3.1 NotationsGiven an input sequence x = {x1, x2, ..., xN},where N denotes the length of x. Let x(i, j) de-note the text span starting at index i, ending at indexj, where x(i, j) = {xi, xi+1, ..., xj−1, xj}. Wewish to predict the label y for x based on p(y|x).Let V denote vocabulary size, and word represen-tations are stored in W ∈ RV×D. The input x isassociated with the label y ∈ Y .

3.2 ModelFor illustration purposes, we use transformers(Vaswani et al., 2017) as the backbone to showhow the proposed framework works. The frame-work can be easily extended to other models suchas BiLSTMs or CNNs. An overview of the pro-posed model is shown in Figure 1.

the movie is my favorite

𝒘𝟏 𝒘𝟐 𝒘𝟑 𝒘𝟒 𝒘𝟓

Intermediate Layer

𝒉(4,5)𝒉(1,3) 𝒉(2,4)

𝜶(4,5)𝜶(1,3) 𝜶(2,4)

Interpretation Layer

෩𝒉

Positive

𝒉𝟏 𝒉𝟐 𝒉𝟑 𝒉𝟒 𝒉𝟓

…

…

𝒉(1,1)

𝜶(1,1)

…

Figure 1: An overview of the proposed model.

Input Layer Similar to standard deep learningmodels in NLP, the input layer for the proposedmodel consists of the stack of word representationsfor words in the input sequence, where xt is repre-sented as a D-dimensional vectorW [xt, :].

Intermediate Layer On top of the input layer arethe K encoder stacks, where each stack consistsof multi-head attentions, layer normalization andresidual connections. The representation for k-thlayer at position t is denoted by hk

t ∈ R1×D. Spe-cially, the presentation for the last layer at positiont is denoted by hK

t ∈ R1×D.

Span Infor Collecting Layer (SIC) To enable di-rect measurement for saliency of an arbitrary textspan, we place a Span Infor Collecting Layer ontop of intermediate layer. For an arbitrary textspan x(i, j), we first obtain a dense representa-tion h(i, j) to represent x(i, j), and h(i, j) needsto contain all semantic and syntactic informationstored in x(i, j). Concretely, h(i, j) is obtained bytaking the representation for the starting index atthe last intermediate layer hK

i , and the representa-tion for the ending index hK

j :

h(i, j) = F (hKi ,h

Kj ) (1)

where F denotes the mapping function, which beFFN or other forms. The details of F will be dis-cussed in section 3.3. The strategy of using the

starting and ending representations to represent atext span has been used in many recent works onspan-level features in texts (Li et al., 2019; Joshiet al., 2020; Wu et al., 2019). The SIC layer iter-ates over all text spans, and collects h(i, j) for alli ∈ [1, N ], j ∈ [i,N ], with the complexity beingO(N2).

Interpretation Layer The interpretation layer ag-gregates information from all text spans h(i, j):this is achieved by first assigning weights α(i, j)to each span x(i, j) and combing these representa-tions using weighted sum. The weight α(i, j) canbe obtained by first mapping h(i, j) to a scalar, andthen normalizing all α(i, j):

o(i, j) = h>h(i, j)

α(i, j) =exp(o(i, j))∑

i∈[1,N ],j∈[i,N ] exp(o(i, j))

(2)

where h ∈ R1×D. The output h ∈ R1×D from theinterpretation layer is the weighted average of allspan representations:

h =∑

i∈[1,N ],j∈[i,N ]

α(i, j)h(i, j) (3)

Output Layer Similar to the standard setup, theoutput layer of the proposed framework is the prob-ability distribution over labels using the softmaxfunction:

p(y|x) =u>y h∑y∈Y u

>y h

(4)

where uy ∈ RD×1. As can be seen, α(i, j) mea-sures how much x(i, j) contributes to the final rep-resentation h, and thus indicates the importanceof the text span x(i, j). The proposed strategy issimilar to gradient-based interpretation methods(Simonyan et al., 2013; Li et al., 2015). Using thechain rule, we have

∂p(y|x)∂h(i, j)

=∂p(y|x)∂h

∂h

∂h(i, j)

=∂p(y|x)∂h

[α(i, j) + h(i, j)

∂α(i, j)

∂h(i, j)

] (5)

Omitting the second part, ∂p(y|x)/∂h is approxi-mately in proportion to α(i, j). There is also a keyadvantage of the proposed framework over existinggradient-based interpretation methods, where theinterpretation layer allows gradients to be straight-forwardly computed with respect to an arbitrary

text span. This is not feasible for vanilla gradient-based methods: because of the highly entanglementof neural networks, it’s impossible to filter out theinformation of a specific text span from interme-diate layers. Only gradients w.r.t. the input layeroffers non-disentangling saliency scores, makingthe model only able to perform word-level expla-nations.

3.3 Efficient Computing

A practical issue with Eq.1 is the computation cost.If F takes the form of FFN, the computational com-plexity for one single text span is O(D2), leadingto a final complexity of O(N2D2) if all spans areiterated over. This is computationally unaffordablefor long texts. Towards efficient computations, wepropose that F takes the following form:

F (hi,hj) = tanh[W (hi,hj ,hi − hj ,hi ⊗ hj)](6)

where W = [W1,W2,W3,W4], Wi ∈ RD×D.⊗ denotes the pairwise dot between two vectors.Elements in (hi,hj ,hi−hj ,hi⊗hj) respectivelycaptures concatenation, element-wise differenceand element-wise closeness between the two vec-tors, the strategy of which is been used in recentwork to model interactions between two semanticrepresentations (Mou et al., 2015; Seo et al., 2016).

In Eq.6, W1hi, W2hj , W3hi and W3hj canbe computed in advance for all i and j, leadingto a computational complexity of O(ND). ForW4hi ⊗ hj , it can be factorized as

√W4hi ⊗√

W4hj , with each of the two parts being com-puted in advance. The final computational com-plexity of W (hi,hj ,hi − hj ,hi ⊗ hj) is thusO(ND). The element-wise tanh operation for allN2 spans leads to a cost of O(N2D), giving acomplexity of O(N2D) + O(ND) = O(N2D)for the SIC layer, which significantly cuts the cost.The computational cost for the interpretation layer,which requires the dot product between h and allh(i, j), is also O(N2D), leading to the final com-putational complexity O(N2D). In this way, wereduce the computation cost from O(N2D2) toO(N2D).

3.4 Training

The training objective is the standard cross entropyloss. An additional regularization on α is needed:the model should only focus on a very small num-ber of text spans. We thus wish the distribution

of α to be sharp. We thus propose the followingtraining objective:

L = log p(y|x) + λ∑i,j

α2(i, j) (7)

which uses∑

i,j α2(i, j) as the regularizer. Under

the constraint of∑

i,j α(i, j) = 1,∑

i,j α2(i, j)

achieves the highest value with one element of αbeing 1 and the rest being 0, and the lowest valueif all α(i, j) have the same value.2 The model canbe trained in an end-to-end fashion.

4 Evaluation

We would like an automatic evaluation method thatis both flexible in evaluating the quality of modelexplanations under any task and dataset, and able tooffer the ability of accurately reflecting the faithful-ness of model explanations, i.e., the extracted ratio-nales ought to semantically influence the model’sprediction for the same instance, as suggested byDeYoung et al. (2020).

Intuitively, if extracted rationales can faithfullyrepresent, in other words, be equivalent to, theircorresponding text inputs with respect to modelpredictions, the following should hold: (1) a modeltrained on the original inputs should perform com-parably well when tested on the extracted ratio-nales; (2) a model trained on the extracted ratio-nales should perform comparably well when testedon the original inputs; (3) a model trained on the ex-tracted rationales should perform comparably wellwhen tested on other extracted rationales. Higherperformances on these three aspects indicate morefaithful extracted rationales, and consequently bet-ter interpretation models. These strategies are in-spired by Lei et al. (2016), who trained a rationalegenerator to extract text pieces as explanations fordifferent sentiment aspects for the task of senti-ment analysis, achieving high precision based onsentence-level aspect annotations.

More formally, we use full to refer to the situationof training or testing a model on the original texts,and span to refer to the situation of training ortesting a model on the extracted rationales. By de-noting the original full dataset by Dtrain, Ddev and

2Another regularization that can be used to fulfill thesame purpose of obtaining sharpening distributions is the en-tropy −

∑i,j αi,j logαi,j which can be viewed as the KL-

divergence between the distribution and the uniform distri-bution. Empirically, we find that the two strategies performalmost the same.

Dtest, and the newly constructed rationale-baseddataset by D′train, D′dev and D′test, the settings de-scribed above are as follows:

• FullTrain-SpanTest: The model is trained onDtrain and tested on D′test, with hyperparame-ters selected on Ddev.

• SpanTrain-FullTest: The model is trained onD′train and tested on Dtest, with hyperparame-ters selected on D′dev.

• SpanTrain-SpanTest: The model is trainedon D′train and tested on D′test, with hyperpa-rameters selected on D′dev.

To construct the new rationale-based dataset, wetrain a rationale-extraction model on Dtrain to ex-tract rationales from the original text inDtrain,Ddevand Dtest, and then replace the original full inputtext with the corresponding extracted rationales,with the labels remaining unchanged.

It is also worthing that the proposed three evalua-tion metrics are far from perfect: the system canbe gamed if the extracted plan is just the same asthe original span. The proposed evaluations thusneed to be combined with other evaluations forcomplement.

5 Experiments

5.1 Tasks and DatasetsWe conduct experiments on three NLP tasks: textclassification on the SST-5 dataset (Socher et al.,2013), natural language inference on the SNLIdataset (Bowman et al., 2015) and machine trans-lation on the IWSLT2014 En→De dataset. Pleaserefer to the appendix section for descriptions of thethree datasets and training details.

For text classification and natural language infer-ence, we use RoBERT (Liu et al., 2019b) as themodel backbone. For reference purposes, we alsotrain a self-explaining model with α set to be uni-form, i.e., α(i, j) = 1/M , where M is the totalnumber of text spans. This model is denoted byAvgSelf-Explaining. For neural machine transla-tion, we use the Transformer-base (Vaswani et al.,2017) model for evaluation.

5.2 Main ResultsTable 1 shows the results for the SST-5 and SNLIdatasets, and Table 2 shows the results for IWSLT2014 En→De. We can see from the tables that

Model Accuracy↑SST-5

BCN+SuBiLSTM+CoVe (Brahma, 2018) 56.2BERT-base (Cheang et al., 2020) 54.9BERT-large (Cheang et al., 2020) 56.2SentiBERT (Yin et al., 2020)[ 56.9SentiLARE (Ke et al., 2020)[† 58.6RoBERTa-base+AvgSelf-Explaining 56.2RoBERTa-base (Liu et al., 2019b) 56.4RoBERTa-base+Self-Explaining 57.8 (+1.4)RoBERTa-large (Liu et al., 2019b) 57.9RoBERTa-large+Self-Explaining 59.1 (+1.2)

SNLI

BERT-base (Zhang et al., 2019) 89.2BERT-large (Zhang et al., 2019) 90.4SJRC (Zhang et al., 2019)† 91.3NILE (Kumar and Talukdar, 2020)[ 91.5MT-DNN (Liu et al., 2019a)† 91.6SemBERT (Zhang et al., 2020b)† 91.9CA-MTL (Pilault et al., 2020)\† 92.1RoBERTa-base+AvgSelf-Explaining 90.8RoBERTa-base (Liu et al., 2019b) 90.7RoBERTa-base+Self-Explaining 91.7 (+1.0)RoBERTa-large (Liu et al., 2019b) 91.4RoBERTa-large+Self-Explaining 92.3 (+0.9)

Table 1: Performances of different models on the SST-5and SNLI datasets. [ denotes using RoBERTa-base asthe model backbone, \ denotes using RoBERTa-largeas the model backbone, and † denotes using externaltraining resources.

Model BLEU↑IWSLT 2014 En→De

Transformer-base (Vaswani et al., 2017) 28.4Transformer-base+Self-Explaining 28.9

Table 2: Performances of different models on theIWSLT 2014 En→De dataset.

the proposed Self-Explaining method significantlyboosts performances over strong RoBERTa base-lines on SST-5 and SNLI. Using RoBerta-large, weachieve a new SOTA performance of 59.1 on SST-5 and a new SOTA performance of 92.3 on SNLI.A surprising observation is that in spite of usingthe SIC layer and the interpretation layer, AvgSelf-Explaining still underperforms RoBERTa, whichindicates that the learned attention weights α areimportant for model performances.

5.3 Interpretability Evaluation

We compare our proposed Self-Explaining modelwith the following three widely used interpretationmodels:

• AvgGradient: The method of averaging wordsaliency scores within a text span (Li et al.,2015; Feng et al., 2018). The saliency scoreof a word w is computed as the derivativeof the probability of the ouput label withrespect to the word embedding: sy(w) =‖∇wp(y|x)‖1, where p(y|x) is the predictedprobability of the ground-truth label y, w isthe corresponding word embedding of wordw and ‖ · ‖1 is the L1 norm. We take the av-erage of saliency scores of all words withina text span as the span-level saliency score.We select the span with the highest span-levelsaliency score as the rationale.

• AvgAttention: The method of averaging at-tention scores within a text span (Vig and Be-linkov, 2019; Tenney et al., 2019; Clark et al.,2019). The attention score for each word wis the normalized attentive probability of thespecial token [CLS] with respect to word win the last intermediate layer. We take the av-erage of attention scores of all words withina text span as the span-level attention score.The span with the highest span-level attentionscore is selected as the rationale.

• Rationale: The rationale extraction modelproposed by Lei et al. (2016). One encoder isused to encode input texts into representativefeatures, and one generator is used to extracttext spans as rationales. These two models arejointly trained to minimize the expected costfunction using the REINFORCE algorithm(Williams, 1992).

All models use RoBERTa (Liu et al., 2019b) as thebackbone and are trained separately for compari-son.

Results are shown in Table 4. For all the three se-tups FullTrain-SpanTest, SpanTrain-FullTest andSpanTrain-SpanTest, we can observe that the pro-posed Self-Explaining outperforms other interpreta-tion methods by a large margin on both SST-5 andSNLI. On SST, Self-Explaining outperforms Ra-tionale by +3.2, +5.0 and +6.7 respectively for theF-S, S-F and S-S setup. On SNLI, Self-Explainingoutperforms Rationale by +3.5, +10.0 and 7.3 re-spectively for the F-S, S-F and S-S setup. Com-paring AvgGradient, AvgAttention and Rationale,we can see that there is no method that shows con-sistent superiority to the other two. For example,

Model IOU F1 Token F1Movie Review

AvgGradient 0.075 0.175AvgAttention 0.067 0.142Rationale (Lei et al., 2016) 0.132 0.281Self-Explaining 0.152 0.314

SNLI

AvgGradient 0.251 0.349AvgAttention 0.301 0.435Rationale (Lei et al., 2016) 0.381 0.459Self-Explaining 0.454 0.571

Table 3: Performances of different models on MovieReview and SNLI of the ERASER benchmark.

Model F-S S-F S-SSST-5

AvgGradient 34.1 45.6 36.9AvgAttention 32.5 40.6 35.8Rationale (Lei et al., 2016) 35.2 49.9 36.2Self-Explaining 38.4 54.9 42.9

SNLI

AvgGradient 70.7 74.5 73.1AvgAttention 64.5 72.5 70.4Rationale (Lei et al., 2016) 71.0 78.5 71.9Self-Explaining 74.5 88.5 79.2

Table 4: Performances of different models on the threeevaluation methods defined in Section 4. “F” refers toFull and “S” refers to Span. Accuracy is reported ineach cell.

Rationale outperforms AvgGradient and AvgAtten-tion under the F-S and S-F setup, but underper-forms AvgGradient under the S-S setup. Instead,Self-Explaining consistently achieves significantlybetter results compared to all baselines under allsetups. These results demonstrate the better inter-pretability of the proposed Self-Explaining method.

5.4 The Effect of λ

We would like to explore the effect of λ in theadditional regularization term in Eq.7. A larger λhas a stronger effect on sharpening the distributionof α, leading to the preference to a small fractionof text spans. Intuitively, a reasonable value of λ iscrucial for model performances: too small λ meansthe selected text spans are not confident enough tosupport model predictions, while too large λmeansthe model only attends to a single text span for itsprediction. The former may cause over-numberedselected text spans, which incurs more noise formodel predictions. The latter may force the modelthe attend to a single span that is not importantfor predictions. Therefore, a sensible λ should

0 1 2 3 4 5λ

56.4

56.6

56.8

57.0

57.2

57.4

57.6

57.8

Accu

racy

SST-5

0.0 0.5 1.0 1.5 2.0λ

90.4

90.6

90.8

91.0

91.2

91.4

91.6

91.8SNLI

SST-5 SST-5 Baseline SNLI SNLI Baseline

Figure 2: Performances of the proposed model with re-spect to different values of λ. Accuracy is reported forcomparison.

be neither too small nor too large, balancing thenumber of attended text spans.

Figure 2 shows the results. As we can see from thefigure, λ = 1.5 leads to the best result on SST-5and λ = 1.0 gives the best result on SNLI, sig-nificantly outperforming models with small λ andlarge λ, as well as the baseline. As the value of λkeeps increasing, the performance drastically drops.When λ = 5.0 for SST-5 and λ = 2.0 for SNLI,the accuracies are respectively 56.4 and 90.3, evenunderperforming baselines without self-explainingstructures. To our surprise, though the model per-formance with λ = 0 is worse than the best modelwith λ = 1.5 and λ = 1.0, it still achieves betterresults compared to baselines (57.0 vs. 56.4 onSST-5 and 91.4 vs. 90.7 on SNLI). This observa-tion verifies that the proposed method can actuallyimprove NLP models in terms of both performanceand interpretability.

6 Analysis

6.1 Examples

Table 5 lists several examples to illustrate how dif-ferent methods extract text spans to interpret modelpredictions. Extracted spans are in bold.

As can be seen from the table, all methods includ-ing Self-Explaining are able to extract correspond-ing text spans as evidential explanation for themodel prediction. For example, both AvgAtten-tion and the proposed Self-Explaining are able toextract “overproduced and generally disappointing”as rationale for predicting the label Very Negative,and AvgGradient selects the key term “disappoint-

Label Model Text

Very Negative(1) this overproduced and generally disappointing effort isn’t likely to rouse the rush hour crowd(2) this overproduced and generally disappointing effort isn’t likely to rouse the rush hour crowd(3) this overproduced and generally disappointing effort isn’t likely to rouse the rush hour crowd

Negative(1) However, it lacks grandeur and that epic quality often associated with Stevenson’s tale as

well as with earlier Disney efforts(2) However, it lacks grandeur and that epic quality often associated with Stevenson’s tale as well

as with earlier Disney efforts(3) However, it lacks grandeur and that epic quality often associated with Stevenson’s tale] as well

as with earlier Disney efforts

Negative(1) Though everything might be literate and smart, it never took off and always seemed static(2) Though everything might be literate and smart, it never took off and always seemed static(3) Though everything might be literate and smart, it never took off and always seemed static

Very Positive(1) One of the greatest family-oriented, fantasy-adventure movies ever(2) One of the greatest family-oriented, fantasy-adventure movies ever(3) One of the greatest family-oriented, fantasy-adventure movies ever

Table 5: Examples of correctly classified texts and the corresponding extracted text spans by different models.(1): Self-Explaining; (2): AvgGradient; (3): AvgAttention.

True Predicted Text

Negative Positive the story is naturally poignant, but first - time screen-writer paul pender overloads it withsugary bits ofbusiness

Negative Neutral a well acted and well intentioned snoozerNegative Positive george, hire a real director and good writers forthe next installment, pleasePositive Negative It ’s like a poemPositive Very Negative There isn’t a weak or careless performance amongst them

Table 6: Examples of mis-classified texts and the corresponding extracted text spans.

Label Text

Positive↪→Negative

In the film, Lumumba, we see the faces behind the monumental shift in the Congo’s history after it isreclaimed from the Belgians, and we see the motives behind those men into whose hands the raped andstarving country fell. Lumumba is not a movie for the hyper masses; it demands the attention of its viewerswith raw, truthful acting and intricate, packed dialogue. Little of the main plot is shown through action,it relies almost solely on words, but there is a recurring strand that is only action, and it is the stroke ofgenius that makes the film an enlightening and powerful panorama of the tense political struggle that theCongo’s independence gave birth to.This film is real. It is raw inits depiction of those in power, and those onthe streets. It is eye-opening in its content. And it is moving in the passions and emotions of its superblyportrayed characters. Whether you are a history fan, a film buff, or simply like good stories, Lumumbais a must-see [Whether you are a history fan, a cinephile or just love good stories, Lumumba is amust-have]

.

Negative↪→Positive

I’m a huge Zack Allan fan and was disappointed that he only got one scene in the movie [I’m a hugefan of Zack Allan and I was disappointed that he only has one scene in the film]. This was also myfavourite scene where he confiscates a character’s weapons and directs her to Down Below. Unfortunatelyunlike Thirdspace & River of Souls, most of the action took place off station. I didn’t care much for Garibaldiafter the first three seasons and think Sheridan is okay but no Sinclair. I like Lochley but she only had limitedscreen time. If you like Crusade or space battles you should enjoy it.

Table 7: Examples of generated adversarial text spans on the SST-5 dataset. The extracted and back-translated textspans are in bold. The text in parentheses [·] is the corresponding paraphrase to flip the prediction.

ing”, which also makes sense in this case. How-ever, compared to AvgGradient and AvgAttention,Self-Explaining can select more global and compre-hensive text spans. For example, Self-Explainingextracts “it lacks grandeur and that epic qualityoften associated with Stevenson’s tale” for predict-ing the Negative label, while AvgAttention andAvgGradient only extract “lacks grandeur”, which

may not be comprehensive for making the decision.Self-Explaining is able to extract “Though every-thing might be literate and smart” as rationale forpredicting Negative, while AvgAttention and Avg-Gradient only extract local text spans “smart” and“literate and smart”, respectively. Although they allmake correct predictions, the rationales providedby AvgAttention and AvgGradient can not explain

Model IMBD Yahoo!Answers

Original 84.86 92.00Random 37.79 74.50Gradient 14.57 73.80TiWo 3.57 62.50WS 3.93 62.50PWWS 2.00 53.00Self-Explaining+Paraphrase 0.86 43.14

Table 8: Classification accuracy of each model onthe original datasets (the first row) and the perturbeddatasets using different adversarial methods. A lowerclassification accuracy corresponds to a more effectiveattacking method.

their behaviors of making the decision. By contrast,Self-Explaining successfully captures the conjunc-tion “though”, a word that reverses the sentimentfrom positive to negative.

6.2 Error Analysis

By examining extracted text spans from erro-neously classified examples, the model providesa direct way for performing error analysis. Thebiggest advantage of the proposed model over pre-vious interpretability methods is that, instead offocusing merely on word-level features as in Liet al. (2015, 2016), the proposed model operatesat arbitrary levels of text units, providing moredirect and accurate views of why models makemistakes. By examining erroneously classified ex-amples shown in Table 6, we can clearly identify afew patterns that make neural models fail: (1) themodel emphasizes the part of sentences that shouldnot be attended to in the contrast conjunction, e.g.,“ [the story is naturally poignant], but first - timescreenwriter paul pender overloads it with sugarybits of business”; (2) the model cannot recognize aword used in a context that changes its sentiment,e.g., “ [a well acted and well intention] ed snoozer”and “There [isn’t a weak or careless performance]amongst them”; (3) the model cannot recognizeirony: “george, [hire a real director] and good writ-ers for the next installment, please”; (4) the modelcannot recognize analogy: “It’s [like a poem]”, etc.

6.3 Span-based Adversarial ExampleGeneration

There has been a growing interest in generatingadversarial examples to attack a neural networkmodel in NLP (Alzantot et al., 2018; Ren et al.,2019; Zhang et al., 2020a). The current protocol foradversarial example generation is based on word

substitution (Liang et al., 2017; Ebrahimi et al.,2017; Samanta and Mehta, 2017; Ren et al., 2019),where an important word is selected by saliencyand then replaced by its synonym if the replace-ment can flip the prediction. The shortcoming forcurrent approaches is that it can only operate at theword level and perform word-level substitutions.This is rooted in the fact that saliency scores canonly be reliably computed at the word level.

The proposed span-based model naturally ad-dresses this issue: we first identify the most salientspan in the input based on α, replace it with its para-phrase if the replacement flips the label. To the bestof our knowledge, this is the first feasible approachthat operates at higher levels beyond words for ad-versarial example generation in NLP. Paraphrasesare generated by back-translation (Sennrich et al.,2016; Edunov et al., 2018).3

Following Ren et al. (2019), we use two datasetsfor test, IMDB (Maas et al., 2011) and Yahoo!Answers. The details of the two datasets are de-scribed in Appendix. For fair comparisons, wefollow Ren et al. (2019) to use the Bi-LSTMs asthe model backbone and compare our proposedSelf-Explaining+Paraphrase model with the follow-ing attacking methods: (1) Random; (2) Gradient;(3) Traversing in Word Order (TiWO); (4) WordSaliency (WS); (5) Probability Weighted WordSaliency (PWWS). We refer readers to Ren et al.(2019) for more details of these methods.

Table 8 shows the classification accuracies ofdifferent methods on the original datasets andthe adversarial samples generated by these at-tacking methods. Results show that our pro-posed Self-Explaining+Paraphrase method reducesthe classification accuracy to the most extent.Compared to the original performance, Self-Explaining+Paraphrase reduces the classificationaccuracies on IMDB and Yahoo! Answers by 84%and 48.86%, respectively, indicating the effective-ness of the proposed Self-Explaining+Paraphrasemethod for generating adversarial samples.

Table 7 gives two examples of flipping model pre-dictions by using the Self-Explaining+Paraphrasemethod to generate adversarial text spans. Throughthe examples, we can see that the selected textspan is crucial for the model prediction and a slight

3We use the off-the-shelf google translator to implementEn→De→En back-translation.

perturbation injected to this span would flip theprediction.

7 Conclusion

In this work, we present a light but effective self-explainable structure to improve both performanceand interpretability in the context of NLP. The ideaof the proposed method is to introduce an interpre-tation layer, aggregating information for each textspan, which is then assigned a specific weight rep-resenting its contribution to interpreting the modelprediction. Extensive experiments show the ef-fectiveness of the proposed method in terms ofimproving performances for the task of sentimentclassification and natural language inference, aswell as endowing the model with the ability of self-explaining, avoiding heavy recourse to additionalexternal interpretation models such as probing mod-els and surrogate models.

References

Philip Adler, Casey Falk, Sorelle A Friedler, TionneyNix, Gabriel Rybeck, Carlos Scheidegger, BrandonSmith, and Suresh Venkatasubramanian. 2018. Audit-ing black-box models for indirect influence. Knowl-edge and Information Systems, 54(1):95–122.

Moustafa Alzantot, Yash Sharma, Ahmed Elgohary,Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang.2018. Generating natural language adversarial exam-ples. arXiv preprint arXiv:1804.07998.

Leila Arras, Franziska Horn, Grégoire Montavon,Klaus-Robert Müller, and Wojciech Samek. 2016. Ex-plaining predictions of non-linear classifiers in nlp.arXiv preprint arXiv:1606.07298.

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez,Javier Del Ser, Adrien Bennetot, Siham Tabik, AlbertoBarbado, Salvador García, Sergio Gil-López, DanielMolina, Richard Benjamins, et al. 2020. Explainableartificial intelligence (xai): Concepts, taxonomies, op-portunities and challenges toward responsible ai. Infor-mation Fusion, 58:82–115.

Sebastian Bach, Alexander Binder, Grégoire Mon-tavon, Frederick Klauschen, Klaus-Robert Müller, andWojciech Samek. 2015a. On pixel-wise explanationsfor non-linear classifier decisions by layer-wise rele-vance propagation. PloS one, 10(7).

Sebastian Bach, Alexander Binder, Grégoire Mon-tavon, Frederick Klauschen, Klaus-Robert Müller, andWojciech Samek. 2015b. On pixel-wise explanationsfor non-linear classifier decisions by layer-wise rele-vance propagation. PLOS ONE, 10:1–46.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-

gio. 2014. Neural machine translation by jointly learn-ing to align and translate.

Samuel R Bowman, Gabor Angeli, Christopher Potts,and Christopher D Manning. 2015. A large annotatedcorpus for learning natural language inference. arXivpreprint arXiv:1508.05326.

Siddhartha Brahma. 2018. Improved sentence mod-eling using suffix bidirectional lstm. arXiv preprintarXiv:1805.07340.

Duo Chai, Wei Wu, Qinghong Han, Fei Wu, and JiweiLi. 2020. Description based text classification with re-inforcement learning.

Shiyu Chang, Yang Zhang, Mo Yu, and TommiJaakkola. 2019. A game theoretic approach to class-wise selective rationalization. In Advances in NeuralInformation Processing Systems, pages 10055–10065.

Brian Cheang, Bailey Wei, David Kogan, Howey Qiu,and Masud Ahmed. 2020. Language representationmodels for fine-grained sentiment classification. arXivpreprint arXiv:2005.13619.

Sihao Chen, Daniel Khashabi, Wenpeng Yin, ChrisCallison-Burch, and Dan Roth. 2019. Seeing thingsfrom a different angle:discovering diverse perspectivesabout claims. In Proceedings of the 2019 Confer-ence of the North American Chapter of the Associa-tion for Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers), Min-neapolis, Minnesota. Association for ComputationalLinguistics.

Kevin Clark, Urvashi Khandelwal, Omer Levy, andChristopher D. Manning. 2019. What does bert lookat? an analysis of bert’s attention.

Anupam Datta, Shayak Sen, and Yair Zick. 2016. Al-gorithmic transparency via quantitative input influence:Theory and experiments with learning systems. In2016 IEEE symposium on security and privacy (SP),pages 598–617. IEEE.

Misha Denil, Alban Demiraj, and Nando de Freitas.2015. Extraction of salient sentences from labelleddocuments.

Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani,Eric Lehman, Caiming Xiong, Richard Socher, and By-ron C. Wallace. 2020. ERASER: A benchmark to eval-uate rationalized NLP models. In Proceedings of the58th Annual Meeting of the Association for Computa-tional Linguistics, Online. Association for Computa-tional Linguistics.

Javid Ebrahimi, Anyi Rao, Daniel Lowd, and De-jing Dou. 2017. Hotflip: White-box adversarialexamples for text classification. arXiv preprintarXiv:1712.06751.

Sergey Edunov, Myle Ott, Michael Auli, and DavidGrangier. 2018. Understanding back-translation atscale. arXiv preprint arXiv:1808.09381.

Shi Feng, Eric Wallace, Alvin Grissom II, MohitIyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018.Pathologies of neural models make interpretations dif-ficult. arXiv preprint arXiv:1804.07781.

Reza Ghaeini, Xiaoli Z Fern, and Prasad Tadepalli.2018. Interpreting recurrent and attention-based neu-ral models: a case study on natural language inference.arXiv preprint arXiv:1808.03894.

Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas RSteunebrink, and Jürgen Schmidhuber. 2016. Lstm: Asearch space odyssey. IEEE transactions on neural net-works and learning systems, 28(10):2222–2232.

Phu Mon Htut, Jason Phang, Shikha Bordia, andSamuel R. Bowman. 2019. Do attention heads in berttrack syntactic dependencies?

Sarthak Jain and Byron C. Wallace. 2019. Attention isnot explanation.

Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and By-ron C. Wallace. 2020. Learning to faithfully rational-ize by construction. In Proceedings of the 58th AnnualMeeting of the Association for Computational Linguis-tics, Online. Association for Computational Linguis-tics.

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld,Luke Zettlemoyer, and Omer Levy. 2020. Spanbert:Improving pre-training by representing and predictingspans. Transactions of the Association for Computa-tional Linguistics, 8:64–77.

Andrej Karpathy, Justin Johnson, and Li Fei-Fei.2015. Visualizing and understanding recurrent net-works. arXiv preprint arXiv:1506.02078.

Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Min-lie Huang. 2020. SentiLARE: Sentiment-aware lan-guage representation learning with linguistic knowl-edge. In Proceedings of the 2020 Conference onEmpirical Methods in Natural Language Processing(EMNLP), pages 6975–6988, Online. Association forComputational Linguistics.

Pieter-Jan Kindermans, Kristof T. Schütt, Maximil-ian Alber, Klaus-Robert Müller, Dumitru Erhan, BeenKim, and Sven Dähne. 2017. Learning how to explainneural networks: Patternnet and patternattribution.

Diederik P Kingma and Jimmy Ba. 2014. Adam: Amethod for stochastic optimization. arXiv preprintarXiv:1412.6980.

Pang Wei Koh and Percy Liang. 2017. Understandingblack-box predictions via influence functions. In Pro-ceedings of the 34th International Conference on Ma-chine Learning-Volume 70, pages 1885–1894. JMLR.org.

Sawan Kumar and Partha Talukdar. 2020. Nile: Nat-ural language inference with faithful natural languageexplanations. arXiv preprint arXiv:2005.12116.

Eric Lehman, Jay DeYoung, Regina Barzilay, and By-ron C. Wallace. 2019. Inferring which medical treat-

ments work from reports of clinical trials. In Proceed-ings of the 2019 Conference of the North AmericanChapter of the Association for Computational Linguis-tics: Human Language Technologies, Volume 1 (Longand Short Papers), Minneapolis, Minnesota. Associa-tion for Computational Linguistics.

Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016.Rationalizing neural predictions. arXiv preprintarXiv:1606.04155.

Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky.2015. Visualizing and understanding neural models innlp. arXiv preprint arXiv:1506.01066.

Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Under-standing neural networks through representation era-sure. arXiv preprint arXiv:1612.08220.

Xiaoya Li, Jingrong Feng, Yuxian Meng, QinghongHan, Fei Wu, and Jiwei Li. 2019. A unified mrc frame-work for named entity recognition. arXiv preprintarXiv:1910.11476.

Bin Liang, Hongcheng Li, Miaoqiang Su, PanBian, Xirong Li, and Wenchang Shi. 2017. Deeptext classification can be fooled. arXiv preprintarXiv:1704.08006.

Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jian-feng Gao. 2019a. Multi-task deep neural networks fornatural language understanding. In Proceedings of the57th Annual Meeting of the Association for Computa-tional Linguistics, pages 4487–4496, Florence, Italy.Association for Computational Linguistics.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, LukeZettlemoyer, and Veselin Stoyanov. 2019b. Roberta:A robustly optimized bert pretraining approach. arXivpreprint arXiv:1907.11692.

Andrew Maas, Raymond E Daly, Peter T Pham, DanHuang, Andrew Y Ng, and Christopher Potts. 2011.Learning word vectors for sentiment analysis. In Pro-ceedings of the 49th annual meeting of the associationfor computational linguistics: Human language tech-nologies, pages 142–150.

David Alvarez Melis and Tommi Jaakkola. 2018. To-wards robust interpretability with self-explaining neu-ral networks. In Advances in Neural Information Pro-cessing Systems, pages 7775–7784.

Yuxian Meng, Chun Fan, Zijun Sun, Eduard Hovy, FeiWu, and Jiwei Li. 2020. Pair the dots: Jointly exam-ining training history and test stimuli for model inter-pretability. arXiv preprint arXiv:2010.06943.

Grégoire Montavon, Sebastian Lapuschkin, AlexanderBinder, Wojciech Samek, and Klaus-Robert Müller.2017. Explaining nonlinear classification decisionswith deep taylor decomposition. Pattern Recognition,65:211–222.

Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, RuiYan, and Zhi Jin. 2015. Natural language inference by

tree-based convolution and heuristic matching. arXivpreprint arXiv:1512.08422.

W James Murdoch, Peter J Liu, and Bin Yu. 2018.Beyond word importance: Contextual decompositionto extract interactions from lstms. arXiv preprintarXiv:1801.05453.

Jonathan Pilault, Amine Elhattami, and ChristopherPal. 2020. Conditionally adaptive multi-task learning:Improving transfer learning in nlp using fewer parame-ters & less data.

Nazneen Fatema Rajani, Bryan McCann, CaimingXiong, and Richard Socher. 2019. Explain yourself!leveraging language models for commonsense reason-ing. arXiv preprint arXiv:1906.02361.

Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che.2019. Generating natural language adversarial exam-ples through probability weighted word saliency. InProceedings of the 57th Annual Meeting of the Associ-ation for Computational Linguistics, pages 1085–1097,Florence, Italy. Association for Computational Linguis-tics.

Marco Tulio Ribeiro, Sameer Singh, and CarlosGuestrin. 2016. "why should i trust you?": Explainingthe predictions of any classifier.

Anna Rogers, Olga Kovaleva, and Anna Rumshisky.2020. A primer in bertology: What we know abouthow bert works.

Suranjana Samanta and Sameep Mehta. 2017. To-wards crafting text adversarial samples. arXiv preprintarXiv:1707.02812.

Patrick Schwab and Walter Karlen. 2019. Cxplain:Causal explanations for model interpretation under un-certainty. In Advances in Neural Information Process-ing Systems, pages 10220–10230.

Ramprasaath R Selvaraju, Michael Cogswell, Ab-hishek Das, Ramakrishna Vedantam, Devi Parikh, andDhruv Batra. 2017. Grad-cam: Visual explanationsfrom deep networks via gradient-based localization. InProceedings of the IEEE international conference oncomputer vision, pages 618–626.

Rico Sennrich, Barry Haddow, and Alexandra Birch.2016. Improving neural machine translation modelswith monolingual data. In Proceedings of the 54th An-nual Meeting of the Association for Computational Lin-guistics (Volume 1: Long Papers), pages 86–96, Berlin,Germany. Association for Computational Linguistics.

Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, andHannaneh Hajishirzi. 2016. Bidirectional attentionflow for machine comprehension. arXiv preprintarXiv:1611.01603.

Sofia Serrano and Noah A Smith. 2019. Is attentioninterpretable? arXiv preprint arXiv:1906.03731.

Xing Shi, Kevin Knight, and Deniz Yuret. 2016. Whyneural translations are the right length. In Proceedings

of the 2016 Conference on Empirical Methods in Nat-ural Language Processing, pages 2278–2282, Austin,Texas. Association for Computational Linguistics.

Avanti Shrikumar, Peyton Greenside, and Anshul Kun-daje. 2017. Learning important features throughpropagating activation differences. arXiv preprintarXiv:1704.02685.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisser-man. 2013. Deep inside convolutional networks: Visu-alising image classification models and saliency maps.arXiv preprint arXiv:1312.6034.

Richard Socher, Alex Perelygin, Jean Wu, JasonChuang, Christopher D Manning, Andrew Y Ng, andChristopher Potts. 2013. Recursive deep models for se-mantic compositionality over a sentiment treebank. InProceedings of the 2013 conference on empirical meth-ods in natural language processing, pages 1631–1642.

Jost Tobias Springenberg, Alexey Dosovitskiy, ThomasBrox, and Martin Riedmiller. 2014. Striving for sim-plicity: The all convolutional net. arXiv preprintarXiv:1412.6806.

Suraj Srinivas and François Fleuret. 2019. Full-gradient representation for neural network visualiza-tion. In Advances in Neural Information ProcessingSystems, pages 4124–4133.

Hendrik Strobelt, Sebastian Gehrmann, Bernd Huber,Hanspeter Pfister, Alexander M Rush, et al. 2016. Vi-sual analysis of hidden state dynamics in recurrent neu-ral networks. arXiv preprint arXiv:1606.07461.

Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.Bert rediscovers the classical nlp pipeline. arXivpreprint arXiv:1905.05950.

Shikhar Vashishth, Shyam Upadhyay, Gaurav SinghTomar, and Manaal Faruqui. 2019. Attention in-terpretability across nlp tasks. arXiv preprintarXiv:1909.11218.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. Attention is all youneed. In Advances in neural information processingsystems, pages 5998–6008.

Jesse Vig and Yonatan Belinkov. 2019. Analyzing thestructure of attention in a transformer language model.arXiv preprint arXiv:1906.04284.

Sarah Wiegreffe and Yuval Pinter. 2019. Attention isnot not explanation. In Proceedings of the 2019 Con-ference on Empirical Methods in Natural LanguageProcessing and the 9th International Joint Conferenceon Natural Language Processing (EMNLP-IJCNLP),pages 11–20, Hong Kong, China. Association for Com-putational Linguistics.

Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcementlearning. Machine Learning, 8:229–256.

Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and JiweiLi. 2019. Coreference resolution as query-based spanprediction. arXiv preprint arXiv:1911.01746.

Da Yin, Tao Meng, and Kai-Wei Chang. 2020. Sentib-ert: A transferable transformer-based architecture forcompositional sentiment semantics.

Matthew D Zeiler and Rob Fergus. 2014. Visualizingand understanding convolutional networks. In Euro-pean conference on computer vision, pages 818–833.Springer.

Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi,and Chenliang Li. 2020a. Adversarial attacks on deep-learning models in natural language processing: A sur-vey. ACM Transactions on Intelligent Systems andTechnology (TIST), 11(3):1–41.

Zhuosheng Zhang, Yuwei Wu, Zuchao Li, and HaiZhao. 2019. Explicit contextual semantics for text com-prehension. In Proceedings of the 33rd Pacific AsiaConference on Language, Information and Computa-tion (PACLIC 33).

Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li,Shuailiang Zhang, Xi Zhou, and Xiang Zhou. 2020b.Semantics-aware bert for language understanding. InProceedings of the AAAI Conference on Artificial Intel-ligence, volume 34, pages 9628–9635.

A Datasets and Training Details

The Stanford Sentiment Treebank (SST) (Socheret al., 2013) is a widely used benchmark for textclassification. The task is to perform both fine-grained (very positive, positive, neutral, negativeand very negative) and coarse-grained (positive andnegative) sentiment classification at both the phraseand sentence level. We adopt the fine-grained setupin this work. We use Adam (Kingma and Ba,2014) to optimize all modesl, with β1 = 0.9, β2 =0.98, ε = 10−8. All hyperparameters includingbatch size, dropout and learning rate are tuned onthe validation set.

The Stanford Natural Language Inference (SNLI)Corpus (Bowman et al., 2015) is a collection of570k human-written English sentence pairs for thetask of natural language inference (NLI). It con-tains a balanced amount of sentence pairs with la-bels entailment, contradiction, and neutral. We fol-low the official train/dev/test splits for both datasets.We also use Adam (Kingma and Ba, 2014) for op-timization. Hyperparameters including batch size,dropout and learning rate are tuned on the valida-tion set

IWSLT2014 En→De contains 160k training se-quences pairs and 7k validation sentence pairs. We

use the concatenation of dev2010, tst2010, tst2011and tst2011 as the test set. A joint BPE vocabularyof about 10k tokens is constructed. We minimizethe cross entropy loss with label smoothing of value0.1 during training. Adam (Kingma and Ba, 2014)is used for optimization.

IMDB contains an even number of positive andnegative reviews. The training and test sets re-spectively contain 25k and 25k examples. Yahoo!Answers is a large dataset for topic classificationover ten largest main categories from Yahoo! An-swers Comprehensive Questions and Answers v1.0.This dataset contains 1,400k training exmaples and60k test examples respectively.