Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601Florence, Italy, July 28 - August 2, 2019. c©2019 Association for Computational Linguistics

4593

BERT Rediscovers the Classical NLP Pipeline

Ian Tenney1 Dipanjan Das1 Ellie Pavlick1,2

1Google Research 2Brown University{iftenney,dipanjand,epavlick}@google.com

Abstract

Pre-trained text encoders have rapidly ad-vanced the state of the art on many NLPtasks. We focus on one such model, BERT,and aim to quantify where linguistic informa-tion is captured within the network. We findthat the model represents the steps of the tra-ditional NLP pipeline in an interpretable andlocalizable way, and that the regions respon-sible for each step appear in the expected se-quence: POS tagging, parsing, NER, semanticroles, then coreference. Qualitative analysisreveals that the model can and often does ad-just this pipeline dynamically, revising lower-level decisions on the basis of disambiguatinginformation from higher-level representations.

1 Introduction

Pre-trained sentence encoders such as ELMo (Pe-ters et al., 2018a) and BERT (Devlin et al., 2019)have rapidly improved the state of the art on manyNLP tasks, and seem poised to displace both staticword embeddings (Mikolov et al., 2013) and dis-crete pipelines (Manning et al., 2014) as the basisfor natural language processing systems. Whilethis has been a boon for performance, it has comeat the cost of interpretability, and it remains un-clear whether such models are in fact learning thekind of abstractions that we intuitively believe areimportant for representing natural language, or aresimply modeling complex co-occurrence statis-tics.

A wave of recent work has begun to “probe”state-of-the-art models to understand whether theyare representing language in a satisfying way.Much of this work is behavior-based, designingcontrolled test sets and analyzing errors in orderto reverse-engineer the types of abstractions themodel may or may not be representing (e.g. Con-neau et al., 2018; Marvin and Linzen, 2018; Poliaket al., 2018). Parallel efforts inspect the structure

of the network directly, to assess whether thereexist localizable regions associated with distincttypes of linguistic decisions. Such work has pro-duced evidence that deep language models can en-code a range of syntactic and semantic informa-tion (e.g. Shi et al., 2016; Belinkov, 2018; Ten-ney et al., 2019), and that more complex structuresare represented hierarchically in the higher layersof the model (Peters et al., 2018b; Blevins et al.,2018).

We build on this latter line of work, focusingon the BERT model (Devlin et al., 2019), and usea suite of probing tasks (Tenney et al., 2019) de-rived from the traditional NLP pipeline to quantifywhere specific types of linguistic information areencoded. Building on observations (Peters et al.,2018b) that lower layers of a language model en-code more local syntax while higher layers capturemore complex semantics, we present two novelcontributions. First, we present an analysis thatspans the common components of a traditionalNLP pipeline. We show that the order in whichspecific abstractions are encoded reflects the tradi-tional hierarchy of these tasks. Second, we quali-tatively analyze how individual sentences are pro-cessed by the BERT network, layer-by-layer. Weshow that while the pipeline order holds in ag-gregate, the model can allow individual decisionsto depend on each other in arbitrary ways, de-ferring ambiguous decisions or revising incorrectones based on higher-level information.

2 Model

Edge Probing. Our experiments are based onthe “edge probing” approach of Tenney et al.(2019), which aims to measure how well infor-mation about linguistic structure can be extractedfrom a pre-trained encoder. Edge probing decom-poses structured-prediction tasks into a common

4594

format, where a probing classifier receives spanss1 = [i1, j1) and (optionally) s2 = [i2, j2) andmust predict a label such as a constituent or rela-tion type.1 The probing classifier has access onlyto the per-token contextual vectors within the tar-get spans, and so must rely on the encoder to pro-vide information about the relation between thesespans and their role in the sentence.

We use eight labeling tasks from the edgeprobing suite: part-of-speech (POS), constituents(Consts.), dependencies (Deps.), entities, seman-tic role labeling (SRL), coreference (Coref.), se-mantic proto-roles (SPR; Reisinger et al., 2015),and relation classification (SemEval). These tasksare derived from standard benchmark datasets, andevaluated with a common metric–micro-averagedF1–to facilitate comparison across tasks. 2

BERT. The BERT model (Devlin et al., 2019)has shown state-of-the-art performance on manytasks, and its deep Transformer architecture(Vaswani et al., 2017) is typical of many recentmodels (e.g. Radford et al., 2018, 2019; Liu et al.,2019). We focus on the stock BERT models(base and large, uncased), which are trained witha multi-task objective (masked language modelingand next-sentence prediction) over a 3.3B wordEnglish corpus. Since we want to understand howthe network represents language as a result of pre-training, we follow Tenney et al. (2019) (depart-ing from standard BERT usage) and freeze the en-coder weights. This prevents the encoder from re-arranging its internal representations to better suitthe probing task.

Given input tokens T = [t0, t1, . . . , tn],a deep encoder produces a set of layer ac-tivations H(0), H(1), . . . ,H(L), where H(`) =

[h(`)0 ,h

(`)1 , . . . ,h

(`)n ] are the activation vectors of

the `th encoder layer and H(0) corresponds to thenon-contextual word(piece) embeddings. We usea weighted sum across layers (§3.1) to pool theseinto a single set of per-token representation vec-tors H = [h0,h1, . . . ,hn], and train a probingclassifier Pτ for each task using the architectureand procedure of Tenney et al. (2019).

1For single-span tasks (POS, entities, and constituents),s2 is not used. For POS, s1 = [i, i+ 1) is a single token.

2We use the code from https://github.com/jsalt18-sentence-repl/jiant. Dependencies isthe English Web Treebank (Silveira et al., 2014), SPR is theSPR1 dataset of (Teichert et al., 2017), and relations is Se-mEval 2010 Task 8 (Hendrickx et al., 2009). All other tasksare from OntoNotes 5.0 (Weischedel et al., 2013).

Limitations This work is intended to be ex-ploratory. We focus on one particular encoder–BERT–to explore how information can be orga-nized in a deep language model, and further workis required to determine to what extent the trendshold in general. Furthermore, our work carriesthe limitations of all inspection-based probing: thefact that a linguistic pattern is not observed by ourprobing classifier does not guarantee that it is notthere, and the observation of a pattern does nottell us how it is used. For this reason, we empha-size the importance of combining structural analy-sis with behavioral studies (as discussed in § 1) toprovide a more complete picture of what informa-tion these models encode and how that informa-tion affects performance on downstream tasks.

3 Metrics

We define two complementary metrics. The first,scalar mixing weights (§3.1) tell us which lay-ers, in combination, are most relevant when aprobing classifier has access to the whole BERTmodel. The second, cumulative scoring (§3.2) tellsus how much higher we can score on a probingtask with the introduction of each layer. Thesemetrics provide complementary views on what ishappening inside the model. Mixing weights arelearned solely from the training data–they tell uswhich layers the probing model finds most useful.In contrast, cumulative scoring is derived entirelyfrom an evaluation set, and tell us how many lay-ers are needed for a correct prediction.

3.1 Scalar Mixing Weights

To pool across layers, we use the scalar mixingtechnique introduced by the ELMo model. Fol-lowing Equation (1) of Peters et al. (2018a), foreach task we introduce scalar parameters γτ anda

(0)τ , a

(1)τ , . . . , a

(L)τ , and let:

hi,τ = γτ

L∑`=0

s(`)τ h

(`)i (1)

where sτ = softmax(aτ ). We learn these weightsjointly with the probing classifier Pτ , in order toallow it to extract information from the many lay-ers of an encoder without adding a large numberof parameters. After the probing model is trained,we extract the learned coefficients in order to es-timate the contribution of different layers to thatparticular task. We interpret higher weights as ev-

4595

Figure 1: Summary statistics on BERT-large. Columnson left show F1 dev-set scores for the baseline (P (0)

τ )and full-model (P (L)

τ ) probes. Dark (blue) are the mix-ing weight center of gravity (Eq. 2); light (purple) arethe expected layer from the cumulative scores (Eq. 4).

idence that the corresponding layer contains moreinformation related to that particular task.

Center-of-Gravity. As a summary statistic, wedefine the mixing weight center of gravity as:

Es[`] =

L∑`=0

` · s(`)τ (2)

This reflects the average layer attended to for eachtask; intuitively, we can interpret a higher value tomean that the information needed for that task iscaptured by higher layers.

3.2 Cumulative ScoringWe would like to estimate at which layer in theencoder a target (s1, s2, label) can be correctlypredicted. Mixing weights cannot tell us this di-rectly, because they are learned as parameters anddo not correspond to a distribution over data. Anaive classifier at a single layer cannot either, be-cause information about a particular span may bespread out across several layers, and as observedin Peters et al. (2018b) the encoder may choose todiscard information at higher layers.

To address this, we train a series of classifiers{P (`)

τ }` which use scalar mixing (Eq. 1) to attendto layer ` as well as all previous layers. P (0)

τ corre-sponds to a non-contextual baseline that uses onlya bag of word(piece) embeddings, while P (L)

τ =Pτ corresponds to probing all layers of the BERTmodel.

These classifiers are cumulative, in the sensethat P (`+1)

τ has a similar number of parameters butwith access to strictly more information than P (`)

τ ,

Figure 2: Layer-wise metrics on BERT-large. Solid(blue) are mixing weights s(`)τ (§3.1); outlined (purple)are differential scores ∆

(`)τ (§3.2), normalized for each

task. Horizontal axis is encoder layer.

and we see intuitively that performance (F1 score)generally increases as more layers are added.3 Wecan then compute a differential score ∆

(`)τ , which

measures how much better we do on the probingtask if we observe one additional encoder layer `:

∆(`)τ = Score(P (`)

τ )− Score(P (`−1)τ ) (3)

Expected Layer. Again, we compute a(pseudo)4 expectation over the differential scoresas a summary statistic. To focus on the behaviorof the contextual encoder layers, we omit the con-tribution of both the “trivial” examples resolved atlayer 0, as well as the remaining headroom from

3Note that if a new layer provides distracting features, theprobing model can overfit and performance can drop. We seethis in particular in the last 1-2 layers (Figure 2).

4This is not a true expectation because the F1 score is notan expectation over examples.

4596

the full model. Let:

E∆[`] =

∑L`=1 ` ·∆

(`)τ∑L

`=1 ∆(`)τ

(4)

This can be thought of as, approximately, the ex-pected layer at which the probing model correctlylabels an example, assuming that example is re-solved at some layer ` ≥ 1 of the encoder.

4 Results

Figure 1 reports summary statistics and absoluteF1 scores, and Figure 2 reports per-layer metrics.Both show results on the 24-layer BERT-largemodel. We also report K(?) = KL(?||Uniform)to estimate how non-uniform5 each statistic (? =sτ ,∆τ ) is for each task.

Linguistic Patterns. We observe a consistenttrend across both of our metrics, with the tasksencoded in a natural progression: POS tags pro-cessed earliest, followed by constituents, depen-dencies, semantic roles, and coreference. That is,it appears that basic syntactic information appearsearlier in the network, while high-level semanticinformation appears at higher layers. We note thatthis finding is consistent with initial observationsby Peters et al. (2018b), which found that con-stituents are represented earlier than coreference.

In addition, we observe that in general, syntacticinformation is more localizable, with weights re-lated to syntactic tasks tending to be concentratedon a few layers (high K(s) and K(∆)), while in-formation related to semantic tasks is generallyspread across the entire network. For example,we find that for semantic relations and proto-roles(SPR), the mixing weights are close to uniform,and that nontrivial examples for these tasks are re-solved gradually across nearly all layers. For en-tity labeling many examples are resolved in layer1, but with a long tail thereafter, and only a weakconcentration of mixing weights in high layers.Further study is needed to determine whether thisis because BERT has difficulty representing thecorrect abstraction for these tasks, or because se-mantic information is inherently harder to localize.

Comparison of Metrics. For many tasks, wefind that the differential scores are highest in the

5KL(?||Uniform) = −H(?)+Constant, so higher val-ues correspond to lower entropy.

first few layers of the model (layers 1-7 for BERT-large), i.e. most examples can be correctly classi-fied very early on. We attribute this to the avail-ability of heuristic shortcuts: while challengingexamples may not be resolved until much later,many cases can be guessed from shallow statis-tics. Conversely, we observe that the learned mix-ing weights are concentrated much later, layers 9-20 for BERT-large. We observe–particularly whenweights are highly concentrated–that the highestweights are found on or just after the highest lay-ers which give an improvement ∆

(`)τ in F1 score

for that task.This helps explain the observations on the se-

mantic relations and SPR tasks: cumulative scor-ing shows continued improvement up to the high-est layers of the model, while the lack of con-centration in the mixing weights suggest that theBERT encoder does not expose a localized set offeatures that encode these more semantic phenom-ena. Similarly for entity types, we see contin-ued improvements in the higher layers – perhapsrelated to fine-grained semantic distinctions like”Organization” (ORG) vs. ”Geopolitical Entity”(GPE) – while the low value for the expected layerreflects that many examples require only limitedcontext to resolve.

Comparison of Encoders. We observe the samegeneral ordering on the 12-layer BERT-basemodel (Figure A.2). In particular, there appearsto be a “stretching effect”, where the representa-tions for a given task tend to concentrate at thesame layers relative to the top of the model; thisis illustrated side-by-side in Figure A.3.

4.1 Per-Example Analysis

We explore, qualitatively, how beliefs about thestructure of individual sentences develop over thelayers of the BERT network. The OntoNotes de-velopment set contains annotations for five of ourprobing tasks: POS, constituents, entities, SRL,and coreference. We compile the predictions ofthe per-layer classifiers P (`)

τ for each task. Be-cause many annotations are uninteresting – for ex-ample, 89% of part-of-speech tags are correct atlayer 0 – we use a heuristic to identify ambigu-ous sentences to visualize.6 Figure 3 shows two

6Specifically, we look for target edges (s1, s2, label)where the highest scoring label has an average score

1L+1

∑L`=0 P

(`)τ (label|s1, s2) ≤ 0.7, and look at sentences

with more than one such edge.

4597

(a) he smoked toronto in the playoffs with six hits, sevenwalks and eight stolen bases ...

(b) china today blacked out a cnn interview that was ...

Figure 3: Probing classifier predictions across lay-ers of BERT-base. Blue is the correct label; or-ange is the incorrect label with highest average scoreover layers. Bar heights are (normalized) probabilitiesP

(`)τ (label|s1, s2). In the interest of space, only se-

lected annotations are shown.

selected examples, and more are presented in Ap-pendix A.2.

We find that while the pipeline order holds onaverage (Figure 2), for individual examples themodel is free to and often does choose a differ-ent order. In the first example, the model origi-nally (incorrectly) assumes that “Toronto” refersto the city, tagging it as a GPE. However, af-ter resolving the semantic role – determining that“Toronto” is the thing getting “smoked” (ARG1)– the entity-typing decision is revised in favor ofORG (i.e. the sports team). In the second exam-ple, the model initially tags “today” as a commonnoun, date, and temporal modifier (ARGM-TMP).However, this phrase is ambiguous, and it laterreinterprets “china today” as a proper noun (i.e.the TV network) and updates its beliefs about theentity type (to ORG), followed by the semantic role(reinterpreting it as the agent ARG0).

5 Conclusion

We employ the edge probing task suite to explorehow the different layers of the BERT network canresolve syntactic and semantic structure within asentence. We present two complementary mea-surements: scalar mixing weights, learned from a

training corpus, and cumulative scoring, measuredon an evaluation set, and show that a consistentordering emerges. We find that while this tradi-tional pipeline order holds in the aggregate, on in-dividual examples the network can resolve out-of-order, using high-level information like predicate-argument relations to help disambiguate low-leveldecisions like part-of-speech. This provides newevidence corroborating that deep language mod-els can represent the types of syntactic and se-mantic abstractions traditionally believed neces-sary for language processing, and moreover thatthey can model complex interactions between dif-ferent levels of hierarchical information.

Acknowledgments

Thanks to Kenton Lee, Emily Pitler, and Jon Clarkfor helpful comments and feedback, and to themembers of the Google AI Language team formany productive discussions.

ReferencesYonatan Belinkov. 2018. On internal language repre-

sentations in deep learning: An analysis of machinetranslation and speech recognition. Ph.D. thesis,Massachusetts Institute of Technology.

Terra Blevins, Omer Levy, and Luke Zettlemoyer.2018. Deep RNNs encode soft hierarchical syntax.In Proceedings of ACL.

Alexis Conneau, German Kruszewski, GuillaumeLample, Loıc Barrault, and Marco Baroni. 2018.What you can cram into a single $&#* vector: Prob-ing sentence embeddings for linguistic properties.In Proceedings of ACL.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of NAACL.

Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva,Preslav Nakov, Diarmuid O Seaghdha, SebastianPado, Marco Pennacchiotti, Lorenza Romano, andStan Szpakowicz. 2009. Semeval-2010 task 8:Multi-way classification of semantic relations be-tween pairs of nominals. In Proceedings ofthe Workshop on Semantic Evaluations: RecentAchievements and Future Directions.

Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jian-feng Gao. 2019. Multi-task deep neural networksfor natural language understanding. arXiv preprint1901.11504.

Christopher Manning, Mihai Surdeanu, John Bauer,Jenny Finkel, Steven Bethard, and David McClosky.

4598

2014. The Stanford CoreNLP natural language pro-cessing toolkit. In Proceedings of ACL: SystemDemonstrations.

Rebecca Marvin and Tal Linzen. 2018. Targeted syn-tactic evaluation of language models. In Proceed-ings of EMNLP.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Cor-rado, and Jeff Dean. 2013. Distributed representa-tions of words and phrases and their compositional-ity. In Proceedings of NIPS.

Matthew Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018a. Deep contextualized word rep-resentations. In Proceedings of NAACL.

Matthew Peters, Mark Neumann, Luke Zettlemoyer,and Wen-tau Yih. 2018b. Dissecting contextualword embeddings: Architecture and representation.In Proceedings of EMNLP.

Adam Poliak, Aparajita Haldar, Rachel Rudinger,J. Edward Hu, Ellie Pavlick, Aaron Steven White,and Benjamin Van Durme. 2018. Collecting di-verse natural language inference problems for sen-tence representation evaluation. In Proceedings ofEMNLP.

Alec Radford, Karthik Narasimhan, Tim Salimans,and Ilya Sutskever. 2018. Improving lan-guage understanding by generative pre-training.https://blog.openai.com/language-unsupervised.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan,Dario Amodei, and Ilya Sutskever. 2019. Lan-guage models are unsupervised multitask learners.https://blog.openai.com/better-language-models.

Drew Reisinger, Rachel Rudinger, Francis Ferraro,Craig Harman, Kyle Rawlins, and Benjamin VanDurme. 2015. Semantic proto-roles. Transactionsof the Association of Computational Linguistics.

Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Doesstring-based neural MT learn source syntax? In Pro-ceedings of EMNLP.

Natalia Silveira, Timothy Dozat, Marie-Catherinede Marneffe, Samuel Bowman, Miriam Connor,John Bauer, and Christopher D. Manning. 2014. Agold standard dependency corpus for English. InProceedings of the Ninth International Conferenceon Language Resources and Evaluation.

Adam Teichert, Adam Poliak, Benjamin Van Durme,and Matthew Gormley. 2017. Semantic proto-rolelabeling. In Proceedings of AAAI.

Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang,Adam Poliak, R Thomas McCoy, Najoung Kim,Benjamin Van Durme, Sam Bowman, Dipanjan Das,and Ellie Pavlick. 2019. What do you learn fromcontext? probing for sentence structure in contextu-alized word representations. In International Con-ference on Learning Representations.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In Proceedings of NIPS.

Ralph Weischedel, Martha Palmer, Mitchell Marcus,Eduard Hovy, Sameer Pradhan, Lance Ramshaw,Nianwen Xue, Ann Taylor, Jeff Kaufman, MichelleFranchini, et al. 2013. OntoNotes release5.0 LDC2013T19. Linguistic Data Consortium,Philadelphia, PA.

4599

A Appendix

A.1 Comparison of EncodersWe reproduce Figure 1 and Figure 2 (which depictmetrics on BERT-large) from the main paper be-low, and show analogous plots for the BERT-basemodels. We observe that the most important layersfor a given task appear in roughly the same relativeposition on both the 24-layer BERT-large and 12-layer BERT-base models, and that tasks generallyappear in the same order.

Additionally, in Figure A.1 we show scalar mix-ing weights for the ELMo encoder (Peters et al.,2018a), which consists of two LSTM layers over aper-word character CNN. We observe that the firstLSTM layer (layer 1) is most informative for alltasks, which corroborates the observations of Fig-ure 2 of Peters et al. (2018a). As with BERT, weobserve that the weights are only weakly concen-trated for the relations and SPR tasks. However,unlike BERT, we see only a weak concentration inthe weights on the coreference task, which agreeswith the finding of Tenney et al. (2019) that ELMopresents only weak features for coreference.

A.2 Additional ExamplesWe provide additional examples in the style ofFigure 3, which illustrate sequential decisions inthe layers of the BERT-base model.

Figure A.1: Scalar mixing weights for the ELMo en-coder. Layer 0 is the character CNN that produces per-word representations, and layers 1 and 2 are the LSTMlayers.

4600

(a) BERT-base (b) BERT-largeFigure A.2: Summary statistics on BERT-base (left) and BERT-large (right). Columns on left show F1 dev-setscores for the baseline (P (0)

τ ) and full-model (P (L)τ ) probes. Dark (blue) are the mixing weight center of gravity;

light (purple) are the expected layer from the cumulative scores.

(a) BERT-base (b) BERT-largeFigure A.3: Layer-wise metrics on BERT-base (left) and BERT-large (right). Solid (blue) are mixing weights s(`)τ ;outlined (purple) are differential scores ∆

(`)τ , normalized for each task. Horizontal axis is encoder layer.

4601

Figure A.4: Trace of selected annotations that intersectthe token “basque” in the above sentence. We see themodel recognize this as part of a proper noun (NNP)in layer 2, which leads it to revise its hypothesis aboutthe constituent “petro basque” from an ordinary nounphrase (NP) to a nominal mention (NML) in layers 3-4.We also see that from layer 3 onwards, the model rec-ognizes “petro basque” as either an organization (ORG)or a national or religious group (NORP), but does notstrongly disambiguate between the two.

Figure A.5: Trace of selected annotations that intersectthe second “today” in the above sentence. The odel ini-tially believes this to be a date and a common noun, butby layer 4 realizes that this is the TV show (entity tagWORK OF ART) and subsequently revises its hypothe-ses about the constituent type and part-of-speech.

Figure A.6: Trace of selected coreference annotationson the above sentence. Not shown are two coreferenceedges that the model has correctly resolved at layer0 (guessing from embeddings alone): “him” and “thehurt man” are coreferent, as are “he” and “he”. We seethat the remaining edges, between non-coreferent men-tions, are resolved in several stages.

Figure A.7: Trace of selected coreference and SRL an-notations on the above sentence. The model resolvesthe semantic role (purpose, ARGM-PRP) of the phrase“to help him” in layers 5-7, then quickly resolves atlayer 8 that “him” and “he” (the agent of “stop”) arenot coreferent. Also shown is the correct predictionthat “him” is the recipient (ARG1, patient) of “help”.