Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 976–988April 19 - 23, 2021. ©2021 Association for Computational Linguistics

976

Identifying Named Entities as they are Typed

Ravneet Singh Arora Chen-Tse Tsai Daniel Preotiuc-PietroBloomberg

{rarora62,ctsai54,dpreotiucpie}@bloomberg.net

Abstract

Identifying named entities in written text isan essential component of the text processingpipeline used in applications such as text ed-itors to gain a better understanding of the se-mantics of the text. However, the typical ex-perimental setup for evaluating Named EntityRecognition (NER) systems is not directly ap-plicable to systems that process text in realtime as the text is being typed. Evaluationis performed on a sentence level assumingthe end-user is willing to wait until the en-tire sentence is typed for entities to be iden-tified and further linked to identifiers or co-referenced. We introduce a novel experimentalsetup for NER systems for applications wheredecisions about named entity boundaries needto be performed in an online fashion. We studyhow state-of-the-art methods perform underthis setup in multiple languages and proposeadaptations to these models to suit this new ex-perimental setup. Experimental results showthat the best systems that are evaluated on eachtoken after its typed, reach performance within1–5 F1 points of systems that are evaluated atthe end of the sentence. These show that entityrecognition can be performed in this setup andopen up the development of other NLP tools ina similar setup.

1 Introduction

Automatically identifying named entities such asorganizations, people and locations is a key com-ponent in processing written text as it aids withunderstanding the semantics of the text. Namedentity recognition is used as a pre-processing stepto subsequent tasks such as linking named entitiesto concepts in a knowledge graph, identifying thesalience of an entity to the text, identifying coref-erential mentions, computing sentiment towardsan entity, in question answering or for extractingrelations.

Figure 1: An example of the proposed task and evalua-tion setup. After the word ‘Foreign’ is typed, the modelimmediately predicts an NER label for this word, onlyusing left context (‘A spokesman for’) and the word it-self. The prediction is then compared against the goldlabel to compute token-level F1 score. This token’s pre-diction will not be changed, even if the model’s internalprediction for it can be revised later as more tokens aretyped.

Identifying named entities as they are typed bene-fits any system that processes text on the fly. Exam-ples of such applications include: a) News editors –where named entities can be highlighted, suggested(auto-completion), co-referenced or linked as theeditor is typing; b) auto-correct – where namedentities that are just typed are less likely to needcorrection as they may come from a different lan-guage or be out-of-vocabulary (OOV); c) simulta-neous machine translation – where translation ofOOV named entities requires different approaches;d) live speech-to-text (e.g., TV shows) – wherenamed entities are more likely to be OOV, hencethe transcription should focus more on the pho-netic transcription rather than on n-gram languagemodelling.

This paper introduces a novel experimental setupof Named Entity Recognition systems illustratedin Figure 1. In this setup, inference about the spanand type of named entities is performed for eachtoken, immediately after it was typed. The sentencelevel tag sequence is composed through appendingall individual token predictions as they were made.The current named entity recognition systems thatare trained and evaluated to predict full sentencesare likely to under-perform in this experimentalsetup as they: expect that right context is available,

977

are faced with unseen types of inputs in the formof truncated sentences and can not reconcile thefinal sentence-level tag sequence across the entiresentence as the result may not be a valid sequence.

The goal of this study is to present a comprehen-sive analysis of the task of NER in the as-you-typescenario, with the following contributions:a) A novel experimental setup for conducting

named entity recognition experiments, denotedas the as-you-type scenario;

b) Experiments with state-of-the-art sentence-levelapproaches to named entity recognition in theas-you-type setup across three languages, whichindicate a 1–5 F1 points decrease compared tosentence-level inference;

c) Tailored methods for as-you-type entity recog-nition models which reduce the gap to entiresentence-level inference by 9–23% compared toregular approaches;

d) An extensive analysis of existing data sets inthe context of this task and model error analysis,which highlight future modelling opportunities.

2 Related Work

Named Entity Recognition is most commonlytreated as a sequence labelling problem, wherea prediction of whether a token is an entity andits type is done jointly for all tokens in the sen-tence. Over the past recent years, the dominant ap-proach is based on recurrent neural networks, suchas LSTMs (Hochreiter and Schmidhuber, 1997).These architectures use a stacked bi-directionalLSTM units to transform the word-level featuresinto distributions over named entity tags (Huanget al., 2015). Usually, an additional ConditionalRandom Field (CRF) (Lafferty et al., 2001) is usedon the BiLSTM output in order to take into bettermodel neighbouring tags. The tokens inputs arerepresented using one or a concatenation of pre-trained static word embeddings such as GloVe (Maand Hovy, 2016), contextual word embeddings (Pe-ters et al., 2018; Akbik et al., 2018; Devlin et al.,2019), pooled contextual word embeddings (Akbiket al., 2019b) or character embeddings trained us-ing BiLSTMs (Lample et al., 2016) or CNNs (Maand Hovy, 2016; Chiu and Nichols, 2016).

In addition to research on improving the perfor-mance of the NER model, other experimental se-tups have been proposed for this task. These in-clude domain adaptation, where a model trainedon data from a source domain is used to tag data

from a different target domain (Guo et al., 2009;Greenberg et al., 2018; Wang et al., 2020), tem-poral drift, where a model is tested on data fromfuture time intervals (Derczynski et al., 2016; Ri-jhwani and Preotiuc-Pietro, 2020), cross-lingualmodelling where models trained in one languageare adapted to other languages (Tsai et al., 2016;Ni et al., 2017; Xie et al., 2018), identifying nestedentities (Alex et al., 2007; Lu and Roth, 2015) orhigh-precision NER models (Arora et al., 2019).

However, all these experimental setups assumethat training is done over full length sentences. Per-haps the most related experimental setup to theone we propose for the task of entity recognitionis the task of simultaneous machine translation.In this setup, the task is to generate an automatictranslation in a target language as the text is beingprocessed in the source language. The goal of thetask is to produce a translation that is as accurateas possible while limiting the delay as compared tothe input. Initial approaches involved identifyingtranslatable segments and translating these indepen-dently (Fugen et al., 2007; Bangalore et al., 2012;Fujita et al., 2013) or by learning where to segmentin order to optimize the system’s performance (Odaet al., 2014). More recent approaches involve learn-ing training an agent, usually using reinforcementlearning, that makes a set of decisions of whetherto should wait for another word from the input orwrite a token to the output (Gu et al., 2017). Otheroperations are shown to help, including predict-ing the verb (Grissom II et al., 2014) or the nextword (Alinejad et al., 2018), better decoding withpartial information (Cho and Esipova, 2016), andconnecting the machine translation system to theagent’s decisions (Gu et al., 2017).

Our experimental setup is different as we do notwant to wait for another input token before wemake a prediction about the named entity. We ana-lyze the impact a delay has, albeit our experimentalsetup does not aim to combine quality and delay.The challenges are related, as the input may containimportant cues for the translation or named entitydecision after the current token or towards the endof the sentence, such as the verb in verb-final (SOV)languages such as German (Grissom II et al., 2014).The proposed as-you-type NER model can be use-ful to improve simultaneous machine translation.

978

3 Experimental Setup

We propose a new experimental setup for the stan-dard task of Named Entity Recognition that wouldbest suit real-time applications that need to processtext in an online fashion.

In the regular NER experimental setup, a modelis presented with a sequence of inputs X ={x1, x2, ..., xn} and it outputs a sequence of la-bels Y = {y1, y2, ..., yn} where yi ∈ K ={O}+ E × T , where E are the set of entity typesand T is the entity tag representation. Through-out the rest of the paper, we use the BIO taggingscheme (T = {B, I}), as this is arguably the mostpopular and differences in results between thistagging scheme and others, such as the BILOUscheme, are very small in practice (Ratinov andRoth, 2009). The types of entities we consider areE = {ORG,PER,LOC,MISC}.

The as-you-type named entity recognition setupassumes that the editor writing the text X ={x1, x2, ..., xn} needs each label prediction yi rightafter the corresponding token xi was typed. In thiscase, the information available for predicting yiis only the sub-sequence X1,i = {x1, x2, ..., xi}.The sequence Y = {y1, y2, ..., yn} is obtained byconcatenating the individual yi predictions madefor each token. Token-level micro F1 score is usedas the metric in our experiments. The evaluationprocess is illustrated in Figure 1.

This setup presents the model with several chal-lenges. First, the model has no access to rightcontext when making the prediction for each tag.However, this information is available in training.Secondly, the output sequence may contain invalidsequences of tags. For example, in the output se-quence, B-ORG could be followed by I-LOC ifthe model decided to revise its predictions basedon new information, but the evaluation setup pre-vents the model from revising the previous wronglypredicted tag (i.e. B-ORG). Lastly, sequences andsentences of the same length are likely to be quali-tatively different and the model might need to adaptin training in order to account for these differences.

We note that this experimental setup can furtherbe extended to account for delays in prediction, totrade-off between delays and quality or to predictentities before they are typed.

4 Data

We test our methods on three different data setscovering three different languages. We use the

data sets released as part of CoNLL shared tasksin 2002 for Spanish (Tjong Kim Sang, 2002)1 andin 2003 for English and German (Tjong Kim Sangand De Meulder, 2003).2 The data sets containfour types of named entities: persons, locations,organizations and names of miscellaneous entitiesthat do not belong to the previous three types. Weuse the standard train, dev and test splits definedfor these data sets.

We chose these data sets as they are arguablythe most popular data sets for performing namedentity recognition and are regularly used as bench-marks for this task. We use the data sets in differentlanguages in order to compare the impact of thelanguage on the experimental results, identify if thecommonalities and peculiarities for performing as-you-type entity recognition in different languagesand draw more robust conclusions regarding ourtask setup.

4.1 Data Analysis

We perform a quantitative analysis of the data setsin order to develop some intuitions about the data inthe context of the as-you-type experimental setup.Sentence Length and Tag Distribution First, westudy the distribution of sentence lengths. Figure 2shows that for English, most sentences are veryshort (under 10 tokens) and the most frequent sen-tence length is two. These are expected to poseproblems in the as-you-type scenario, as the contextis limited. German sentences are slightly longer,while the Spanish sentences are longest, except fora spike of sentences of length one.

Figure 3 shows the distribution of entity typesin sentences of varying lengths for English. Forclarity, we remove MISC tags from this plot asthese are infrequent. We observe there are majordifferences in tag distributions, especially shortersentences (<5 tokens) containing both more entitytags as well as different tag distributions. For ex-ample, almost 30% of locations (B-LOC or I-LOC)are in sentences of length two, which are the mostfrequent in the English data, while in longer sen-tences, these are around 5%. Organizations aremost frequent in sentences of length between 4and 7 tokens, while persons are most frequent insentences longer than 7 tokens.Token Position and Tag Distribution To furtherinvestigate the positional bias of different tags, Fig-

1https://www.clips.uantwerpen.be/conll2002/ner/2https://www.clips.uantwerpen.be/conll2003/ner/

979

Figure 2: Distribution of sentence lengths.

Figure 3: Distribution of entity types in terms of sen-tence length in the English CoNLL data set.

Figure 4: Distribution of entity types at each token po-sition in the English CoNLL data set. B and I tags aremerged for the same entity types and removed MISCtags as infrequent for clarity. O tag frequency can beinferred from the rest.

ure 4 shows the distribution of tags in the k-thtoken of the sentence. We observe that the firsttokens of a sentence are much more likely to con-tain entities. The first position is most likely to bean ORG, with PER being the most frequent in thesecond to fourth positions, followed by LOC beingthe most prevalent for the next position, with PERagain most frequent in further positions. These ob-servations are likely to complicate the as-you-typeinference for named entities, as a higher proportionof tokens will have to be inferred with no or littleright context. Comparing Figures 3 and 4 showsthat the model will be faced with different tag distri-butions when inferring the tag for the k-th token ina truncated sentence then to what it has observed insentences of length k, which provides an intuitionfor our modelling strategies.

5 Methods

This section describes the methods used to per-form named entity recognition in the as-you-typescenario. We use a base neural architecture thatachieves state-of-the-art performance on the stan-dard sentence-level NER task. We study its per-formance and observe the impact of different vari-ants of the architecture in the as-you-type scenario.Following, we propose changes to the model toadapt to the as-you-type setup. We use the Flairpackage to conduct our experiments (Akbik et al.,2019a).3 Implementation details and hyperparam-eter choices for all models are listed in the Ap-pendix.

5.1 Base Architecture

We adopt the BiLSTM-CRF model proposedin (Huang et al., 2015) with the addition of char-acter representation (Lample et al., 2016; Ma andHovy, 2016). In this architecture, the word repre-sentations are fed into a bi-directional LSTM, andthen the concatenated forward and backward vec-tors are passed through one layer of feed-forwardneural network to produce a |K| dimensional out-put for each word, where each value represents ascore associated with each label. Finally, a Con-ditional Random Field (CRF) layer is applied tomake a global decision for the entire sentence. Thishas the role of reconciling the independent predic-tions and modeling the constraints in the outputspace (e.g. I-PER can not follow B-ORG).

5.2 Architecture Variants

We start with studying different variants of the baseneural architecture for the as-you-type scenario.The key challenge in the as-you-type setting is thatthe model is not presented with the future or rightcontext (words after the current word) at test time.A natural idea is to remove information from thiscontext during training as well. The variants weconsider are based on changing the following threemodeling componentsEmbeddings We first study the impact of differentways in which input tokens are represented. Pre-trained word embeddings obtained state-of-the-artperformance on the NER task when they were intro-duced (Lample et al., 2016). These representationsare used to initialize the word embeddings, are thenfine-tuned on the training data and are concatenated

3https://github.com/zalandoresearch/flair

980

with a character-level representation of the wordobtained using BiLSTMs initialized with randomcharacter embeddings.

Contextual word embeddings extend this con-cept to obtain different word representations forthe same token in based on its context. In the stan-dard sentence-level evaluation, contextual wordembeddings were shown to obtain 2–3 F1 pointsimprovement on the English CoNLL data set (Pe-ters et al., 2018; Akbik et al., 2018; Devlin et al.,2019). Without right context, the quality of wordrepresentations could be more crucial than in thestandard setting. In this study, we test the perfor-mance of using classic embeddings – GloVe em-beddings for English (Pennington et al., 2014) andFastText embeddings (Bojanowski et al., 2017) forGerman and Spanish as well as the character basedcontextual Flair embeddings, which achieve state-of-the-art performance on the English and GermanCoNLL data sets (Akbik et al., 2018). We alsoexperimented with contextual ELMO embeddings(Peters et al., 2018) which showed slightly lowerperformance when compared to the Flair embed-dings and hence only Flair numbers are reporteddue to space limitations.

However, contextual embeddings are trained withright context available. We experiment with remov-ing this dependency from the trained embeddingsand observe if this improves the performance in theas-you-type setting, as the test scenario is more sim-ilar to the training one. We note that right contextis never observed in inference beyond the currenttoken such that there is no leakage of information.

BiLSTM Bidirectional LSTM stacks two recurrentneural networks: one starts from the beginning ofthe sentence, and another starts from the end ofthe sentence. This performs better than the uni-directional variant on sentence-level experimentsand shows that both types of context (left and right)are important for identifying and typing entity men-tions. In the as-you-type setting, we compare uni-directional LSTM modelling left context with thebidirectional LSTM model that models both typesof contexts in training.

Conditional Random Field The CRF assigns la-bels for words in a sentence jointly, ensuring labelassignments are coherent. When running inferencein the as-you-type setting, the model often seestruncated sentences which, as shown in Section 4may have different label distributions. This dis-crepancy between training and test sequences may

Figure 5: Architecture diagram highlighting the pro-posed feature weighting method described in Section5.3.3

degrade the usefulness of the CRF. We experimentto see if and how the CRF is useful in the as-you-type scenario.

5.3 Model Adaptations for the As-you-typeSetup

The as-you-type experimental setup presents themodel with a new type of evaluation, which doesnot correspond to the one used in training. We thuspropose the following approaches to bridge the gapbetween the setup and how the model is trained.

5.3.1 Weighted Loss

The model only observes the partial backward con-text for the tokens in the sequence during the as-you-type inference. In training, since the modelhas access to the entire sequence, it is likely thatthe model becomes too dependent on the presenceand reliability of the backward features, especiallyfor predicting the initial tokens.

In order to bias the model to be more robust tothe absence or unreliable backward context, we de-sign a new loss function that combines the originalBiLSTM-CRF loss with the loss from the unidi-rectional LSTM features. From the latter loss, wealso remove the influence of CRF as it also capturessignal from the right context and, for contextual em-beddings, remove the backward embeddings fromthe input to the LSTM. The resulting loss is:

Lweighted = LBiLSTM-CRF + w ∗ LLSTM-NoCRF

where w is a weight treated as a hyper-parameterin our experiments.

981

5.3.2 Final Token RepresentationThe final token in a training sequence is treated ina special way in the base model. First, a stop tag isregularly used to capture the information associatedwith a the last token in the sequence. Secondly, thebackward features for the final token in a sequenceare not observed, as there is no right context, sothese are initialized with a random vector. Whileboth are useful in the regular setup as it capturesa consistent pattern the final words follow, thisdoes not hold for the as-you-type setup, where eachtoken in a sequence will be the final token for itsprediction, as partial sequences are observed.

We thus assume these choices add noise in oursetup, thus we both remove the stop tag togetherwith any transition scores to it during training andevaluation and remove the backward feature of thelast token and initialize it with a zero vector.

5.3.3 Feature WeightingThe previous approach relied on the intuition oflearning the trade-off between forward and back-ward features in order to better adapt to the infer-ence setup. However, this trade-off is likely im-pacted by the position of the token in the sequence.

Thus, we explore a technique similar to (Moonet al., 2018) that allows the model to learn the im-portance of backward features based on the positionof tokens. The is illustrated in Figure 5. We imple-ment this using position based attention weights.We apply these weights before combining the for-ward and backward features (instead of concatenat-ing) from the LSTMs as follows:

ht = hft + at ∗ hbtwhere t is the position of the token, hft and hbt areforward and backward LSTM features at t, ht isthe new output feature at t and at is the attentionweight for the backward features at position t. Theattention weights at are computed as follows:

ut = [hft ;hbt ; pt]; at = σ(W.ut + b)

At every position t, the feature vector is calculatedby concatenating the forward, backward and the po-sitional embedding for that token. Attention weightat is calculated by applying attention weight matrixW followed by the sigma activation. We do notapply at for forward features since they are alwayscomplete and reliable even for partial sentences.

We follow the structure of positional embeddingsintroduced in (Vaswani et al., 2017) and defined as:

p2it = sin(t

100002i/d); p2i+1

t = cos(t

100002i/d)

where d is the size of positional embedding, i isthe dimension and t is the position. The valuesin a positional embedding are sin and cosine func-tions whose period is 100002i/d ∗ 2π. Positionalsinusoidal embedding allows to encode longer se-quences that are not present in training.

Since the right hand side context decreases as wemove from left to right of a sequence in training, wewould like our attention weights to consider howfar a token lies from the final token in a sequence.To achieve this, we calculate position index of to-kens from the end of the sentence which makessure that a token lying at the final position alwaysreceives an index of 0, producing the same posi-tional embedding and the input to attention weightsdoes not fluctuate from one sequence to another.

5.3.4 Embedding WeightingWe perform a similar operation using attention atthe embedding stage to trade-off between backwardand forward contextual token representations. Theinput embeddings are calculated as follows:

xt = ewt + eft + aet ∗ ebtvt = [ewt ; e

ft ; e

bt ; pt]; a

et = σ(We.vt + be)

where ew are the classical word embeddings and ef ,eb are Flair forward and backward embeddings. Anarchitecture diagram is presented in the Appendix.

6 Results

We present the experimental results of the variousNER models in the as-you-type setup, contrastingthem with the regular sentence-level setup. Allmodels are trained using the standard splits for theCoNLL data sets. The evaluation metric is token-level micro F1, as this is reflective of our evaluationsetup where each token is evaluated separately.

The top section of Table 1 shows the results ofthe different variants of the base architecture inSection 5.2. The overall performance drop in theas-you-type setup compared to the sentence-levelsetup ranges from 4.80 F1 for English to only 1.53F1 for Spanish when comparing the best perform-ing models. This is expected, as the as-you-typescenario is more challenging, especially for Englishwhere our data analysis from Section 4.1 showedthat tokens are more prevalent in short sentenceswhich are overall more frequent. For Spanish,where the performance difference is smallest, iswhere we have on average the longest sentencesand in which left context alone is in most casesenough to make the correct inference.

982

English German Spanish

Embedding LSTM CRF As-you-type Sentence As-you-type Sentence As-you-type Sentence

Classic

� 3 83.27 90.97 72.20 78.67 67.56 80.40→ 3 85.12 88.99 70.54 77.21 64.84 80.28� 7 79.06 86.87 73.49 77.89 74.61 80.74→ 7 83.75 83.27 75.73 75.73 77.30 77.30

Flair (�)

� 3 84.15 92.75 79.79 84.32 81.80 89.43→ 3 84.82 91.63 79.98 84.11 82.23 88.82� 7 84.50 92.23 79.84 83.73 85.37 88.80→ 7 85.87 90.73 80.50 82.74 85.32 89.06

Flair (→)

� 3 85.60 92.19 79.21 82.94 81.61 88.76→ 3 86.92 90.36 78.34 81.99 82.60 88.21� 7 85.13 91.76 79.30 81.88 86.79 88.83→ 7 87.95 87.95 80.79 80.79 87.90 87.90

Adaptations for the as-you-type setup

Flair (�)

� Weighted Loss 87.77 92.46 80.39 83.71 84.13 89.48+ Final Token Rep 88.00 92.40 80.59 83.71 87.23 89.48+ Feature weighting 88.21 92.24 80.61 84.16 87.38 89.23+ Embedding weighting 88.40 92.29 81.62 84.77 87.72 89.79

Table 1: Evaluation results of LSTM-based NER models in the as-you-type and sentence-level evaluation setups asmeasured using token-level micro F1. Arrows indicate if uni- (→) or bi-directional (�) training is used. Modelswith the best results across their setup are in bold. Best results within the class of methods are underlined. Forclassic word embeddings, we use GloVe for English, and FastText for German and Spanish. Results are averagedacross three runs.

We note that in all three data sets, the best resultsin the as-you-type setting are obtained by match-ing the training setup to that of testing by onlykeeping a uni-directional LSTM that processes thetext left to right and Flair embeddings only trainedusing left context. Flair embeddings trained onlyusing left context are in all cases better than the bi-directional ones, which is natural as those embed-dings would conflate information that is not avail-able in inference. Uni-directional LSTMs performoverall better than bi-directional LSTMs by an av-erage of a few percentage points as bi-directionalLSTMs are likely learning information that will notbe available in testing.

Adding the CRF hurts performance in all exceptone case when holding everything else constant,sometimes by wide margins (e.g. 5.3 F1 drop onSpanish with Flair forward embeddings and uni-directional LSTM). We attribute this to the mis-match between the structure of the sequences inthe training data containing only full sentences,when compared to truncated sentences which canbe observed by comparing Figures 3 to Figures 4.

The bottom section of Table 1 shows the results ofour as-you-type adaptation strategies. All proposedmethods are added on top of each other in orderto study their individual contributions. For brevity,we only present the results of using Flair forwardand backward embeddings as these performed best.

The changes to the last token representationand weighted loss improves on the regular bi-directional LSTM model by a substantial margin,adding between 0.45 for Spanish up to 4.25 F1on English on the as-you type setup performance.We also notice that the sentence-level evaluationis near to the regular model performance (-0.39for German to +0.05 for Spanish), showing that theweighted loss is able to achieve a good compromisebetween the representations.

Adding the feature weighting based on positionmarginally improves performance, between 0.02 onGerman to 0.21 on English. However, the weight-ing through attention is more effective at the em-bedding level improving on the previous model bybetween 0.19 F1 on English to 1.01 F1 on German.Overall, our as-you-type adaptation methods im-prove on the best variation of the base architectureon English (+0.45 F1) and German (+0.83 F1). Themodel is competitive on Spanish (-0.12 F1) to theFlair forward unidirectional LSTM with no CRF,albeit this is likely driven by the very long aver-age sentence length in the Spanish data set (seeFigure 2). Overall, the improvements represent be-tween 9.3% - 23% error reduction when comparingto the best as-you-type and sentence level setups.

We highlight that an additional benefit of theproposed adaptation methods is that the model re-tains high performance on the sentence-level setup,

983

in contrast with the Flair forward uni-directionalLSTM, which performs between 1.89 (Spanish)and 4.34 (English) worse on the sentence-level.

Finally, the results of our proposed model are ac-tually marginally better than the state-of-the-art ap-proach of BiLSTM+CRF using Flair embeddingson German (+0.45 F1) and Spanish (+0.73 F1),albeit this was not the original goal of our addi-tions. This highlights that the proposed modellingideas are more generally beneficial as they push themodel to learn more robust representations.

7 Error Analysis

Finally, we perform error analysis to identify thelimitations of our approaches.

7.1 Confusion Matrix

We first study prediction difference between as-you-type and sentence-level setup. Figure 6 showsthe confusion matrix on the English data set. Bothmodels are BiLSTM-CRF with Flair embeddings,but the as-you-type model is trained with the bestsetting from Table 1. We can see that most confu-sions are between LOC and ORG: 7.9% of I-ORGtokens in the full-sentence setting are classifiedas I-LOC in the as-you-type setting, and 7.6% ofB-ORG tokens are classified as B-LOC. Withoutright context, it is very challenging to distinguishthese two types. For example, the data set con-tains many sport teams that contain location nametokens. Another noticeable difference is that theas-you-type model makes more O predictions. Forinstance, 5.8% of B-MISC are classified as O. Thiscan be due to the limited availability of cues foridentifying entities when right context is missing.

7.2 Positional Prediction

We expect that the size of the context impacts thequality of the inference more acutely in the as-you-type scenario when compared to the sentence-levelsetup. Figure 7 plots the predictive performance ofthree English models across each token’s position.This confirms our intuition that the as-you-typesetup especially impacts prediction on the first to-kens in a sentence, which are more entity rich asshown in Section 4.1. However, we see that thereis still a difference compared to the standard NERsetup (blue curve) across all positions, confirmingthat indeed the right context can add more infor-mation regardless of the position of the token. Theplot also highlights the performance gains of our

Figure 6: Confusion matrix of tag type predictionwhen comparing the best as-you-type and sentence-level models.

Figure 7: F1 scores at various token positions averagedacross all the sentences in the English data set. Theoverall performance of each model is listed in the leg-end.

adaptation methods at the first tokens when com-pared with the BiLSTM-CRF model evaluated inthe as-you-type setting (orange curve).

7.3 Delayed Entity Prediction

Based on the previous results and examples, wewant to study the impact of delaying the entitydecision by one token. This would account forcases where the first entity word is ambiguous andalso reduce the number of invalid sequences thatcan be produced by the as-you-type inference. Forexample, in the case of the ‘Zimbabwe Open’ en-tity, if the model predicted the tag of the token‘Zimbabwe’ as B-LOC and after seeing the nexttoken (‘Open’), it revises this token’s predictionas I-MISC, it is unable to change the B-LOC tag-ging, thus creating an invalid sequence, but stillobtains partial credit on the token F1 score. De-laying the decision of the first token could haveallowed the model to correct its decision for thefirst token (‘Zimbabwe’) to B-MISC, resulting in avalid and more accurate sequence.

We study the possibility of delaying the predic-tion of a single token (not multiple tokens) by using

984

English German Spanish

Thr. Tok% F1 Tok% F1 Tok% F1

0 0% 88.4 0% 81.62 0% 87.720.6 0.77% 89.18 0.97% 82.65 0.66% 88.200.7 1.28% 89.56 1.65% 83.09 1.15% 88.470.8 2.17% 90.05 2.43% 83.74 1.80% 88.660.9 3.25% 90.53 3.61% 84.01 2.80% 88.641.0 100% 91.20 100% 84.22 100% 88.91

Full sentence 92.75 84.32 89.43

Table 2: Results of the proposed wait-one policy. Ifthe probability of prediction is less than or equal to thegiven threshold (Thr. column), we wait for one moretoken and predict again. The column Tok% indicatespercentage of tokens which have a delayed prediction.The best performing model in the as-you-type setup isused. The performance of the best full-sentence modelis listed in the last row for comparison purposes.

a threshold on the tag output. Table 2 shows theresults of several threshold values and their impacton the total F1.

We observe that if we delay prediction by onetoken for all tokens (Thr = 1.0), the performance isvery close to the best full-sentence model, obtain-ing an error reduction rate of 64% (1.55 comparedto 4.35) for English. Moreover, we can obtain a50% reduction rate by only delaying the decisionon 3.25% of the tokens if the downstream applica-tion deems this acceptable. These results highlightthe importance of the immediate right context.

8 Untyped Entity Detection

Named entity recognition combines two differentcomponents: entity detection – identifying thenamed entity spans – and typing – identifying en-tity types. We study the impact of typing in the as-you-type setup by removing the typing informationfrom the output labels (E = {ENT}), thus reduc-ing the output space to K = {B-ENT, I-ENT, O}.

Results using the best as-you-type models withand without as-you-type adaptations are shown inTable 3. Comparing with the numbers in Table 1,the untyped F1 score of as-you-type setting is muchcloser to the standard sentence-level evaluation, be-ing within 1 point of F1 for all languages. Thishighlights that the challenging part of the as-you-type setting is entity typing. For example, ‘Zim-babwe’ is a location on its own, but ‘ZimbabweOpen’ is an event (MISC entity type) while ‘West’is usually indicative of a first location token (e.g.‘West Pacific’), but can also refer to an organization(e.g. ‘West Indies’ when referencing the cricketteam). The proposed technique results are in thiscase less conclusive, which is somewhat expected

Sentence As-you-type

Original Embedding weighting

English 97.49 97.11 97.09German 91.37 89.29 89.49Spanish 97.70 97.05 96.96

Table 3: Entity identification performance. The fourentity types are collapsed into one type when comput-ing token-level F1 scores. The model for “Embeddingweighting” is BiLSTM-CRF with bidirectional Flairembeddings for all three languages. For the “Original”setting, we use forward LSTM with forward Flair em-beddings.

as the differences in entity frequencies between fullsentences and truncated sentences are smaller.

9 Conclusions

This paper introduced and motivated the as-you-type experimental setup for the popular NER task.We presented results across three different lan-guages, which show the extent to which sentence-level state-of-the-art models degrade in this setup.Through insights gained from data analysis, weproposed modelling improvements to further re-duce the gap to the regular sentence-level perfor-mance. Our error analysis highlights the cases thatpose challenges to the as-you-type scenario anduncovers insights into way to further improve themodelling of this task.

This setup is tailored for end-applications suchas text editors, speech-to-text, machine translation,auto-completion, or auto-correct. For text editors,the editor would be able to receive suggestionsfor entities inline, right after they type the entity,which can further be coupled with a linking algo-rithm. This would increase the user experience andefficiency of the editor, as they can make selectionsabout entities inline (similar to a phone’s autocor-rect), rather than having to go back over the entiresentence after it was completed.

Another avenue of future work would be to cou-ple the NER as-you-type with ASR data and usingmethods that adapt NER to noisy ASR input (Ben-ton and Dredze, 2015) for building an end-to-endlive speech to entities system.

Acknowledgements

We would like to thank Leslie Barrett, MayankKulkarni, Amanda Stent, Umut Topkara and theother members of the Bloomberg AI group. Theyprovided invaluable feedback on the experimentsand the paper.

985

ReferencesAlan Akbik, Tanja Bergmann, Duncan Blythe, Kashif

Rasul, Stefan Schweter, and Roland Vollgraf. 2019a.FLAIR: An easy-to-use framework for state-of-the-art NLP. In Proceedings of the 2019 Confer-ence of the North American Chapter of the Associa-tion for Computational Linguistics (Demonstrations),pages 54–59, Minneapolis, Minnesota. Associationfor Computational Linguistics.

Alan Akbik, Tanja Bergmann, and Roland Vollgraf.2019b. Pooled contextualized embeddings fornamed entity recognition. In Proceedings of the2019 Conference of the North American Chapter ofthe Association for Computational Linguistics: Hu-man Language Technologies, Volume 1 (Long andShort Papers), pages 724–728, Minneapolis, Min-nesota. Association for Computational Linguistics.

Alan Akbik, Duncan Blythe, and Roland Vollgraf.2018. Contextual string embeddings for sequence la-beling. In Proceedings of the 27th International Con-ference on Computational Linguistics, pages 1638–1649, Santa Fe, New Mexico, USA. Association forComputational Linguistics.

Beatrice Alex, Barry Haddow, and Claire Grover. 2007.Recognising nested named entities in biomedical text.In Biological, translational, and clinical languageprocessing, pages 65–72, Prague, Czech Republic.Association for Computational Linguistics.

Ashkan Alinejad, Maryam Siahbani, and Anoop Sarkar.2018. Prediction improves simultaneous neural ma-chine translation. In Proceedings of the 2018 Confer-ence on Empirical Methods in Natural Language Pro-cessing, pages 3022–3027, Brussels, Belgium. Asso-ciation for Computational Linguistics.

Ravneet Arora, Chen-Tse Tsai, Ketevan Tsereteli, Prab-hanjan Kambadur, and Yi Yang. 2019. A semi-Markov structured support vector machine model forhigh-precision named entity recognition. In Pro-ceedings of the 57th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 5862–5866, Florence, Italy. Association for ComputationalLinguistics.

Srinivas Bangalore, Vivek Kumar Rangarajan Sridhar,Prakash Kolan, Ladan Golipour, and Aura Jimenez.2012. Real-time incremental speech-to-speech trans-lation of dialogs. In Proceedings of the 2012 Con-ference of the North American Chapter of the As-sociation for Computational Linguistics: HumanLanguage Technologies, pages 437–445, Montreal,Canada. Association for Computational Linguistics.

Adrian Benton and Mark Dredze. 2015. Entity link-ing for spoken language. In Proceedings of the2015 Conference of the North American Chapter ofthe Association for Computational Linguistics: Hu-man Language Technologies, pages 225–230, Den-ver, Colorado. Association for Computational Lin-guistics.

Piotr Bojanowski, Edouard Grave, Armand Joulin, andTomas Mikolov. 2017. Enriching word vectors withsubword information. Transactions of the Associa-tion for Computational Linguistics, 5:135–146.

Jason P.C. Chiu and Eric Nichols. 2016. Named entityrecognition with bidirectional LSTM-CNNs. Trans-actions of the Association for Computational Linguis-tics, 4:357–370.

Kyunghyun Cho and Masha Esipova. 2016. Can neu-ral machine translation do simultaneous translation?arXiv preprint arXiv:1606.02012.

Leon Derczynski, Kalina Bontcheva, and Ian Roberts.2016. Broad Twitter corpus: A diverse named en-tity recognition resource. In Proceedings of COLING2016, the 26th International Conference on Compu-tational Linguistics: Technical Papers, pages 1169–1179, Osaka, Japan. The COLING 2016 OrganizingCommittee.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, Volume 1 (Long and Short Papers), pages4171–4186, Minneapolis, Minnesota. Association forComputational Linguistics.

Christian Fugen, Alex Waibel, and Muntsin Kolss.2007. Simultaneous translation of lectures andspeeches. Machine Translation, 21(4):209–252.

Tomoki Fujita, Graham Neubig, Sakriani Sakti,Tomoki Toda, and Satoshi Nakamura. 2013. Simple,lexicalized choice of translation timing for simulta-neous speech translation. In INTERSPEECH, pages3487–3491.

Nathan Greenberg, Trapit Bansal, Patrick Verga, andAndrew McCallum. 2018. Marginal likelihood train-ing of BiLSTM-CRF for biomedical named entityrecognition from disjoint label sets. In Proceedingsof the 2018 Conference on Empirical Methods in Nat-ural Language Processing, pages 2824–2829, Brus-sels, Belgium. Association for Computational Lin-guistics.

Alvin Grissom II, He He, Jordan Boyd-Graber, JohnMorgan, and Hal Daume III. 2014. Don’t until thefinal verb wait: Reinforcement learning for simul-taneous machine translation. In Proceedings of the2014 Conference on Empirical Methods in NaturalLanguage Processing (EMNLP), pages 1342–1352,Doha, Qatar. Association for Computational Linguis-tics.

Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Vic-tor O.K. Li. 2017. Learning to translate in real-timewith neural machine translation. In Proceedings ofthe 15th Conference of the European Chapter of theAssociation for Computational Linguistics: Volume

986

1, Long Papers, pages 1053–1062, Valencia, Spain.Association for Computational Linguistics.

Honglei Guo, Huijia Zhu, Zhili Guo, Xiaoxun Zhang,Xian Wu, and Zhong Su. 2009. Domain adapta-tion with latent semantic association for named en-tity recognition. In Proceedings of Human LanguageTechnologies: The 2009 Annual Conference of theNorth American Chapter of the Association for Com-putational Linguistics, pages 281–289, Boulder, Col-orado. Association for Computational Linguistics.

Sepp Hochreiter and Jurgen Schmidhuber. 1997. Longshort-term memory. Neural computation, 9(8):1735–1780.

Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidi-rectional LSTM-CRF models for sequence tagging.arXiv preprint arXiv:1508.01991.

John D. Lafferty, Andrew McCallum, and FernandoC. N. Pereira. 2001. Conditional random fields:Probabilistic models for segmenting and labeling se-quence data. In Proceedings of the Eighteenth In-ternational Conference on Machine Learning, pages282–289.

Guillaume Lample, Miguel Ballesteros, Sandeep Sub-ramanian, Kazuya Kawakami, and Chris Dyer. 2016.Neural architectures for named entity recognition. InProceedings of the 2016 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,pages 260–270, San Diego, California. Associationfor Computational Linguistics.

Wei Lu and Dan Roth. 2015. Joint mention extractionand classification with mention hypergraphs. In Pro-ceedings of the 2015 Conference on Empirical Meth-ods in Natural Language Processing, pages 857–867,Lisbon, Portugal. Association for Computational Lin-guistics.

Xuezhe Ma and Eduard Hovy. 2016. End-to-endsequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting ofthe Association for Computational Linguistics (Vol-ume 1: Long Papers), pages 1064–1074, Berlin, Ger-many. Association for Computational Linguistics.

Seungwhan Moon, Leonardo Neves, and Vitor Car-valho. 2018. Multimodal named entity disambigua-tion for noisy social media posts. In Proceedingsof the 56th Annual Meeting of the Association forComputational Linguistics (Volume 1: Long Papers),pages 2000–2008, Melbourne, Australia. Associationfor Computational Linguistics.

Jian Ni, Georgiana Dinu, and Radu Florian. 2017.Weakly supervised cross-lingual named entity recog-nition via effective annotation and representation pro-jection. arXiv preprint arXiv:1707.02483.

Yusuke Oda, Graham Neubig, Sakriani Sakti, TomokiToda, and Satoshi Nakamura. 2014. Optimizing seg-mentation strategies for simultaneous speech transla-tion. In Proceedings of the 52nd Annual Meeting of

the Association for Computational Linguistics (Vol-ume 2: Short Papers), pages 551–556, Baltimore,Maryland. Association for Computational Linguis-tics.

Jeffrey Pennington, Richard Socher, and ChristopherManning. 2014. GloVe: Global vectors for word rep-resentation. In Proceedings of the 2014 Conferenceon Empirical Methods in Natural Language Process-ing (EMNLP), pages 1532–1543, Doha, Qatar. Asso-ciation for Computational Linguistics.

Matthew Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018. Deep contextualized word repre-sentations. In Proceedings of the 2018 Conferenceof the North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, Volume 1 (Long Papers), pages 2227–2237,New Orleans, Louisiana. Association for Computa-tional Linguistics.

Lev Ratinov and Dan Roth. 2009. Design challengesand misconceptions in named entity recognition. InProceedings of the Thirteenth Conference on Compu-tational Natural Language Learning (CoNLL-2009),pages 147–155, Boulder, Colorado. Association forComputational Linguistics.

Shruti Rijhwani and Daniel Preotiuc-Pietro. 2020.Temporally-informed analysis of named entity recog-nition. In Proceedings of the 58th Annual Meeting ofthe Association for Computational Linguistics, pages7605–7617, Online. Association for ComputationalLinguistics.

Erik F. Tjong Kim Sang. 2002. Introduction to theCoNLL-2002 shared task: Language-independentnamed entity recognition. In COLING-02: The6th Conference on Natural Language Learning 2002(CoNLL-2002).

Erik F. Tjong Kim Sang and Fien De Meulder.2003. Introduction to the CoNLL-2003 shared task:Language-independent named entity recognition. InProceedings of the Seventh Conference on Natu-ral Language Learning at HLT-NAACL 2003, pages142–147.

Chen-Tse Tsai, Stephen Mayhew, and Dan Roth. 2016.Cross-lingual named entity recognition via wikifica-tion. In Proceedings of The 20th SIGNLL Confer-ence on Computational Natural Language Learning,pages 219–228, Berlin, Germany. Association forComputational Linguistics.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, undefine-dukasz Kaiser, and Illia Polosukhin. 2017. Attentionis all you need. In Proceedings of the 31st Interna-tional Conference on Neural Information ProcessingSystems, NIPS’17, page 6000–6010, Red Hook, NY,USA. Curran Associates Inc.

987

Jing Wang, Mayank Kulkarni, and Daniel Preotiuc-Pietro. 2020. Multi-domain named entity recogni-tion with genre-aware and agnostic inference. InProceedings of the 58th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 8476–8488, Online. Association for Computational Lin-guistics.

Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A.Smith, and Jaime Carbonell. 2018. Neural cross-lingual named entity recognition with minimal re-sources. In Proceedings of the 2018 Conference onEmpirical Methods in Natural Language Processing,pages 369–379, Brussels, Belgium. Association forComputational Linguistics.

A Appendices

A.1 Implementation and HyperparametersTo train our models we use Stochastic GradientDescent with a learning rate of 0.1 and mini batchsize of 32. The LSTM model includes 1 layer ofLSTM with hidden state size 256. We also em-ploy a dropout of 0.5 for the LSTM layer. For thepositional embeddings, the dimension d is set as100 for feature weighting and 1000 for embeddingweighting. We tried different dimensions between100 and 2000. w for weighted loss is identified as1.5 for English and 1 for German and Spanish fromthe dev set. ForW we considered values between 0and 2. All the models are trained for 70 epochs andthe best model is selected based on the token-levelF-1 score4 on dev set. We perform manual hyper-parameter selection and the final performance isreported based on the 5-10 runs of the best hyper-parameter setting.We use Flair’s standard settingsfor English.

All the models are trained on nvidia GPU andoverall training for 70 epochs takes around 5-6hours. This run-time complexity is very close tothe complexity achieved by the the Flair implemen-tation for standard NER training.

Numbers reported in (Akbik et al., 2018) are gen-erated by training models on combined train anddev sets, hence they are higher than the numberswe report when training only on the training data.We also report token-level F1, rather than entity-level F1, which leads to results that are not directlycomparable with (Akbik et al., 2018).

A.2 Visualization of Attention WeightsTo better understand the impact of positional atten-tion weights, we visualize and compare the feature-

4https://sklearn-crfsuite.readthedocs.io/en/latest/tutorial.html#evaluation

level attention weights for different tokens on a fewhand-picked English sentences. Figure 8 highlightstokens using different color intensities. Higher in-tensity represents a larger weight value and hencea stronger influence of backward context. First, itis evident that tokens in the first position rely moreheavily on backward features in the absence of anyforward context, which is further reflected in thehigher attention weights achieved by these tokens.Moreover, first tokens of multi-token entities suchas Persons (‘Nabil Abu Rdainah’), Organizations(‘NY Rangers’) and Events (‘Zimbabwe Open’) areassigned larger weights due to a high influence ofimmediate next tokens. Also, quite often the last to-ken in the sentences are weighted lower which canbe attributed to the positional information capturedby the attention weights.

A.3 Performance on Dev SetTo facilitate reproducibility of results, Table 4 re-ports the development set performance of the basemodel (Bi-LSTM CRF Flair) and the proposedmodel for both as-you-type and sentence level se-tups.

A.4 ParametersTable 5 lists different trainable parameters used inthe model along with their sizes.

988

Figure 8: Sample attention weights from the English CoNLL Data Set.

English German Spanish

Model As-you-type Sentence As-you-type Sentence As-you-type Sentence

Base Model 88.16 96.08 82.74 86.48 81.34 87.54Proposed Model 92.54 96.43 83.71 86.69 85.75 87.78

Table 4: Performance on Conll Dev set for both Bi-LSTM-CRF Flair and the proposed final model

Parameter Size

GloVe Embedding Dimension (English) 100Fast-text Embedding Dimension (Spanish, German) 300Flair Embedding Size 2,000Feature-Level Positional Embedding Size 100Embedding-Level Positional Embedding Size 1,000LSTM output size 100Bi-Directional LSTM 1,680,800Linear Output Layer 200 * 9CRF Transition Matrix 6 * 6Feature Level Attention Matrix (W ) 300*1Embedding Level Attention Matrix (We) (English) 5,100 * 1 + 1Embedding Level Attention Matrix (We) (Spanish, German) 5,300 * 1 + 1

Table 5: Number of parameters for different components of our models. When not explicitly mentioned, parametersare for models in all three languages.