Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 50–62November 7–11, 2021. ©2021 Association for Computational Linguistics

50

COMBO: State-of-the-Art Morphosyntactic Analysis

Mateusz Klimaszewski1,2 Alina Wróblewska2

1Warsaw University of Technology2Institute of Computer Science, Polish Academy of Sciences

m.klimaszewski@ii.pw.edu.plalina@ipipan.waw.pl

Abstract

We introduce COMBO – a fully neuralNLP system for accurate part-of-speech tag-ging, morphological analysis, lemmatisation,and (enhanced) dependency parsing. It pre-dicts categorical morphosyntactic featureswhilst also exposes their vector representa-tions, extracted from hidden layers. COMBOis an easy to install Python package with au-tomatically downloadable pre-trained modelsfor over 40 languages. It maintains a bal-ance between efficiency and quality. As itis an end-to-end system and its modules arejointly trained, its training is competitivelyfast. As its models are optimised for accu-racy, they achieve often better prediction qual-ity than SOTA. The COMBO library is avail-able at: https://gitlab.clarin-pl.eu/syntactic-tools/combo.

1 Introduction

Natural language processing (NLP) has long recog-nised morphosyntactic features as necessary forsolving advanced natural language understanding(NLU) tasks. An enormous impact of contextuallanguage models on presumably all NLP tasks hasslightly weakened the importance of morphosyn-tactic analysis. As morphosytnactic features areencoded to some extent in contextual word embed-dings (e.g. Tenney et al., 2019; Lin et al., 2019),doubts arise as to whether explicit morphosyntacticknowledge is still needed. For example, Glavaš andVulic (2021) have recently investigated an inter-mediate fine-tuning contextual language models onthe dependency parsing task and suggested that thisstep does not significantly contribute to advanceNLU models. Conversely, Warstadt et al. (2019) re-veal the powerlessness of contextual language mod-els in encoding linguistic phenomena like negation.This is in line with our intuition about representingnegation in Polish sentences (see Figure 1). It doesnot seem trivial to differentiate between the con-

tradicting meanings of these sentences using con-textual language models, as the word context issimilar. The morphosyntactic features, e.g. partsof speech PART vs. INTJ, and dependency labelsadvmod:neg vs. discourse:intj, could be beneficialin determining correct reading.

In order to verify the influence of explicit mor-phosyntactic knowledge on NLU tasks, it is neces-sary to design a technique for injecting this knowl-edge into models or to build morphosyntax-awarerepresentations. The first research direction wasinitiated by Glavaš and Vulic (2021). Our objec-tive is to provide a tool for predicting high-qualitymorphosyntactic features and exposing their em-beddings. These vectors can be directly combinedwith contextual word embeddings to build mor-phosyntactically informed word representations.

The emergence of publicly available NLPdatasets, e.g. Universal Dependencies (Zeman et al.,2019), stimulates the development of NLP systems.Some of them are optimised for efficiency, e.g.spaCy (Honnibal et al., 2020), and other for ac-curacy, e.g. UDPipe (Straka, 2018), the Stanfordsystem (Dozat and Manning, 2018), Stanza (Qiet al., 2020). In this paper, we introduce COMBO,an open-source fully neural NLP system whichis optimised for both training efficiency and pre-diction quality. Due to its end-to-end architec-ture, which is an innovation within morphosyn-tactic analysers, COMBO is faster in training thanthe SOTA pipeline-based systems, e.g. Stanza. Asa result of applying modern NLP solutions (e.g.contextualised word embeddings), it qualitativelyoutperforms other systems.

COMBO analyses tokenised sentences and pre-dicts morphosyntactic features of tokens (i.e. partsof speech, morphological features, and lemmata)and syntactic structures of sentences (i.e. depen-dency trees and enhanced dependency graphs). Atthe same time, its module, COMBO-vectoriser, ex-tracts vector representations of the predicted fea-

51

(1) Nie spiePART VERB

(I) don’t sleep

advmod:negroot

(2) Nie , nie spieINTJ PUNCT PART VERBNo , (I) don’t sleep

advmod:negroot

punctdiscourse:intj

(3) Nie , spieINTJ PUNCT VERBNo , (I) sleep

rootpunct

discourse:intj

Figure 1: UD trees of Polish sentences: (1) and (2) mean a non-sleeping situation and (3) means sleeping.

tures from hidden layers of individual predictors.COMBO user guide is in §4 and a live demo isavailable on the website http://combo-demo.nlp.ipipan.waw.pl.

Contributions 1) We implement COMBO (§2),a fully neural NLP system for part-of-speechtagging, morphological analysis, lemmatisation,and (enhanced) dependency parsing, together withCOMBO-vectoriser for revealing vector represen-tations of predicted categorical features. COMBOis implemented as a Python package which is easyto install and to integrate into a Python code. 2)We pre-train models for over 40 languages that canbe automatically downloaded and directly used toprocess new texts. 3) We evaluate COMBO andcompare its performance with two state-of-the-artsystems, spaCy and Stanza (§3).

2 COMBO Architecture

COMBO’s architecture (see Figure 2) is basedon the forerunner (Rybak and Wróblewska, 2018)implemented in the Keras framework. Apartfrom a new implementation in the PyTorch li-brary (Paszke et al., 2019), the novelties arethe BERT-based encoder, the EUD prediction mod-ule, and COMBO-vectoriser extracting embed-dings of UPOS and DEPREL from the last hid-den layers of COMBO’s tagging and dependencyparsing module, respectively. This section providesan overview of COMBO’s modules. Implementa-tion details are in Appendix A.

Local Feature Extractors Local feature extrac-tors (see Figure 2) encode categorical features (i.e.words, parts of speech, morphological features,lemmata) into vectors. The feature bundle is con-figurable and limited by the requirements set forCOMBO. For instance, if we train only a depen-dency parser, the following features can be inputto COMBO: internal character-based word em-beddings (CHAR), pre-trained word embeddings(WORD), and embeddings of lemmata (LEMMA),

WORDLEMMACHARUPOSUFEATS

HEAD DEPREL

UPOS UFEATS

enhHEAD enhDEPREL

TOKENISEDSENTENCES

CONCATENATED LOCAL FEATURESLO

CAL

EXTR

ACTO

RS

PRED

ICTO

RS

LEMMA

ENCODER

Figure 2: COMBO architecture. Explanations:CNN  FC System-trained biLSTM optional required

parts of speech (UPOS) and morphological features(UFEATS). If we train a morphosyntactic analyser(i.e. tagger, lemmatiser and parser), internal wordembeddings (CHAR) and pre-trained word embed-dings (WORD), if available, are input to COMBO.

Words and lemmata are always encoded us-ing character-based word embeddings (CHAR andLEMMA) estimated during system training with a di-lated convolutional neural network (CNN) encoder(Yu and Koltun, 2016; Strubell et al., 2017).

Additionally, words can be represented using pre-trained word embeddings (WORD), e.g. fastText(Grave et al., 2018), or BERT (Devlin et al., 2019).The use of pre-trained embeddings is an optionalfunctionality of the system configuration. COMBOfreezes pre-trained embeddings (i.e. no fine-tuning)and uses their transformations, i.e. embeddings aretransformed by a single fully connected (FC) layer.

Part-of-speech and morphological embeddings(UPOS and UFEATS) are estimated during systemtraining. Since more than one morphological fea-ture can attribute a word, embeddings of all pos-

52

sible features are estimated and averaged to builda final morphological representation.

Global Feature Encoder The encoder uses con-catenations of local feature embeddings. A se-quence of these vectors representing all the wordsin a sentence is processed by a bidirectional LSTM(Hochreiter and Schmidhuber, 1997; Graves andSchmidhuber, 2005). The network learns the con-text of each word and encodes its global (contex-tualised) features (see Figure 3). Global featureembeddings are input to the prediction modules.

ROOT The car is red

Figure 3: Estimation of global feature vectors.biLSTM GLOBAL

Tagging Module The tagger takes global featurevectors as input and predicts a universal part ofspeech (UPOS), a language-specific tag (XPOS),and morphological features (UFEATS) for eachword. The tagger consists of two linear layers fol-lowed by a softmax. Morphological features builda disordered set of category-value pairs (e.g. Num-ber=Plur). Morphological feature prediction is thusimplemented as several classification problems.The value of each morphological category is pre-dicted with a FC network. Different parts of speechare assigned different sets of morphological cate-gories (e.g. a noun can be attributed with grammati-cal gender, but not with grammatical tense). The setof possible values is thus extended with the NA (notapplicable) symbol. It allows the model to learn thata particular category is not a property of a word.

Lemmatisation Module The lemmatiser usesan approach similar to character-based word em-bedding estimation. A character embedding is con-catenated with the global feature vector and trans-formed by a linear layer. The lemmatiser takes a se-quence of such character representations and trans-

forms it using a dilated CNN. The softmax functionover the result produces the sequence of probabili-ties over a character vocabulary to form a lemma.

RO

OT

GLOBAL F

EATURES

ADJACENCYMATRIXDOT PRODUCT

HEAD

DEP

END

ENT

The

car

is red

ROOT

caris

red

The

Figure 4: Prediction of dependency arcs.

Parsing Module Two single FC layers transformglobal feature vectors into head and dependent em-beddings (see Figure 4). Based on these represen-tations, a dependency graph is defined as an adja-cency matrix with columns and rows correspondingto heads and dependents, respectively. The adja-cency matrix elements are dot products of all pairsof the head and dependent embeddings (the dotproduct determines the certainty of the edge be-tween two words). The softmax function applied toeach row of the matrix predicts the adjacent head-dependent pairs. This approach, however, does notguarantee that the resulting adjacency matrix isa properly built dependency tree. The Chu-Liu-Edmonds algorithm (Chu and Liu, 1965; Edmonds,1967) is thus applied in the last prediction step.

ROOT

carisred

The

HEAD

ADJACENCYMATRIX

. =

ROOT

carisred

The

DEPENDENT

root

nsubjcoproot

det

Figure 5: Prediction of grammatical functions.

53

The procedure of predicting words’ grammaticalfunctions (aka dependency labels) is shown in Fig-ure 5. A dependent and its head are represented asvectors by two single FC layers. The dependent em-bedding is concatenated with the weighted averageof (hypothetical) head embeddings. The weightsare the values from the corresponding row of the ad-jacency matrix, estimated by the arc predictionmodule. Concatenated vector representations arethen fed to a FC layer with the softmax activationfunction to predict dependency labels.

EUD Parsing Module Enhanced Universal De-pendencies (EUD) are predicted similarly to depen-dency trees. The EUD parsing module is describedin details in Klimaszewski and Wróblewska (2021).

3 COMBO Performance

Data COMBO is evaluated on treebanks fromthe Universal Dependencies repository (Zemanet al., 2019), preserving the original splits into train-ing, validation, and test sets. The treebanks repre-senting distinctive language types are summarisedin Table 4 in Appendix B.

By default, pre-trained 300-dimensional fastTextembeddings (Grave et al., 2018) are used. We alsotest encoding data with pre-trained contextual wordembeddings (the tested BERT models are listed inTable 5 in Appendix B). The UD datasets providegold-standard tokenisation. If BERT intra-tokenisersplits a word into sub-words, the last layer embed-dings are averaged to obtain a single vector repre-sentation of this word.

Qualitative Evaluation Table 1 shows COMBOresults of processing the selected UD treebanks.1

COMBO is compared with Stanza (Qi et al.,2020) and spaCy.2 The systems are evaluated withthe standard metrics (Zeman et al., 2018): F1, UAS

(unlabelled attachment score), LAS (labelled attach-ment score), MLAS (morphology-aware LAS) andBLEX (bi-lexical dependency score).3

COMBO and Stanza undeniably outrun spaCymodels. COMBO using non-contextualised word

1Check the prediction quality for other languages at:https://gitlab.clarin-pl.eu/syntactic-tools/combo/-/blob/master/docs/performance.md.

2https://spacy.io We use the project templatehttps://github.com/explosion/projects/tree/v3/pipelines/tagger_parser_ud. The lemmatiser isimplemented as a standalone pipeline component in spaCy v3and we do not test it.

3http://universaldependencies.org/conll18/conll18_ud_eval.py (CoNLL 2018 evaluation script).

embeddings is outperformed by Stanza in manylanguage scenarios. However, COMBO supportedwith BERT-like word embeddings beats all othersolutions and is currently the SOTA system formorphosyntactic analysis.

Regarding lemmatisation, Stanza has an advan-tage over COMBO in most tested languages. Thisis probably due to the fact that Stanza lemmatiseris enhanced with a key-value dictionary, whilstCOMBO lemmatiser is fully neural. It is not sur-prising that a dictionary helps in lemmatisation ofisolating languages (English). However, the dictio-nary approach is also helpful for agglutinative lan-guages (Finnish, Korean, Basque) and for Arabic,but not for Polish (fusional languages). Compar-ing COMBO models estimated with and withoutBERT embeddings, we note that BERT boost onlyslightly increases the quality of lemma predictionin the tested fusional and agglutinative languages.

For a complete insight into the prediction quality,we evaluate individual UPOS and UDEPREL predic-tions in English (the isolating language), Korean(agglutinative) and Polish (fusional). Result visual-isations are in Appendix C.

COMBO took part in IWPT 2021 Shared Taskon Parsing into Enhanced Universal Dependencies(Bouma et al., 2021), where it ranked 4th.4 In addi-tion to ELAS and EULAS metrics, the third evalua-tion metric was LAS. COMBO ranked 2nd, achiev-ing the average LAS of 87.84%. The score is evenhigher than the average LAS of 86.64% in Table 1,which is a kind of confirmation that our evaluationis representative, reliable, and fair.

Downstream Evaluation According to the re-sults in Table 1, COMBO predicts high-quality de-pendency trees and parts of speech. We thereforeconduct a preliminary evaluation of morphosyntac-tically informed word embeddings in the textualentailment task (aka natural language inference,NLI) in English (Bentivogli et al., 2016) and Pol-ish (Wróblewska and Krasnowska-Kieras, 2017).We compare the quality of entailment classifierswith two FC layers trained on max/mean-pooledBERT embeddings and sentence representations es-timated by a network with two transformer layerswhich is given morphosyntactically informed wordembeddings (i.e. BERT-based word embeddingsconcatenated with UPOS embeddings, DEPREL em-beddings, and BERT-based embeddings of the head

4https://universaldependencies.org/iwpt21/results.html

54

System UPOS XPOS UFeat Lemma UAS LAS CLAS MLAS BLEXEnglish EWT (isolating)

spaCy 93.79 93.10 94.89 NA 83.38 79.76 75.74 68.91 NAStanza 96.36 96.15 97.01 98.18 89.64 86.89 83.84 79.44 82.03COMBO 95.60 95.21 96.60 97.43 88.56 85.58 82.35 76.56 79.78COMBOBERT 96.57 96.44 97.24 97.86 91.76 89.28 86.83 81.71 84.38

Arabic PADT (fusional)spaCy 90.27 82.15 82.70 NA 74.24 67.28 63.28 50.48 NAStanza 96.98 93.97 94.08 95.26 87.96 83.74 80.57 74.96 76.80COMBO 96.71 93.72 93.83 93.54 87.06 82.70 79.46 73.25 73.64COMBOBERT 97.04 94.83 95.05 93.95 89.21 85.09 82.36 76.82 76.67

Polish PDB (fusional)spaCy 96.14 86.94 87.41 NA 86.73 82.06 79.00 65.42 NAStanza 98.47 94.20 94.42 97.43 93.15 90.84 88.73 81.98 85.75COMBO 98.24 94.26 94.53 97.47 92.87 90.45 88.07 81.31 85.53COMBOBERT 98.97 96.54 96.80 98.06 95.60 93.93 92.34 87.59 89.91

Finnish TDT (agglutinative)spaCy 92.15 93.34 87.89 NA 80.06 74.75 71.52 61.95 NAStanza 97.24 97.96 95.58 95.24 89.57 87.14 85.52 80.52 81.05COMBO 96.72 98.02 94.04 88.73 89.73 86.70 84.56 77.63 72.42COMBOBERT 98.29 99.00 97.30 89.48 94.11 92.52 91.34 87.18 77.84

Korean Kaist (agglutinative)spaCy 85.21 72.33 NA NA 76.15 68.13 61.98 57.52 NAStanza 95.45 86.31 NA 93.02 88.42 86.39 83.97 80.64 77.59COMBO 94.46 81.66 NA 89.16 87.31 85.12 82.70 78.38 72.79COMBOBERT 95.89 85.16 NA 89.95 89.77 87.83 85.96 82.66 75.89

Turkish IMST (agglutinative)spaCy 87.66 86.18 82.26 NA 60.43 51.32 47.74 37.28 NAStanza 95.98 95.18 93.77 96.73 74.14 67.52 64.03 58.13 61.91COMBO 93.60 92.36 88.88 96.47 72.00 64.48 60.48 49.88 58.75COMBOBERT 95.14 94.27 93.56 97.54 78.53 72.03 68.88 60.55 67.13

Basque BDT (agglutinative with fusional verb morphology)spaCy 91.96 NA 86.67 NA 76.11 70.28 66.96 54.46 NAStanza 96.23 NA 93.09 96.52 86.19 82.76 81.30 73.56 78.27COMBO 94.28 NA 90.44 95.47 84.64 80.44 78.82 67.33 74.95COMBOBERT 96.26 NA 93.84 96.38 88.73 85.80 84.93 75.96 81.25

Average scoresspaCy 91.03 85.67 86.97 NA 76.73 70.51 66.60 56.57 NAStanza 96.67 93.96 94.66 96.05 87.01 83.61 81.14 75.60 77.63COMBO 95.66 92.54 93.05 94.04 86.02 82.21 79.49 72.05 73.98COMBOBERT 96.88 94.37 95.63 94.75 89.67 86.64 84.66 78.92 79.01

Table 1: Processing quality (F1 scores) of spaCy, Stanza and COMBO on the selected UD treebanks (the languagetypes are given in parentheses). The highest scores are marked in bold.

Treebank spaCy Stanza COMBOTagger Lemmatiser Parser Total fastText BERT

English EWT 00:22:34 02:08:51 02:12:17 02:29:13 06:50:21 01:26:55 1:54:11Polish PDB 01:07:55 04:36:51 03:19:04 05:08:41 13:04:36 02:39:44 3:31:41

Table 2: Training time of spaCy, Stanza and COMBO.

word). The morphosyntactically informed EnglishNLI classifier achieves an accuracy of 78.84% andoutperforms the max/mean-pooled classifiers by

20.77 pp and 5.44 pp, respectively. The Polishsyntax-aware NLI classifier achieves an accuracyof 91.60% and outperforms the max/mean-pooled

55

classifiers by 17.2 pp and 7.7 pp, respectively.

Efficiency Evaluation We also compare spaCy,Stanza and COMBO in terms of their efficiency,i.e. training and prediction speed.5 According tothe results (see Tables 2 and 3), spaCy is the SOTAsystem, and the other two are not even close to itsprocessing time. Considering COMBO and Stanza,whose prediction quality is significantly better thanspaCy, COMBO is 1.5 times slower (2 times slowerwith BERT) than Stanza in predicting, but it isdefinitely faster in training. The reason for largediscrepancies in training times is the different archi-tecture of these two systems. Stanza is a pipeline-based system, i.e. its modules are trained one afterthe other. COMBO is an end-to-end system, i.e. itsmodules are jointly trained and the training processis therefore faster.

Treebank Stanza COMBO COMBOBERT

English EWT 4.7× 6.8× 10.8×Polish PDB 4.1× 5.8× 10.6×

Table 3: Prediction time of Stanza and COMBO rela-tive to spaCy (1×) on English and Polish test data.

4 Getting Started with COMBO

Prediction COMBO provides two main predic-tion modes: a Python library and a command-lineinterface (CLI). The Python package mode sup-ports automated model download. The code snip-pet demonstrates downloading a pre-trained Polishmodel and processing a sentence:

from combo.predict import COMBO

nlp = COMBO.from_pretrained("polish")

sentence = nlp("Ala ma kota.")

print(sentence.tokens)

To download a model for another language, se-lect its name from the list of pre-trained models.6

The Python mode also supports acquisition of DE-PREL or UPOS embeddings, for example:

sentence = nlp("Ala ma kota.")

chosen_token = sentence.tokens[1]

print(chosen_token.embeddings["upostag"])

5A single NVIDIA V100 card is used in all tests.6The list of the pretrained COMBO models: https:

//gitlab.clarin-pl.eu/syntactic-tools/combo/-/blob/master/docs/models.md#pre-trained-models

In CLI mode, COMBO processes sentences us-ing either a downloaded model or a model trainedby yourself. CLI works on raw texts and onthe CoNLL-U files (i.e. with tokenised sentencesand even morphologically annotated tokens):

combo --mode predict \

--model_path model.tar.gz \

--input_file input.conllu \

--output_file output.conllu

Model Training COMBO CLI allows to trainnew models for any language. The only require-ment is a training dataset in the CoNLL-U/CoNLL-X format. In the default setup, tokenised sentencesare input and all possible predictors are trained:

combo --mode train \

--training_data training.conllu \

--validation_data valid.conllu

If we only train a dependency parser, the defaultsetup should be changed with configuration flags:--features with a list of input features and--targets with a list of prediction targets.

5 Conclusion

We have presented COMBO, the SOTA systemfor morphosyntacic analysis, i.e. part-of-speechtagging, morphological analysis, lemmatisation,and (enhanced) dependency parsing. COMBO isa language-agnostic and format-independent sys-tem (i.e. it supports the CoNLL-U and CoNLL-X formats). Its implementation as a Python pack-age allows effortless installation, and incorpora-tion into any Python code or usage in the CLImode. In the Python mode, COMBO supports au-tomated download of pre-trained models for mul-tiple languages and outputs not only categoricalmorphosyntactic features, but also their embed-dings. In the CLI mode, pre-trained models canbe manually downloaded or trained from scratch.The system training is fully configurable in respectof the range of input features and output predic-tions, and the method of encoding input data.Last but not least, COMBO maintains a balancebetween efficiency and quality. Admittedly, it isnot as fast as spaCy, but it is much more efficientthan Stanza considering the training time. Tested onthe selected UD treebanks, COMBO morphosyn-tactic models enhanced with BERT embeddingsoutperform spaCy and Stanza models.

56

Acknowledgments

The authors would like to thank Piotr Rybak forhis design and explanations of the architecture ofCOMBO’s forerunner. The research presented inthis paper was founded by SONATA 8 grant no2014/15/D/HS2/03486 from the National ScienceCentre Poland and the European Regional Devel-opment Fund as a part of the 2014-2020 SmartGrowth Operational Programme, CLARIN – Com-mon Language Resources and Technology Infras-tructure, project no. POIR.04.02.00-00C002/19.The computing was performed at Poznan Super-computing and Networking Center.

ReferencesRodrigo Agerri, Iñaki San Vicente, Jon Ander Campos,

Ander Barrena, Xabier Saralegi, Aitor Soroa, andEneko Agirre. 2020. Give your text representationmodels some love: the case for Basque. In Proceed-ings of the 12th Language Resources and EvaluationConference, pages 4781–4788, Marseille, France.European Language Resources Association.

Wissam Antoun, Fady Baly, and Hazem Hajj. 2020.AraBERT: Transformer-based model for Arabic lan-guage understanding. In Proceedings of the 4thWorkshop on Open-Source Arabic Corpora and Pro-cessing Tools, with a Shared Task on Offensive Lan-guage Detection, pages 9–15, Marseille, France. Eu-ropean Language Resource Association.

Maria Jesus Aranzabe, Aitziber Atutxa, Kepa Ben-goetxea, Arantza Diaz de Ilarraza, Iakes Goenaga,Koldo Gojenola, and Larraitz Uria. 2015. AutomaticConversion of the Basque Dependency Treebank toUniversal Dependencies. In Proceedings of the 14thInternational Workshop on Treebanks and LinguisticTheories (TLT14), pages 233–241.

Luisa Bentivogli, Raffaella Bernardi, Marco Marelli,Stefano Menini, Marco Baroni, and Roberto Zam-parelli. 2016. SICK through the SemEval Glasses.Lesson learned from the evaluation of composi-tional distributional semantic models on full sen-tences through semantic relatedness and textual en-tailment. Journal of Language Resources and Eval-uation, 50:95–124.

Gosse Bouma, Djamé Seddah, and Daniel Zeman.2021. From Raw Text to Enhanced Universal De-pendencies: The Parsing Shared Task at IWPT 2021.In Proceedings of the 17th International Conferenceon Parsing Technologies and the IWPT 2021 SharedTask on Parsing into Enhanced Universal Dependen-cies (IWPT 2021), pages 146–157, Online. Associa-tion for Computational Linguistics.

Y. J. Chu and T. H. Liu. 1965. On the Shortest Arbores-cence of a Directed Graph. Science Sinica, 14:1396–1400.

Jayeol Chun, Na-Rae Han, Jena D. Hwang, andJinho D. Choi. 2018. Building Universal Depen-dency treebanks in Korean. In Proceedings of theEleventh International Conference on Language Re-sources and Evaluation (LREC 2018), Miyazaki,Japan. European Language Resources Association(ELRA).

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers),pages 4171–4186, Minneapolis, Minnesota. Associ-ation for Computational Linguistics.

Timothy Dozat and Christopher D. Manning. 2018.Simpler but more accurate semantic dependencyparsing. In Proceedings of the 56th Annual Meet-ing of the Association for Computational Linguis-tics (Volume 2: Short Papers), pages 484–490, Mel-bourne, Australia. Association for ComputationalLinguistics.

Jack Edmonds. 1967. Optimum Branchings. Journalof Research of the National Bureau of Standards,71B(4):233–240.

Yarin Gal and Zoubin Ghahramani. 2016. A theoret-ically grounded application of dropout in recurrentneural networks. In Proceedings of the 30th Interna-tional Conference on Neural Information ProcessingSystems, NIPS’16, page 1027–1035, Red Hook, NY,USA. Curran Associates Inc.

Matt Gardner, Joel Grus, Mark Neumann, OyvindTafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Pe-ters, Michael Schmitz, and Luke Zettlemoyer. 2018.AllenNLP: A deep semantic natural language pro-cessing platform. In Proceedings of Workshop forNLP Open Source Software (NLP-OSS), pages 1–6, Melbourne, Australia. Association for Computa-tional Linguistics.

Goran Glavaš and Ivan Vulic. 2021. Is supervisedsyntactic parsing beneficial for language understand-ing tasks? an empirical investigation. In Proceed-ings of the 16th Conference of the European Chap-ter of the Association for Computational Linguistics:Main Volume, pages 3090–3104, Online. Associa-tion for Computational Linguistics.

Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Ar-mand Joulin, and Tomas Mikolov. 2018. Learningword vectors for 157 languages. In Proceedings ofthe Eleventh International Conference on LanguageResources and Evaluation (LREC 2018), Miyazaki,Japan. European Language Resources Association(ELRA).

Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise Phoneme Classification with BidirectionalLSTM and Other Neural Network Architectures.Neural Networks, 18(5):602–610.

57

Jan Hajic, Otakar Smrž, Petr Zemánek, Petr Pajas,Jan Šnaidauf, Emanuel Beška, Jakub Kracmar, andKamila Hassanová. 2009. Prague arabic depen-dency treebank 1.0. LINDAT/CLARIAH-CZ dig-ital library at the Institute of Formal and AppliedLinguistics (ÚFAL), Faculty of Mathematics andPhysics, Charles University.

Katri Haverinen, Jenna Nyblom, Timo Viljanen,Veronika Laippala, Samuel Kohonen, Anna Missilä,Stina Ojala, Tapio Salakoski, and Filip Ginter. 2014.Building the essential resources for finnish: Theturku dependency treebank. Lang. Resour. Eval.,48(3):493–531.

Sepp Hochreiter and Jürgen Schmidhuber. 1997.Long short-term memory. Neural Computation,9(8):1735–1780.

Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy:Industrial-strength Natural Language Processing inPython.

Kiyoung Kim. 2020. Pretrained Language Models ForKorean. https://github.com/kiyoungkim1/LMkor.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: Amethod for stochastic optimization. In 3rd Inter-national Conference on Learning Representations,ICLR 2015, San Diego, CA, USA, May 7-9, 2015,Conference Track Proceedings.

Mateusz Klimaszewski and Alina Wróblewska. 2021.COMBO: A new module for EUD parsing. In Pro-ceedings of the 17th International Conference onParsing Technologies and the IWPT 2021 SharedTask on Parsing into Enhanced Universal Dependen-cies (IWPT 2021), pages 158–166, Online. Associa-tion for Computational Linguistics.

Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019.Open sesame: Getting inside BERT’s linguisticknowledge. In Proceedings of the 2019 ACL Work-shop BlackboxNLP: Analyzing and Interpreting Neu-ral Networks for NLP, pages 241–253, Florence,Italy. Association for Computational Linguistics.

Robert Mroczkowski, Piotr Rybak, Alina Wróblewska,and Ireneusz Gawlik. 2021. HerBERT: Efficientlypretrained transformer-based language model forPolish. In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, pages 1–10,Kiyv, Ukraine. Association for Computational Lin-guistics.

Vinod Nair and Geoffrey E. Hinton. 2010. Rectifiedlinear units improve restricted boltzmann machines.In Proceedings of the 27th International Conferenceon International Conference on Machine Learning,ICML’10, page 807–814, Madison, WI, USA. Om-nipress.

Adam Paszke, Sam Gross, Francisco Massa, AdamLerer, James Bradbury, Gregory Chanan, TrevorKilleen, Zeming Lin, Natalia Gimelshein, LucaAntiga, Alban Desmaison, Andreas Kopf, EdwardYang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang,Junjie Bai, and Soumith Chintala. 2019. PyTorch:An Imperative Style, High-Performance Deep Learn-ing Library. In H. Wallach, H. Larochelle,A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Gar-nett, editors, Advances in Neural Information Pro-cessing Systems 32, pages 8024–8035. Curran Asso-ciates, Inc.

Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton,and Christopher D. Manning. 2020. Stanza: Apython natural language processing toolkit for manyhuman languages. In Proceedings of the 58th An-nual Meeting of the Association for ComputationalLinguistics: System Demonstrations, pages 101–108,Online. Association for Computational Linguistics.

Piotr Rybak and Alina Wróblewska. 2018. Semi-Supervised Neural System for Tagging, Parsing andLematization. In Proceedings of the CoNLL 2018Shared Task: Multilingual Parsing from Raw Text toUniversal Dependencies, pages 45–54.

Stefan Schweter. 2020. BERTurk - BERT models forTurkish.

Natalia Silveira, Timothy Dozat, Marie-Catherinede Marneffe, Samuel Bowman, Miriam Connor,John Bauer, and Chris Manning. 2014. A gold stan-dard dependency corpus for English. In Proceedingsof the Ninth International Conference on LanguageResources and Evaluation (LREC’14), pages 2897–2904, Reykjavik, Iceland. European Language Re-sources Association (ELRA).

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky,Ilya Sutskever, and Ruslan Salakhutdinov. 2014.Dropout: A Simple Way to Prevent Neural Networksfrom Overfitting. Journal of Machine Learning Re-search, 15:1929–1958.

Milan Straka. 2018. UDPipe 2.0 prototype at CoNLL2018 UD shared task. In Proceedings of the CoNLL2018 Shared Task: Multilingual Parsing from RawText to Universal Dependencies, pages 197–207,Brussels, Belgium. Association for ComputationalLinguistics.

Emma Strubell, Patrick Verga, David Belanger, andAndrew McCallum. 2017. Fast and Accurate En-tity Recognition with Iterated Dilated Convolutions.In Proceedings of the 2017 Conference on Empiri-cal Methods in Natural Language Processing, pages2670–2680, Copenhagen, Denmark. Association forComputational Linguistics.

Umut Sulubacak, Memduh Gokirmak, Francis Tyers,Çagrı Çöltekin, Joakim Nivre, and Gülsen Eryigit.2016. Universal Dependencies for Turkish. In Pro-ceedings of COLING 2016, the 26th International

58

Conference on Computational Linguistics: Techni-cal Papers, pages 3444–3454, Osaka, Japan. TheCOLING 2016 Organizing Committee.

Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang,Adam Poliak, R Thomas McCoy, Najoung Kim,Benjamin Van Durme, Sam Bowman, Dipanjan Das,and Ellie Pavlick. 2019. What do you learn fromcontext? probing for sentence structure in contextu-alized word representations. In International Con-ference on Learning Representations.

Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma,Juhani Luotolahti, Tapio Salakoski, Filip Ginter, andSampo Pyysalo. 2019. Multilingual is not enough:BERT for Finnish.

Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Ha-gen Blix, Yining Nie, Anna Alsop, Shikha Bordia,Haokun Liu, Alicia Parrish, Sheng-Fu Wang, JasonPhang, Anhad Mohananey, Phu Mon Htut, PalomaJeretic, and Samuel R. Bowman. 2019. Investi-gating BERT’s knowledge of language: Five anal-ysis methods with NPIs. In Proceedings of the2019 Conference on Empirical Methods in Natu-ral Language Processing and the 9th InternationalJoint Conference on Natural Language Processing(EMNLP-IJCNLP), pages 2877–2887, Hong Kong,China. Association for Computational Linguistics.

Alina Wróblewska. 2018. Extended and Enhanced Pol-ish Dependency Bank in Universal DependenciesFormat. In Proceedings of the Second Workshop onUniversal Dependencies (UDW 2018), pages 173–182.

Alina Wróblewska and Katarzyna Krasnowska-Kieras.2017. Polish evaluation dataset for compositionaldistributional semantics models. In Proceedingsof the 55th Annual Meeting of the Association forComputational Linguistics (Volume 1: Long Papers),pages 784–792, Vancouver, Canada. Association forComputational Linguistics.

Fisher Yu and Vladlen Koltun. 2016. Multi-Scale Con-text Aggregation by Dilated Convolutions. CoRR,abs/1511.07122.

Daniel Zeman, Jan Hajic, Martin Popel, Martin Pot-thast, Milan Straka, Filip Ginter, Joakim Nivre, andSlav Petrov. 2018. CoNLL 2018 shared task: Mul-tilingual parsing from raw text to universal depen-dencies. In Proceedings of the CoNLL 2018 SharedTask: Multilingual Parsing from Raw Text to Univer-sal Dependencies, pages 1–21, Brussels, Belgium.Association for Computational Linguistics.

Daniel Zeman, Joakim Nivre, Mitchell Abrams, NoëmiAepli, Željko Agic, Lars Ahrenberg, Gabriele Alek-sandraviciute, Lene Antonsen, Katya Aplonova,Maria Jesus Aranzabe, Gashaw Arutie, MasayukiAsahara, Luma Ateyah, Mohammed Attia, Aitz-iber Atutxa, Liesbeth Augustinus, Elena Badmaeva,Miguel Ballesteros, Esha Banerjee, Sebastian Bank,Verginica Barbu Mititelu, Victoria Basmov, Colin

Batchelor, John Bauer, Sandra Bellato, Kepa Ben-goetxea, Yevgeni Berzak, Irshad Ahmad Bhat,Riyaz Ahmad Bhat, Erica Biagetti, Eckhard Bick,Agne Bielinskiene, Rogier Blokland, Victoria Bo-bicev, Loïc Boizou, Emanuel Borges Völker, CarlBörstell, Cristina Bosco, Gosse Bouma, Sam Bow-man, Adriane Boyd, Kristina Brokaite, AljoschaBurchardt, Marie Candito, Bernard Caron, GauthierCaron, Tatiana Cavalcanti, Gülsen Cebiroglu Ery-igit, Flavio Massimiliano Cecchini, Giuseppe G. A.Celano, Slavomír Céplö, Savas Cetin, Fabri-cio Chalub, Jinho Choi, Yongseok Cho, JayeolChun, Alessandra T. Cignarella, Silvie Cinková,Aurélie Collomb, Çagrı Çöltekin, Miriam Con-nor, Marine Courtin, Elizabeth Davidson, Marie-Catherine de Marneffe, Valeria de Paiva, Elvisde Souza, Arantza Diaz de Ilarraza, Carly Dicker-son, Bamba Dione, Peter Dirix, Kaja Dobrovoljc,Timothy Dozat, Kira Droganova, Puneet Dwivedi,Hanne Eckhoff, Marhaba Eli, Ali Elkahky, BinyamEphrem, Olga Erina, Tomaž Erjavec, Aline Eti-enne, Wograine Evelyn, Richárd Farkas, HectorFernandez Alcalde, Jennifer Foster, Cláudia Fre-itas, Kazunori Fujita, Katarína Gajdošová, DanielGalbraith, Marcos Garcia, Moa Gärdenfors, Se-bastian Garza, Kim Gerdes, Filip Ginter, IakesGoenaga, Koldo Gojenola, Memduh Gökırmak,Yoav Goldberg, Xavier Gómez Guinovart, BertaGonzález Saavedra, Bernadeta Griciute, Matias Gri-oni, Normunds Gruzıtis, Bruno Guillaume, CélineGuillot-Barbance, Nizar Habash, Jan Hajic, Jan Ha-jic jr., Mika Hämäläinen, Linh Hà My, Na-RaeHan, Kim Harris, Dag Haug, Johannes Heinecke, Fe-lix Hennig, Barbora Hladká, Jaroslava Hlavácová,Florinel Hociung, Petter Hohle, Jena Hwang,Takumi Ikeda, Radu Ion, Elena Irimia, O. lájídéIshola, Tomáš Jelínek, Anders Johannsen, FredrikJørgensen, Markus Juutinen, Hüner Kasıkara, An-dre Kaasen, Nadezhda Kabaeva, Sylvain Kahane,Hiroshi Kanayama, Jenna Kanerva, Boris Katz,Tolga Kayadelen, Jessica Kenney, Václava Ket-tnerová, Jesse Kirchner, Elena Klementieva, ArneKöhn, Kamil Kopacewicz, Natalia Kotsyba, JolantaKovalevskaite, Simon Krek, Sookyoung Kwak,Veronika Laippala, Lorenzo Lambertino, LuciaLam, Tatiana Lando, Septina Dian Larasati, AlexeiLavrentiev, John Lee, Phuong Lê Hông, AlessandroLenci, Saran Lertpradit, Herman Leung, Cheuk YingLi, Josie Li, Keying Li, KyungTae Lim, Maria Li-ovina, Yuan Li, Nikola Ljubešic, Olga Loginova,Olga Lyashevskaya, Teresa Lynn, Vivien Macke-tanz, Aibek Makazhanov, Michael Mandl, Christo-pher Manning, Ruli Manurung, Catalina Maran-duc, David Marecek, Katrin Marheinecke, Héc-tor Martínez Alonso, André Martins, Jan Mašek,Yuji Matsumoto, Ryan McDonald, Sarah McGuin-ness, Gustavo Mendonça, Niko Miekka, Mar-garita Misirpashayeva, Anna Missilä, Catalin Mi-titelu, Maria Mitrofan, Yusuke Miyao, SimonettaMontemagni, Amir More, Laura Moreno Romero,Keiko Sophie Mori, Tomohiko Morioka, Shin-suke Mori, Shigeki Moro, Bjartur Mortensen,Bohdan Moskalevskyi, Kadri Muischnek, Robert

59

Munro, Yugo Murawaki, Kaili Müürisep, PinkeyNainwani, Juan Ignacio Navarro Horñiacek, AnnaNedoluzhko, Gunta Nešpore-Berzkalne, LuongNguyên Thi., Huyên Nguyên Thi. Minh, Yoshi-hiro Nikaido, Vitaly Nikolaev, Rattima Nitisaroj,Hanna Nurmi, Stina Ojala, Atul Kr. Ojha, Adédayò.Olúòkun, Mai Omura, Petya Osenova, RobertÖstling, Lilja Øvrelid, Niko Partanen, Elena Pas-cual, Marco Passarotti, Agnieszka Patejuk, Guil-herme Paulino-Passos, Angelika Peljak-Łapinska,Siyao Peng, Cenel-Augusto Perez, Guy Perrier,Daria Petrova, Slav Petrov, Jason Phelan, JussiPiitulainen, Tommi A Pirinen, Emily Pitler, Bar-bara Plank, Thierry Poibeau, Larisa Ponomareva,Martin Popel, Lauma Pretkalnin, a, Sophie Prévost,Prokopis Prokopidis, Adam Przepiórkowski, TiinaPuolakainen, Sampo Pyysalo, Peng Qi, AndrielaRääbis, Alexandre Rademaker, Loganathan Ra-masamy, Taraka Rama, Carlos Ramisch, Vinit Rav-ishankar, Livy Real, Siva Reddy, Georg Rehm, IvanRiabov, Michael Rießler, Erika Rimkute, Larissa Ri-naldi, Laura Rituma, Luisa Rocha, Mykhailo Ro-manenko, Rudolf Rosa, Davide Rovati, ValentinRos, ca, Olga Rudina, Jack Rueter, Shoval Sadde,Benoît Sagot, Shadi Saleh, Alessio Salomoni, TanjaSamardžic, Stephanie Samson, Manuela Sanguinetti,Dage Särg, Baiba Saulıte, Yanin Sawanakunanon,Nathan Schneider, Sebastian Schuster, Djamé Sed-dah, Wolfgang Seeker, Mojgan Seraji, Mo Shen,Atsuko Shimada, Hiroyuki Shirasu, Muh Shohibus-sirri, Dmitry Sichinava, Aline Silveira, Natalia Sil-veira, Maria Simi, Radu Simionescu, Katalin Simkó,Mária Šimková, Kiril Simov, Aaron Smith, IsabelaSoares-Bastos, Carolyn Spadine, Antonio Stella,Milan Straka, Jana Strnadová, Alane Suhr, UmutSulubacak, Shingo Suzuki, Zsolt Szántó, DimaTaji, Yuta Takahashi, Fabio Tamburini, TakaakiTanaka, Isabelle Tellier, Guillaume Thomas, Li-isi Torga, Trond Trosterud, Anna Trukhina, ReutTsarfaty, Francis Tyers, Sumire Uematsu, ZdenkaUrešová, Larraitz Uria, Hans Uszkoreit, AndriusUtka, Sowmya Vajjala, Daniel van Niekerk, Gert-jan van Noord, Viktor Varga, Eric Villemonte de laClergerie, Veronika Vincze, Lars Wallin, AbigailWalsh, Jing Xian Wang, Jonathan North Washing-ton, Maximilan Wendt, Seyi Williams, Mats Wirén,Christian Wittern, Tsegay Woldemariam, Tak-sumWong, Alina Wróblewska, Mary Yako, Naoki Ya-mazaki, Chunxiao Yan, Koichi Yasuoka, Marat M.Yavrumyan, Zhuoran Yu, Zdenek Žabokrtský, AmirZeldes, Manying Zhang, and Hanzhi Zhu. 2019.Universal dependencies 2.5. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Ap-plied Linguistics (ÚFAL), Faculty of Mathematicsand Physics, Charles University.

60

A COMBO Implementation

COMBO is a Python package that uses the Py-Torch (Paszke et al., 2019) and AllenNLP (Gardneret al., 2018) libraries. The COMBO models usedin the evaluation presented in Section 3 are trainedwith the empirically set default parameters speci-fied below. The training parameters can be easilyconfigured and adjusted to the specifics of an indi-vidual model.

A.1 Network Hyperparameters

Embeddings An internal character-based wordembedding is calculated with three convolutionallayers with 512, 256 and 64 filters with dilationrates equal to 1, 2 and 4. All filters have the kernelsize of 3. The internal word embedding has a sizeof 64 dimensions. All external word embeddingsare reduced to 100-dimensional vectors by a singleFC layer. As only words are used as input featuresin the system evaluation, the local feature embed-ding is a concatenation of the 64-dimensional inter-nal and 100-dimensional external word embedding.The global feature vectors are computed by twobiLSTM layers with 512 hidden units.

Prediction modules The tagger uses a FC net-work with a hidden layer of the size 64 to predictUPOS and FC networks with 128-dimensional hid-den layers to predict XPOS and UFEATS.The lemmatiser uses three convolutional layerswith 256 filters and dilation rates equal to 1, 2 and4. All filters have the kernel size of 3. The fourthconvolutional layer with the number of filters equalto the number of character instances in training datais used to predict the probability of each character.The final layer filters have the kernel size of 1. The256-dimensional embeddings of input charactersare concatenated with the global feature vectorsreduced to 32 dimensions with a single FC layer.The arc prediction module uses 512-dimensionalhead, and dependent embeddings and the labellingmodule uses 128-dimensional vectors.

COMBO-vectoriser currently outputs 64-dimensional UPOS and 128-dimensional DEPREL

embeddings.

Activation function FC and CNN layers use hy-perbolic tangent and rectified linear unit (Nair andHinton, 2010) activation functions, respectively.

A.2 RegularisationDropout technique for Variational RNNs (Galand Ghahramani, 2016) with 0.33 rate is appliedto the local feature embeddings and on top ofthe stacked biLSTM estimating global feature em-beddings. The same dropout, for output and re-current values, is used in the context of each biL-STM layer. The FC layers use the standard dropout(Srivastava et al., 2014) with 0.25 rate. Moreover,the biLSTM and convolutional layers use L2 regu-larisation with the rate of 1×10−6, and the trainableembeddings use L2 with the rate of 1× 10−5.

A.3 TrainingThe cross-entropy loss is used for all parts ofthe system. The final loss is the weighted sum oflosses with the following weights for each task:

• 0.05 for predicting UPOS and LEMMA,• 0.2 for predicting UFEATS and (enh)HEAD,• 0.8 for predicting (enh)DEPREL.

The whole system is optimised with ADAM(Kingma and Ba, 2015) with the learning rate of0.002 and β1 = β2 = 0.9. The model is trainedfor a maximum of 400 epochs, and the learningrate is reduced twice by the factor of two whenthe validation score reaches a plateau.

B External Data Summary

Tables 4 and 5 list the UD dependency treebanksand BERT models used in the evaluation experi-ments presented in Section 3.

C Evaluation of UPOS and UDEPREL

The comparison of the universal parts of speechpredicted by the tested systems in English, Koreanand Polish data is shown in the charts in Figures 6,7 and 8, respectively. The comparison of the qual-ity of the predicted universal dependency types inEnglish, Korean and Polish data is presented inFigures 9, 10 and 11, respectively.

61

Language Language Type UD Treebank #Words #Trees ReferenceEnglish isolating English-EWT 254,856 16,622 Silveira et al. (2014)Arabic fusional Arabic-PADT 282,384 7,664 Hajic et al. (2009)Polish fusional Polish-PDB 350,036 22,152 Wróblewska (2018)Finnish agglutinative Finnish-TDT 202,453 15,136 Haverinen et al. (2014)Korean agglutinative Korean-Kaist 350,090 27,363 Chun et al. (2018)Turkish agglutinative Turkish-IMST 57,859 5,635 Sulubacak et al. (2016)Basque agglutinative (fusional Basque-BDT 121,443 8,993 Aranzabe et al. (2015)

verb morphology)

Table 4: The UD treebanks used in the evaluation experiments.

Language BERT model ReferenceArabic bert-base-arabertv2 Antoun et al. (2020)Basque berteus-base-cased Agerri et al. (2020)English bert-base-cased Devlin et al. (2019)Finnish bert-base-finnish-cased-v1 Virtanen et al. (2019)Korean bert-kor-base Kim (2020)Polish herbert-base-cased Mroczkowski et al. (2021)Turkish bert-base-turkish-cased Schweter (2020)

Table 5: The BERT models used in the evaluation experiments.

ADJADP

ADVAUX

CCONJDET

INTJ

NOUNNUM

PART

PRON

PROPN

PUNCT

SCONJSYM

VERB X0

20

40

60

80

100spaCy stanza COMBOBERT

Figure 6: Evaluation of predicted universal parts of speech (UPOS) in the English test set (F-1-scores).

ADJADP

ADVAUX

CCONJDET

INTJ

NOUNNUM

PRON

PROPN

PUNCT

SCONJSYM

VERB X0

20

40

60

80

100spaCy stanza COMBOBERT

Figure 7: Evaluation of predicted universal parts of speech (UPOS) in the Korean test set (F-1-scores).

62

ADJADP

ADVAUX

CCONJDET

INTJ

NOUNNUM

PART

PRON

PROPN

PUNCT

SCONJSYM

VERB X0

20

40

60

80

100spaCy stanza COMBOBERT

Figure 8: Evaluation of predicted universal parts of speech (UPOS) in the Polish test set (F-1-scores).

aclad

vcl

advm

odam

odap

pos

auxca

se cccc

omp

compo

undco

nj copcsu

bj det

disco

urseex

plfixe

d flat

goesw

ith iobj

listmark

nmodns

ubj

nummod ob

job

l

parat

axispu

nct

repara

ndum roo

t

voca

tive

xcom

p0

20

40

60

80

100spaCy stanza COMBOBERT

Figure 9: Evaluation of predicted grammatical functions (UDEPREL) in the English test set (F-1-scores).

aclad

vcl

advm

odam

odap

pos

aux

case cc

ccom

p

compo

und

conj co

pcsu

bj dep de

t

disco

urse

disloc

atedfixe

d flat iobjmark

nmod

nsub

j

nummod ob

job

lpu

nct

root

xcom

p0

20

40

60

80

100spaCy stanza COMBOBERT

Figure 10: Evaluation of predicted grammatical functions (UDEPREL) in the Korean test set (F-1-scores).

aclad

vcl

advm

odam

odap

pos

aux

case cc

ccom

pco

nj copcsu

bj det

disco

urseex

plfixe

d flat iobj

listmark

nmod

nsub

j

nummod ob

job

l

orpha

n

parat

axispu

nct

root

voca

tive

xcom

p0

20

40

60

80

100spaCy stanza COMBOBERT

Figure 11: Evaluation of predicted grammatical functions (UDEPREL) in the Polish test set (F-1-scores).