Proceedings of the BioNLP 2020 workshop, pages 167–176Online, July 9, 2020 c©2020 Association for Computational Linguistics

167

Benchmark and Best Practices for Biomedical Knowledge GraphEmbeddings

David Chang1, Ivana Balažević2, Carl Allen2, Daniel Chawla1Cynthia Brandt1, Richard Andrew Taylor1

1Yale Center for Medical Informatics, Yale University2School of Informatics, University of Edinburgh, UK{david.chang, richard.taylor}@yale.edu{ivana.balazevic, carl.allen}@ed.ac.uk

Abstract

Much of biomedical and healthcare data isencoded in discrete, symbolic form such astext and medical codes. There is a wealthof expert-curated biomedical domain knowl-edge stored in knowledge bases and ontolo-gies, but the lack of reliable methods for learn-ing knowledge representation has limited theirusefulness in machine learning applications.While text-based representation learning hassignificantly improved in recent years throughadvances in natural language processing, at-tempts to learn biomedical concept embed-dings so far have been lacking. A recent fam-ily of models called knowledge graph embed-dings have shown promising results on generaldomain knowledge graphs, and we exploretheir capabilities in the biomedical domain.We train several state-of-the-art knowledgegraph embedding models on the SNOMED-CT knowledge graph, provide a benchmarkwith comparison to existing methods and in-depth discussion on best practices, and makea case for the importance of leveraging themulti-relational nature of knowledge graphsfor learning biomedical knowledge represen-tation. The embeddings, code, and materialswill be made available to the community1.

1 Introduction

A vast amount of biomedical domain knowledge isstored in knowledge bases and ontologies. For ex-ample, SNOMED Clinical Terms (SNOMED-CT)2

is the most widely used clinical terminology in theworld for documentation and reporting in health-care, containing hundreds of thousands of medicalterms and their relations, organized in a polyhierar-chical structure. SNOMED-CT can be thought ofas a knowledge graph: a collection of triples con-sisting of a head entity, a relation, and a tail entity,denoted (h, r, t). SNOMED-CT is one of over a

1https://github.com/dchang56/snomed kge2https://www.nlm.nih.gov/healthit/snomedct

hundred terminologies under the Unified MedicalLanguage System (UMLS) (Bodenreider, 2004),which provides a metathesaurus that combines mil-lions of biomedical concepts and relations undera common ontological framework. The uniqueidentifiers assigned to the concepts as well as theResource Release Format (RRF) standard enableinteroperability and reliable access to information.The UMLS and the terminologies it encompassesare a crucial resource for biomedical and healthcareresearch.

One of the main obstacles in clinical and biomed-ical natural language processing (NLP) is the abil-ity to effectively represent and incorporate domainknowledge. A wide range of downstream applica-tions such as entity linking, summarization, patient-level modeling, and knowledge-grounded languagemodels could all benefit from improvements in ourability to represent domain knowledge. While re-cent advances in NLP have dramatically improvedtextual representation (Alsentzer et al., 2019), at-tempts to learn analogous dense vector represen-tations for biomedical concepts in a terminologyor knowledge graph (concept embeddings) so farhave several drawbacks that limit their usabilityand wide-spread adoption. Further, there is cur-rently no established best practice or benchmarkfor training and comparing such embeddings. Inthis paper, we explore knowledge graph embedding(KGE) models as alternatives to existing methodsand make the following contributions:

• We train five recent KGE models onSNOMED-CT and demonstrate their advan-tages over previous methods, making a casefor the importance of leveraging the multi-relational nature of knowledge graphs forbiomedical knowledge representation.

• We establish a suite of benchmark tasks toenable fair comparison across methods andinclude much-needed discussion on best prac-

168

tices for working with biomedical knowledgegraphs.

• We also serve the general KGE community byproviding benchmarks on a new dataset withreal-world relevance.

• We make the embeddings, code, and other ma-terials publicly available and outline severalavenues of future work to facilitate progressin the field.

2 Related Work and Background

2.1 Biomedical concept embeddingsEarly attempts to learn biomedical concept embed-dings have applied variants of the skip-gram model(Mikolov et al., 2013) on large biomedical or clini-cal corpora. Med2Vec (Choi et al., 2016) learnedembeddings for 27k ICD-9 codes by incorporatingtemporal and co-occurrence information from pa-tient visits. Cui2Vec (Beam et al., 2019) used anextremely large collection of multimodal medicaldata to train embeddings for nearly 109k conceptsunder the UMLS.

These corpus-based methods have several draw-backs. First, the corpora are inaccessible due todata use agreements, rendering them irreproducible.Second, these methods tend to be data-hungry andextremely data inefficient for capturing domainknowledge. In fact, one of the main limitationsof language models in general is their reliance onthe distributional hypothesis, essentially makinguse of mostly co-occurrence level information inthe training corpus (Peters et al., 2019). Third,they do a poor job of achieving sufficient conceptcoverage: Cui2Vec, despite its enormous trainingdata, was only able to capture 109k concepts out ofover 3 million concepts in the UMLS, drasticallylimiting its downstream usability.

A more recent trend has been to apply networkembedding (NE) methods directly on a knowledgegraph that represents structured domain knowl-edge. NE methods such as Node2Vec (Grover andLeskovec, 2016) learn embeddings for nodes in anetwork (graph) by applying a variant of the skip-gram model on samples generated using randomwalks, and they have shown impressive results onnode classification and link prediction tasks on awide range of network datasets. In the biomedi-cal domain, CANode2Vec (Kotitsas et al., 2019)applied several NE methods on single-relation sub-sets of the SNOMED-CT graph, but the lack of

comparison to existing methods and the disregardfor the heterogeneous structure of the knowledgegraph substantially limit its significance.

Notably, Snomed2Vec (Agarwal et al., 2019)applied NE methods on a clinically relevant multi-relational subset of the SNOMED-CT graph andprovided comparisons to previous methods todemonstrate that applying NE methods directly onthe graph is more data efficient, yields better em-beddings, and gives explicit control over the subsetof concepts to train on. However, one major limita-tion of NE approaches is that they relegate relation-ships to mere indicators of connectivity, discardingthe semantically rich information encoded in multi-relational, heterogeneous knowledge graphs.

We posit that applying KGE methods on a knowl-edge graph is more principled and should there-fore yield better results. We now provide a briefoverview of the KGE literature and describe ourexperiments in Section 3.

2.2 Knowledge Graph Embeddings

Knowledge graphs are collections of facts in theform of ordered triples (h, r, t), where entity his related to entity t by relation r. Because knowl-edge graphs are often incomplete, an ability to inferunknown facts is a fundamental task (link predic-tion). A series of recent KGE models approach linkprediction by learning embeddings of entities andrelations based on a scoring function that predictsa probability that a given triple is a fact.

RESCAL (Nickel et al., 2011) represents rela-tions as a bilinear product between subject andobject entity vectors. Although a very expressivemodel, RESCAL is prone to overfitting due to thelarge number of parameters in the full rank relationmatrix, increasing quadratically with the numberof relations in the graph.

DistMult (Yang et al., 2015) is a special caseof RESCAL with a diagonal matrix per relation,reducing overfitting. However, by limiting lineartransformations on entity embeddings to a stretch,DistMult cannot model asymmetric relations.

ComplEx (Trouillon et al., 2016) extends Dist-Mult to the complex domain, enabling it to modelasymmetric relations by introducing complex con-jugate operations into the scoring function.

SimplE (Kazemi and Poole, 2018) modifiesCanonical Polyadic (CP) decomposition (Hitch-cock, 1927) to allow two embeddings for each en-tity (head and tail) to be learned dependently.

169

A recent model TuckER (Balažević et al., 2019)is shown to be a fully expressive, linear modelthat subsumes several tensor factorization basedapproaches including all models described above.

TransE (Bordes et al., 2013) is an example ofan alternative translational family of KGE models,which regard a relation as a translation (vector off-set) from the subject to the object entity vectors.Translational models have an additive componentin the scoring function, in contrast to the multiplica-tive scoring functions of bilinear models.

RotatE (Sun et al., 2019) extends the notion oftranslation to rotation in the complex plane, en-abling the modeling of symmetry/antisymmetry,inversion, and composition patterns in knowledgegraph relations.

We restrict our experiments to five models dueto their available implementation under a common,scalable platform (Zhu et al., 2019): TransE, Com-plEx, DistMult, SimplE, and RotatE.

3 Experimental Setup

3.1 DataGiven the complexity of the UMLS, we detail ourpreprocessing steps to generate the final dataset.We subset the 2019AB version of the UMLS toSNOMED_CT_US terminology, taking all activeconcepts and relations in the MRCONSO.RRF andMRREL.RRF files. We extract semantic type in-formation from MRSTY.RRF and semantic groupinformation from the Semantic Network website3

to filter concepts and relations to 8 broad semanticgroups of interest: Anatomy (ANAT), Chemicals& Drugs (CHEM), Concepts & Ideas (CONC),Devices (DEVI), Disorders (DISO), Phenomena(PHEN), Physiology (PHYS), and Procedures(PROC). We also exclude specific semantic typesdeemed unnecessary. A full list of the semantictypes included in the dataset and their broader se-mantic groups can be found in the Supplements.

The resulting list of triples comprises our fi-nal knowledge graph dataset. Note that theUMLS includes reciprocal relations (ISA andINVERSE_ISA), making the graph bidirectional.A random split results in train-to-test leakage,which can inflate the performance of weaker mod-els (Dettmers et al., 2018). We fix this by ensuringreciprocal relations are in the same split, not acrosssplits. Descriptive statistics of the final dataset areshown in Table 1. After splitting, we also ensure

3https://semanticnetwork.nlm.nih.gov

there are no unseen entities or relations in the vali-dation and test sets by simply moving them to thetrain set. More details and the code used for datapreparation are included in the Supplements.

Descriptions StatisticsEntities 293,884Relation types 170Facts 2,073,848- Train 1,965,032- Valid / Test 48,936 / 49,788

Table 1: Statistics of the final SNOMED dataset.

3.2 ImplementationConsidering the non-trivial size of SNOMED-CTand the importance of scalability and consistentimplementation for running experiments, we useGraphVite (Zhu et al., 2019) for the KGE mod-els. GraphVite is a graph embedding frameworkthat emphasizes scalability, and its speedup relativeto existing implementations is well-documented4.While the backend is written largely in C++, aPython interface allows customization. We makeour customized Python code available. We use thefive models available in GraphVite in our experi-ments: TransE, ComplEx, DistMult, SimplE, andRotatE. While we restrict our current work to thesemodels, future work should also consider otherstate-of-the-art models such as TuckER (Balaževićet al., 2019) and MuRP (Balažević et al., 2019),especially since MuRP is shown to be particularlyeffective for graphs with hierarchical structure. Pre-trained embeddings for Cui2Vec and Snomed2Vecwere used as provided by the authors, with dimen-sionality 500 and 200, respectively.

All experiments were run on 3 GTX-1080tiGPUs, and final runs took ∼6 hours on a sin-gle GPU. Hyperparameters were either tuned onthe validation set for each model: margin (4,6, 8, 10) and learning_rate (5e-4, 1e-4, 5e-5, 1e-5); set: num_negative (60), dim (512),num_epoch (2000); or took default values fromGraphVite. The final hyperparameter configurationcan be found in the Appendix.

3.3 Evaluation and Benchmark3.3.1 KGE Link PredictionA standard evaluation task in the KGE literatureis link prediction. However, NE methods also use

4https://github.com/DeepGraphLearning/graphVite

170

link prediction as a standard evaluation task. Whileboth predict whether two nodes are connected, NElink prediction performs binary classification on abalanced set of positive and negative edges basedon the assumption that the graph is complete. Incontrast, knowledge graphs are typically assumedincomplete, making link prediction for KGE aranking-based task in which the model’s scoringfunction is used to rank candidate samples withoutrelying on ground truth negatives. In this paper,link prediction refers to the latter ranking-basedKGE method.

Candidate samples are generated for each triplein the test set using all possible entities as the targetentity, where the target can be set to head, tail,or both. For example, if the target is tail, themodel predicts scores for all possible candidatesfor the tail entity in (h, r, ?). For a test set with50k triples and 300k possible unique entities, themodel calculates scores for fifteen billion candi-date triples. The candidates are filtered to excludetriples seen in the train, validation, and test sets,so that known triples do not affect the ranking andcause false negatives. Several ranking-based met-rics are computed based on the sorted scores. Notethat SNOMED-CT contains a transitive closurefile, which lists explicit transitive closures for thehierarchical relations ISA and INVERSE_ISA (ifA ISA B, and B ISA C, then the transitive closureincludes A ISA C). This file should be included inthe file list used to filter candidates to best enablethe model to learn hierarchical structure.

Typical link prediction metrics include MeanRank (MR), Mean Reciprocal Rank (MRR), andHits@k (H@k). MR is considered to be sensitiveto outliers and unreliable as a metric. Guu et al.proposed using Mean Quantile (MQ) as a more ro-bust alternative to MR and MRR. We use MQ100 asa more challenging version of MQ that introduces acut-off at the top 100th ranking, appropriate for thelarge numbers of possible entities. Link predictionresults are reported in Table 2.

3.3.2 Embedding EvaluationFor fair comparison with existing methods, we per-form some of the benchmark tasks for assessingmedical concept embeddings proposed by Beamet al.. However, we discuss their methodologicalflaws in Section 5 and suggest more appropriateevaluation methods.

Since non-KGE methods are not directly com-parable on tasks that require both relation and con-

cept embeddings, to compare embeddings acrossmethods we perform entity semantic classification,which requires only concept embeddings.

We generate a dataset for entity classification bytaking the intersection of the concepts covered inall (7) models, comprising 39k concepts with 32unique semantic types and 4 semantic groups. Wesplit the data into train and test sets with 9:1 ratio,and train a simple linear layer with 0.1 dropout andno further hyperparameter tuning. The single linearlayer for classification assesses the linear separabil-ity of semantic information in the entity embeddingspace for each model. Results for semantic typeand group classification are reported in Table 3.

4 Visualization

We first discuss the embedding visualizations ob-tained through LargeVis (Tang et al., 2016), anefficient large-scale dimensionality reduction tech-nique available as an application in GraphVite.

Figure 1 shows concept embeddings for RotatE,ComplEx, Snomed2Vec, and Cui2Vec, with colorscorresponding to broad semantic groups. Cui2Vecembeddings show structure but not coherent seman-tic clusters. Snomed2Vec shows tighter groupingsof entities, though the clusters are patchy and scat-tered across the embedding space. ComplEx pro-duces globular clusters centered around the origin,with clearer boundaries between groups. RotatEgives visibly distinct clusters with clear group sep-aration that appear intuitive: entities of the Physi-ology semantic group (black) overlap heavily withthose of Disorders (magenta); also entities underthe Concepts semantic group (red) are relativelyscattered, perhaps due to their abstract nature, com-pared to more concrete entities like Devices (cyan),Anatomy (blue), and Chemicals (green), whichform tighter clusters.

Interestingly, the embedding visualizations forthe 5 KGE models fall into 2 types: RotatE andTransE produce well-separated clusters while Com-plEx, DistMult and SimplE produce globular clus-ters around the origin. Since the plots for eachtype appear almost indistinguishable we show onefrom each (RotatE and ComplEx). We attribute thecharacteristic difference between the two modeltypes to the nature of their scoring functions: Ro-tatE and TransE have an additive component whileComplEx, DistMult and SimplE are multiplicative.

Figure 2 shows more fine-grained semantic struc-ture by coloring 5 selected semantic types under

171

Figure 1: Concept embedding visualization (RotatE, ComplEx, Snomed2Vec, Cui2Vec) by semantic group.

the Procedures semantic group and greying outthe rest. We see that RotatE produces subclustersthat are also intuitive. Laboratory procedures arewell-separated on their own, health care activityand educational activity overlap significantly, anddiagnostic procedures and therapeutic or preven-tative procedures overlap significantly. ComplExalso reveals subclusters with globular shape, andSnomed2Vec captures laboratory procedures wellbut leaves other types scattered. These observa-tions are consistent across other semantic groups.We include similar visualizations for the Chemicals& Drugs semantic group in the Supplements.

While semantic class information is not theonly significant aspect of SNOMED-CT, since theSNOMED-CT graph is largely organized aroundsemantic group and type information, it is promis-ing that embeddings learned (without supervision)preserve it.

5 Results

5.1 Link PredictionTable 2 shows results for the link prediction taskfor the 5 KGE models on SNOMED-CT. Having

Model MRR MQ100 H@1 H@10TransE .346 .739 .212 .597ComplEx .461 .761 .360 .652DistMult .420 .752 .309 .626SimplE .432 .735 .337 .615RotatE .317 .742 .162 .599TransEFB .294 - - .465TransEWN .226 - - .501RotatEFB .338 - .241 .533RotatEWN .476 - .428 .571

Table 2: Link prediction results: for the 5 KGE modelson SNOMED-CT (top); and for TransE and RotatE ontwo standard KGE datasets (Sun et al., 2019) (bottom).

no previous results to compare to, we include per-formance of TransE and RotatE on two standardKGE benchmark datasets for reference: FB15k-237(14,541 entities, 237 relations, and 310,116 triples)and WN18RR (40,943 entities, 11 relations, and93,003 triples). Given that SNOMED-CT is largerand arguably a more complex knowledge graphthan the two datasets, the link prediction resultssuggest that the KGE models learn a reasonable

172

Figure 2: Visualization of selected semantic types under the Procedures semantic group for RotatE, ComplEx, andSnomed2Vec. Semantic types with more than 2,000 entities were subsampled to 1,200 for visibility. Cui2Vec (notshown) was similar to Snomed2Vec but more dispersed.

representation of SNOMED-CT. We include sam-ple model outputs for the top 10 entity scores forlink prediction in the Supplements.

5.2 Embedding Evaluation and RelationPrediction

Test set accuracy for entity semantic type (STY)and semantic group (SG) classification are reportedin Table 3. In accordance with the visualiza-tions of semantic clusters (Figures 1 and 2), theKGE and NE methods perform significantly bet-ter than the corpus-based method (Cui2Vec). No-tably, TransE and RotatE attain near-perfect accu-racy for the broader semantic group classification(4 classes). ComplEx, DistMult, and SimplE per-form slighty worse, Snomed2Vec slightly belowthem, and Cui2Vec falls behind by a significantmargin. We see a greater discrepancy in relativeperformance by model type in semantic type clas-sification (32 classes), in which more fine-grainedsemantic information is required.

Two advantages of the semantic type and groupentity classification tasks are: (i) information isprovided by the UMLS, making the task non-proprietary and standardized; (ii) it readily showswhether a model preserves the semantic structure ofthe ontology, an important aspect of the data. Thetasks can also easily be modified for custom dataand specific domains, e.g. class labels for genesand proteins relevant to a particular biomedical ap-plication can be used in classification to assess howwell the model captures relevant domain-specificinformation.

For comparison to related work, we also exam-ine the benchmark tasks to assess medical con-cept embeddings based on statistical power and

cosine similarity bootstrapping, proposed by Beamet al.. For a given known relationship pair (e.g.x cause_of y), a null distribution of pairwisecosine similarity scores is computed by bootstrap-ping 10,000 samples of the same semantic cate-gory as x and y respectively. The cosine similarityof the observed sample is compared to the 95thpercentile of the bootstrap distribution (statisticalsignificance at the 0.05 level). The authors claimthat, when applied to a collection of known relation-ships (causative, comorbidity, etc), the procedureestimates the fraction of true relationships discov-ered given a tolerance for some false positive rate.Following this, we report the statistical power ofall 7 models for two of the tasks: semantic typeand causative relationships. The former (ST) aimsto assess a model’s ability to determine if two con-cepts share the same semantic type. The latterconsists of two relation types: cause_of (Co)and causative_agent_of (CA). Results arereported in Table 3. The cosine similarity boot-strap results, particularly for the causative relation-ship tasks, illustrate a major flaw in the protocol.While Snomed2Vec and Cui2Vec attain similarstatistical powers for CA and Co, we see largediscrepancies between the two tasks for the KGEmodels, especially for ComplEx, DistMult, andSimplE, which produce globular embedding clus-ters. Examining the dataset, we observe that thecause_of relations occur mostly between con-cepts within the same semantic group/cluster (e.g.Disorder), whereas the causative_agent_ofrelations occur between concepts in different se-mantic groups/clusters (e.g. Chemicals to Disor-ders). The large discrepancy in CA task results

173

Entity Classification Cosine-Sim Bootstrap Relation PredictionModel SG (4) STY (32) ST CA Co MRR H@1 H@10Snomed2Vec .944 .769 .387 .903 .894 - - -Cui2Vec .891 .673 .416 .584 .559 - - -TransE .993 .827 .579 .765 .978 .800 .727 .965ComplEx .956 .786 .249 .001 .921 .731 .606 .914DistMult .971 .794 .275 .014 .971 .734 .569 .946SimplE .953 .768 .242 .011 .791 .854 .803 .946RotatE .995 .829 .544 .242 .943 .849 .799 .957

Table 3: Results for (i) entity classification of semantic type and group (test accuracy); (ii) selected tasks from(Beam et al., 2019); and (iii) relation prediction. Best results in bold.

for the KGE models is because using cosine simi-larity embeds the assumption that all related enti-ties are close, regardless of the relation type. Theassumption that cosine similarity in the conceptembedding space is an appropriate measure of adiverse range of relatedness (a much broader ab-straction that subsumes semantic similarity andcausality), renders this evaluation protocol unsuit-able for assessing a model’s ability to capture spe-cific types of relational information in the embed-dings. Essentially, all that can be said about thecosine similarity-based procedure is that it assesseshow close entities are in that space as measuredby cosine distance. It does not reveal the nature oftheir relationship or what kind of relational infor-mation is encoded in the space to begin with.

In contrast, KGE methods explicitly model re-lations and are better equipped to make inferencesabout the relational structure of the knowledgegraph embeddings. Thus, we propose relation pre-diction as a standard evaluation task for assessinga model’s ability to capture information about re-lations in the knowledge graph. We simply mod-ify the link prediction task described above to ac-commodate relation as a target (formulated as(h, ?, t), generating ranking-based metrics for themodel’s ability to prioritize the correct relation typegiven a pair of concepts. This provides a more prin-cipled and interpretable way to evaluate the models’relation representations directly based on the modelprediction. The last 3 columns of Table 3 reportrelation prediction metrics for the 5 KGE models.In particular, RotatE and SimplE perform well, at-taining around 0.8 Hits@1 and around 0.85 MRR.

We conduct error analysis to gain further insightby categorizing relation types into 6 groups basedon the cardinality and homogeneity of their sourceand target semantic groups. If the set of unique

head or tail entities for a relation type in the datasetbelongs to only one semantic group, then it has acardinality of 1, and a cardinality of many other-wise. If the mapping of the source semantic groupsto the target semantic groups are one-to-one (e.g.DISO to DISO and CHEM to CHEM), then it isconsidered homogeneous. We report relation pre-diction metrics for each of the 6 groups of relationtypes for RotatE and ComplEx in Table 4.

We see that RotatE gives impressive rela-tion prediction performance for all groups ex-cept for many-to-many-homogeneous, a seem-ingly challenging group of relations contain-ing ambiguous and synonymous relation types,e.g. possibly_equivalent_to, same_as,refers_to, isa. The full list of M-M-hom re-lations are shown in the Appendix. In contrast,ComplEx struggles with a wider array of relationtypes, suggesting that it is generally less able tomodel different types than RotatE. The last tworows under each model show per-relation results forthe causative relationships mentioned previously:cause_of and causative_agent_of. Ro-tatE again shows significantly better results com-pared to ComplEx, in line with its theoreticallysuperior representation capacity (Sun et al., 2019).

6 Discussion

Based on our findings, we recommend the use ofKGE models to leverage the multi-relational na-ture of knowledge graphs for learning biomedicalconcept and relation embeddings; and of appropri-ate evaluation tasks such as link prediction, entityclassification and relation prediction for fair com-parison across models. We also encourage analysisbeyond standard validation metrics, e.g. visual-ization, examining model predictions, reportingmetrics for different relation groupings and devis-

174

Relation MRR H@1 H@10 CountComplEx

1-1-hom .600 .319 .944 72M-M-hom .605 .417 .877 29,028

M-1 .683 .557 .884 2,5091-M .738 .640 .916 2,4971-1 .889 .817 .995 420

M-M .867 .819 .941 15,044Co .706 .662 .779 145CA .857 .822 .908 303

RotatEM-M-hom .784 .718 .934 29,028

M-M .973 .944 .992 15,044M-1 .971 .945 .998 2,5091-M .975 .953 .998 2,4971-1 .985 .959 1. 420

1-1-hom .972 .976 1. 72Co .803 .738 .890 145CA .996 .993 1. 303

Table 4: Relation prediction results for RotatE andComplEx by category of relation type (last two rowsrelate to causative relation types).

ing problem or domain-specific validation tasks. Afurther promising evaluation task is the triple pre-diction proposed in (Allen et al., 2019), which weleave for future work. A more ideal way to assessconcept embeddings in biomedical NLP applica-tions and patient-level modeling would be to designa suite of benchmark downstream tasks that incor-porate the embeddings, but that warrants a rigorouspaper of its own and is left for future work.

We believe this paper serves the biomedical NLPcommunity as an introduction to KGEs and theirevaluation and analyses, and also the KGE commu-nity by providing a potential standard benchmarkdataset with real-world relevance.

7 Conclusion and Future Work

We present results from applying 5 leading KGEmodels to the SNOMED-CT knowledge graph andcompare them to related work through visualiza-tions and evaluation tasks, making a case for theimportance of using models that leverage the multi-relation nature of knowledge graphs for learningbiomedical knowledge representation. We discussbest practices for working with biomedical knowl-edge graphs and evaluating the embeddings learnedfrom them, proposing link prediction, entity classi-fication, and relation prediction as standard evalua-

tion tasks. We encourage researchers to engage infurther validation through visualizations, error anal-yses based on model predictions, examining strat-ified metrics, and devising domain-specific tasksthat can assess the usefulness of the embeddingsfor a given application domain.

There are several immediate avenues of futurework. While we focus on the SNOMED-CT datasetand the KGE models implemented in GraphVite,other biomedical terminologies such as the GeneOntology (The Gene Ontology Consortium, 2018)and RxNorm (Nelson et al., 2011) could be ex-plored and more recent KGE models, e.g. TuckER(Balažević et al., 2019) and MuRP (Balažević et al.,2019), applied. Additional sources of informationcould also potentially be incorporated, such as tex-tual descriptions of entities and relations. In prelim-inary experiments, we initialized entity and relationembeddings with the embeddings of their textualdescriptors extracted using Clinical Bert (Alsentzeret al., 2019), but it did not yield gains. This maysuggest that the concept and language spaces aresubstantially different and strategies to jointly trainwith linguistic and knowledge graph informationrequire further study. Other sources of informationinclude entity types (e.g. UMLS semantic type) andpaths, or multi-hop generalizations of the 1-hoprelations (triples) typically used in KGE models(Guu et al., 2015). Notably, CoKE trains contex-tual knowledge graph embeddings using path-levelinformation under an adapted version of the BERTtraining paradigm (Wang et al., 2019).

Lastly, the usefulness of biomedical knowledgegraph embeddings should be investigated in down-stream applications in biomedical NLP such asinformation extraction, concept normalization andentity linking, computational fact checking, ques-tion answering, summarization, and patient trajec-tory modeling. In particular, entity linkers act asa bottleneck between text and concept spaces, andleveraging KGEs could help develop sophisticatedtools to parse existing biomedical and clinical textdatasets for concept-level annotations and addi-tional insights. Well performing entity linkers maythen enable training knowledge-grounded large-scale language models like KnowBert (Peters et al.,2019). Overall, methods for learning and incorpo-rating domain-specific knowledge representationare still at an early stage and further discussionsare needed.

175

ReferencesKhushbu Agarwal, Tome Eftimov, Raghavendra Ad-

danki, Sutanay Choudhury, Suzanne Tamang, andRobert Rallo. 2019. Snomed2Vec: Random Walkand Poincare Embeddings of a Clinical KnowledgeBase for Healthcare Analytics. arXiv:1907.08650[cs, stat]. ArXiv: 1907.08650.

Carl Allen, Ivana Balazevic, and Timothy M.Hospedales. 2019. On Understanding KnowledgeGraph Representation. arXiv:1909.11611 [cs, stat].

Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and MatthewMcDermott. 2019. Publicly available clinical BERTembeddings. In Proceedings of the 2nd ClinicalNatural Language Processing Workshop, pages 72–78, Minneapolis, Minnesota, USA. Association forComputational Linguistics.

Ivana Balažević, Carl Allen, and Timothy MHospedales. 2019. Tucker: Tensor factorization forknowledge graph completion. In Empirical Methodsin Natural Language Processing.

Ivana Balažević, Carl Allen, and Timothy Hospedales.2019. Multi-relational poincar\’e graph embed-dings. In Advances in Neural Information Process-ing Systems.

Andrew L. Beam, Benjamin Kompa, Allen Schmaltz,Inbar Fried, Griffin Weber, Nathan P. Palmer, Xu Shi,Tianxi Cai, and Isaac S. Kohane. 2019. Clinical Con-cept Embeddings Learned from Massive Sources ofMultimodal Medical Data. arXiv:1804.01486 [cs,stat]. ArXiv: 1804.01486.

Olivier Bodenreider. 2004. The Unified Med-ical Language System (UMLS): integratingbiomedical terminology. Nucleic Acids Research,32(90001):267D–270.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko.2013. Translating embeddings for modeling multi-relational data. In C. J. C. Burges, L. Bottou,M. Welling, Z. Ghahramani, and K. Q. Weinberger,editors, Advances in Neural Information ProcessingSystems 26, pages 2787–2795. Curran Associates,Inc.

Edward Choi, Mohammad Taha Bahadori, Eliza-beth Searles, Catherine Coffey, Michael Thompson,James Bost, Javier Tejedor-Sojo, and Jimeng Sun.2016. Multi-layer Representation Learning for Med-ical Concepts. In Proceedings of the 22nd ACMSIGKDD International Conference on KnowledgeDiscovery and Data Mining - KDD ’16, pages 1495–1504, San Francisco, California, USA. ACM Press.

Tim Dettmers, Minervini Pasquale, Stenetorp Pon-tus, and Sebastian Riedel. 2018. Convolutional 2dknowledge graph embeddings. In Proceedings ofthe 32th AAAI Conference on Artificial Intelligence,pages 1811–1818.

Aditya Grover and Jure Leskovec. 2016. node2vec:Scalable feature learning for networks. In Proceed-ings of the 22nd ACM SIGKDD International Con-ference on Knowledge Discovery and Data Mining.

Kelvin Guu, John Miller, and Percy Liang. 2015.Traversing knowledge graphs in vector space. InProceedings of the 2015 Conference on EmpiricalMethods in Natural Language Processing, pages318–327, Lisbon, Portugal. Association for Compu-tational Linguistics.

Frank L. Hitchcock. 1927. The expression of a tensoror a polyadic as a sum of products. Journal of Math-ematics and Physics, 6(1-4):164–189.

Seyed Kazemi and David Poole. 2018. Simple embed-ding for link prediction in knowledge graphs. InAdvances in Neural Information Processing Systems32.

Sotiris Kotitsas, Dimitris Pappas, Ion Androutsopoulos,Ryan McDonald, and Marianna Apidianaki. 2019.Embedding Biomedical Ontologies by Jointly En-coding Network Structure and Textual Node De-scriptors. In Proceedings of the 18th BioNLP Work-shop and Shared Task, pages 298–308, Florence,Italy. Association for Computational Linguistics.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor-rado, and Jeff Dean. 2013. Distributed representa-tions of words and phrases and their composition-ality. In C. J. C. Burges, L. Bottou, M. Welling,Z. Ghahramani, and K. Q. Weinberger, editors, Ad-vances in Neural Information Processing Systems26, pages 3111–3119. Curran Associates, Inc.

Stuart Nelson, Kelly Zeng, John Kilbourne, TammyPowell, and Robin Moore. 2011. Normalized namesfor clinical drugs: RxNorm at 6 years. JAMIA,18(4):441–448.

Maximilian Nickel, Volker Tresp, and Hans-Peter Krie-gal. 2011. A three-way model for collective learn-ing on multi-relational data. In Proceedings of the28th International Conference on Machine Learn-ing, pages 809–816.

Matthew E. Peters, Mark Neumann, Robert L Lo-gan, Roy Schwartz, Vidur Joshi, Sameer Singh, andNoah A. Smith. 2019. Knowledge enhanced contex-tual word representations. In EMNLP.

Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and JianTang. 2019. Rotate: Knowledge graph embeddingby relational rotation in complex space. In Interna-tional Conference on Learning Representations.

Jian Tang, Jingzhou Liu, Ming Zhang, and QiaozhuMei. 2016. Visualizing large-scale and high-dimensional data. In Proceedings of the 25th Inter-national Conference on World Wide Web, pages 287–297. International World Wide Web ConferencesSteering Committee.

http://arxiv.org/abs/1907.08650http://arxiv.org/abs/1907.08650http://arxiv.org/abs/1907.08650http://arxiv.org/abs/1909.11611http://arxiv.org/abs/1909.11611https://doi.org/10.18653/v1/W19-1909https://doi.org/10.18653/v1/W19-1909http://arxiv.org/abs/1804.01486http://arxiv.org/abs/1804.01486http://arxiv.org/abs/1804.01486https://doi.org/10.1093/nar/gkh061https://doi.org/10.1093/nar/gkh061https://doi.org/10.1093/nar/gkh061http://papers.nips.cc/paper/5071-translating-embeddings-for-modeling-multi-relational-data.pdfhttp://papers.nips.cc/paper/5071-translating-embeddings-for-modeling-multi-relational-data.pdfhttps://doi.org/10.1145/2939672.2939823https://doi.org/10.1145/2939672.2939823https://arxiv.org/abs/1707.01476https://arxiv.org/abs/1707.01476https://doi.org/10.18653/v1/D15-1038https://doi.org/10.1002/sapm192761164https://doi.org/10.1002/sapm192761164https://doi.org/10.18653/v1/W19-5032https://doi.org/10.18653/v1/W19-5032https://doi.org/10.18653/v1/W19-5032http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdfhttp://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdfhttp://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdfhttps://doi.org/10.1136/amiajnl-2011-000116https://doi.org/10.1136/amiajnl-2011-000116https://openreview.net/forum?id=HkgEQnRqYQhttps://openreview.net/forum?id=HkgEQnRqYQ

176

The Gene Ontology Consortium. 2018. The Gene On-tology Resource: 20 years and still GOing strong.Nucleic Acids Research, 47(D1):D330–D338.

Théo Trouillon, Johannes Welbl, Sebastian Riedel, ÉricGaussier, and Guillaume Bouchard. 2016. Complexembeddings for simple link prediction. In Inter-national Conference on Machine Learning (ICML),volume 48, pages 2071–2080.

Quan Wang, Pingping Huang, Haifeng Wang, SongtaiDai, Wenbin Jiang, Jing Liu, Yajuan Lyu, and HuaWu. 2019. Coke: Contextualized knowledge graphembedding. arXiv:1911.02168.

Bishan Yang, Scott Wen-tau Yih, Xiaodong He, Jian-feng Gao, and Li Deng. 2015. Embedding Enti-ties and Relations for Learning and Inference inKnowledge Bases. In Proceedings of the Inter-national Conference on Learning Representations(ICLR) 2015.

Zhaocheng Zhu, Shizhen Xu, Meng Qu, and Jian Tang.2019. Graphvite: A high-performance cpu-gpu hy-brid system for node embedding. In The World WideWeb Conference, pages 2494–2504. ACM.

https://doi.org/10.1093/nar/gky1055https://doi.org/10.1093/nar/gky1055https://www.microsoft.com/en-us/research/publication/embedding-entities-and-relations-for-learning-and-inference-in-knowledge-bases/https://www.microsoft.com/en-us/research/publication/embedding-entities-and-relations-for-learning-and-inference-in-knowledge-bases/https://www.microsoft.com/en-us/research/publication/embedding-entities-and-relations-for-learning-and-inference-in-knowledge-bases/