Design Challenges for Entity Linking - GitHub Pagesxiaoling.github.io/pubs/ling-tacl15-slides.pdf · 2018-05-18 · Design Challenges for Entity Linking Xiao Ling, Sameer Singh, Daniel

Design Challenges for Entity Linking

Xiao Ling, Sameer Singh, Daniel S. Weld

Entity Linking

2

Seattle beat Portland yesterday.

Entity Linking

3


Entity Linking

4


Seattle (city)

Seattle Sounders

Sea-Tac(airport)

Entity Linking

5


Seattle (city)

Seattle Sounders

Sea-Tac(airport) ~3-4 M

entries

Applications• Relation Extraction

(e.g. Koch et al. 2014)

• Coreference Resolution (e.g. Hajishirzi et al. 2013, Durrett & Klein 2014)

• Question Answering (e.g. Sun et al. 2015)

• Web Search (e.g. Knowledge Graph)

• many others… (see Shen et al. 2014; Roth et al. 2014)

6

Ambiguity

• Seattle beat Portland yesterday.

• Seattle scores high in the latest report of startup hubs.

• The Emerald City Council To Make Decision on Antibiotic Resolution

7

Ambiguity




Seattle Sounders

8

Ambiguity




Seattle Sounders

Seattle (city)

9

Variability• Seattle scores high in the latest report of startup

hubs.


10

Variability• Seattle scores high in the latest report of startup

hubs.


Seattle (city)

11

Related Work• Cucerzan (2007)

• Milne and Witten (2008)

• Kulkarni et al. (2009)

• Ratinov et al. (2011)

• Hoffart et al. (2011)

• Han and Sun (2012)

• He et al. (2013a)12

• He et al. (2013b)

• Cheng and Roth (2013)

• Sil and Yates (2013)

• Li et al. (2013)

• Cornolti et al. (2013)

• … many others







• He et al. (2013a)13

• He et al. (2013b)



• Li et al. (2013)


• … many others

Joint Inference







• He et al. (2013a)14

• He et al. (2013b)



• Li et al. (2013)


• … many others

Joint Inference

Learning to rank







• He et al. (2013a)15

• He et al. (2013b)



• Li et al. (2013)


• … many others

Joint Inference

Deep Neural Networks

Learning to rank

Popular Data Sets

16

Dataset

# of Mentions Knowledge Base

UIUCACE 244 Wikipedia

MSNBC 654 WikipediaAIDA

(Hoffart et al. 2011)

AIDA-D 5917 YagoAIDA-T 5616 Yago

TAC KBP

TAC09 3904 Wikipedia 2008TAC10 2250 Wikipedia 2008TAC10T 1500 Wikipedia 2008TAC11 2250 Wikipedia 2008TAC12 2226 Wikipedia 2008

Unfortunately…

17

ACE MSNBC AIDA-D AIDA-T KBP09 KBP10 KBP10T KBP11 KBP12

Cucerzan (2007) ⎷Milne & Witten (2008)Kulkarni et al. (2009) ⎷Ratinov et al. (2011) ⎷ ⎷Hoffart et al. (2011) ⎷Han & Sun (2012) ⎷He et al. (2013a) ⎷ ⎷He et al. (2013b) ⎷ ⎷

Cheng & Roth (2013) ⎷ ⎷ ⎷Sil & Yates (2013) ⎷ ⎷ ⎷

Li et al. (2013) ⎷ ⎷Cornolti et al. (2013) ⎷ ⎷TAC-KBP participants ⎷ ⎷ ⎷ ⎷ ⎷

Unfortunately…

18





Joint Inference

Deep Neural Networks

Learning to rank

Metonymy

19





… Moscow ’s as yet undisclosed

proposals …

Moscow (city)

Russia (country)

Government of Russia

Nested Entities

20





… Florida Green Party …

Green Party of the US

Green Party of Florida

Contributions• Vinculum: a simple, deterministic, modular EL sys.

• comprehensive evaluation over nine data sets

• candidate conditional prob. can work quite well

• entity types are important to the final performance

• comparable results with two state-of-the-art sys.

21

Agenda

• Introduction

• Vinculum

• Experiments

• Conclusion

22

Entity Type

Candidate Generation

Coreference

Coherence

Mention Extraction

Vinculum ArchitectureSeattle beat Portland yesterday.

23

Input:

Entity Type


Coreference

Coherence

Mention Extraction

Mention ExtractionSeattle beat Portland yesterday.

24

Mention Extraction

Candidate GenerationSeattle beat Portland yesterday.

25

Candidate Entities

- Seattle (city)

- Seattle Sounders

- Seattle-Tacoma (airport)


Mention Extraction

Conditional probability

26

… capital of the state of Washington .

In 1990, Washington starred as Bleek Gilliam …

Washington refused to run for a third term …

… Washington …

p(e | m) = # [m -> e]

# m

Entity Type


Coreference

Coherence

Mention Extraction

27

… capital of the state of Washington .

In 1990, Washington starred as Bleek Gilliam …

Washington refused to run for a third term …

… Washington …

p( | “Washington”) = # “W” ->

# “W”

Entity Type


Coreference

Coherence

Mention Extraction

Conditional probability


28

Candidate Entities

- Seattle (city)

- Seattle Sounders



Mention Extraction


29

Candidate Entities

- Seattle (city)

- Seattle Sounders


0.6

0.2

0.1


Mention Extraction

Entity TypesSeattle beat Portland yesterday.

30

Candidate Entities

- Seattle (city)

- Seattle Sounders


0.6

0.2

0.1

Entity Type Prediction• city • sports_team • facility/airport

0.10.4

0.1

Entity Type


Mention Extraction


31

Candidate Entities

- Seattle (city)

- Seattle Sounders


0.6

0.2

0.1


0.10.4

0.1

p(e | m) = ∑t p(e,t | m) p(e | m) = ∑t p(e | t,m) p(t | m)

Entity Type


Coreference

Coherence

Mention Extraction


32

Candidate Entities

- Seattle (city)

- Seattle Sounders


0.6

0.2

0.1


p(e | t,m) : re-normalization of cond. prob.

Entity Type


Coreference

Coherence

Mention Extraction


33

Candidate Entities

- Seattle (city)

- Seattle Sounders


0.6

0.2

0.1


p(e | t,m) : re-normalization of cond. prob. e.g. t = LOC p(Seattle-city | LOC, “Seattle”) = 0.6 / 0.7 p(Sea-Tac | LOC, “Seattle”) = 0.1 / 0.7

Entity Type


Coreference

Coherence

Mention Extraction


34

Candidate Entities

- Seattle (city)

- Seattle Sounders


0.6

0.2

0.1


0.10.4

0.1

p(e | m) = ∑t p(e,t | m) p(e | m) = ∑t p(e,t | m) p(t | m)

p(t | m)

Entity Type


Coreference

Coherence

Mention Extraction


35

Entity Type Prediction

• city • sports_team • facility/airport

0.10.4

0.1

Entity Type


Coreference

Coherence

Mention Extraction

Candidate Entities

- Seattle (city)

- Seattle Sounders


0.6

0.2

0.1

Candidate Generation Candidate Entities

- Seattle (city)

- Seattle Sounders


0.2

0.4

0.1

Entity Type


36

Entity Type Prediction

• city • sports_team • facility/airport

0.10.4

0.1

Entity Type


Coreference

Coherence

Mention Extraction

Candidate Entities

- Seattle (city)

- Seattle Sounders


0.6

0.2

0.1

Candidate Generation Candidate Entities

- Seattle Sounders

- Seattle (city)


0.2

0.4

0.1

Entity Type

Coreference

37

Seattle Sounders head coach Sigi Schmid has some ideas … Seattle beat Portland yesterday.

Entity Type


Coreference

Mention Extraction

Coherence

38

Candidate Entities

- Seattle (city)

- Seattle Sounders


Candidate Entities

- Portland, OR

- Univ. of Portland

- Portland Timbers


0.2

0.4

0.1

0.2

0.2

0.1

Entity Type


Coreference

Coherence

Mention Extraction

Normalized Google Distance (NGD)

39

(Milne & Witten, 2008)Entity Type


Coreference

Coherence

Mention Extraction


40

(Milne & Witten, 2008)

George Washington

Denzel Washington

President of the US

US Constitution

American Revolutionary

War

Tony Award

Golden Globe Training Day

Entity Type


Coreference

Coherence

Mention Extraction


41


George Washington

Denzel Washington

President of the US

US Constitution


War

Tony Award

Golden Globe Training Day

Entity Type


Coreference

Coherence

Mention Extraction


42


George Washington John Adams

President of the US

US Constitution


WarQuasi-WarPresident

of the USUS

Constitution

Entity Type


Coreference

Coherence

Mention Extraction

Relational Score

• Relation triples from Freebase

• A binary score =

• 1, if two entities appear in a triple

• 0, otherwise

• E.g. (Barack Obama, birthplace, United States) => r (Barack Obama, United States) = 1

43

(Cheng & Roth, 2013)Entity Type


Coreference

Coherence

Mention Extraction

Context

44

Entity Type


Coreference

Coherence

Mention Extraction

Agenda

• Introduction

• Vinculum

• Experiments

• Conclusion

45

ACE MSNBC AIDA-D AIDA-T KBP09 KBP10

KBP10T

KBP11

KBP12Cucerzan (2007) ⎷

Milne & Witten (2008)Kulkarni et al. (2009) ⎷

Ratinov et al. (2011) ⎷ ⎷

Hoffart et al. (2011) ⎷

Han & Sun (2012) ⎷

He et al. (2013a) ⎷ ⎷

He et al. (2013b) ⎷ ⎷

Cheng & Roth (2013) ⎷ ⎷ ⎷

Sil & Yates (2013) ⎷ ⎷ ⎷

Li et al. (2013) ⎷ ⎷

Cornolti et al. (2013) ⎷ ⎷

TAC-KBP participants ⎷ ⎷ ⎷ ⎷ ⎷

Data Sets

46

Dataset # of Mentions Knowledge BaseACE 244 Wikipedia

MSNBC 654 WikipediaAIDA-D 5917 YagoAIDA-T 5616 YagoTAC09 3904 Wikipedia 2008TAC10 2250 Wikipedia 2008TAC10T 1500 Wikipedia 2008TAC11 2250 Wikipedia 2008TAC12 2226 Wikipedia 2008

Mention based F1

Official Eval.


• intra-Wikipedia

• CrossWikis(Spitkovsky & Chang, 2012)

• Freebase Search API

47

Entity Type


Coreference

Coherence

Mention ExtractionConditional Probability p(e | m) e.g. p( | “Washington”)

Entity Type


Coreference

Coherence

Mention Extraction

Aggregate Recall

48

1 3 5 10 20 30 50 100 Inf0.5

0.6

0.7

0.8

0.9

1

k

Recall@k

CrossWikisIntra−WikipediaFreebase Search

49

• Coarse-grained NER(Stanford NER)

• Fine-grained Entity Types (Ling & Weld, 2012) Entity Type


Coreference

Coherence

Mention Extraction

Effect of Entity Types

p(e | m) = ∑t p(e,t | m) p(e | m) = ∑t p(e,t | m) p(t | m)

Entity Type Probability

FIGER

• 112 entity types

• multi-label multi-class

50

(Ling & Weld, 2012)

Entity Type


Coreference

Coherence

Mention Extraction

Entity Type


Coreference

Coherence

Mention Extraction

Predicted Entity Types

51

F1

50

62.5

75

87.5

100

ACE MSNBC AIDA-D AIDA-T TAC09 TAC10 TAC10T TAC11 TAC12 Overall+NER +FIGER

Overall Performance

52

F1

50

62.5

75

87.5

100

ACE MSNBC AIDA-D AIDA-T TAC09 TAC10 TAC10T TAC11 TAC12Candidate Generation +Entity Types +Coref. +Coherence

Entity Type


Coreference

Coherence

Mention Extraction

Overall Performance

53

F1

50

62.5

75

87.5

100

AverageCand +Entity Type +Coref +Coherence

79.078.076.775.0

Entity Type


Coreference

Coherence

Mention Extraction

Overall Performance

• AIDA (Hoffart et al. 2011)

• Illinois Wikifier 2.0 (Cheng & Roth, 2013)

54

F1

50

62.5

75

87.5

100

AverageCand +Entity Type +Coref +Coherence AIDA Wikifier

79.6

72.279.078.076.775.0

Entity Type


Coreference

Coherence

Mention Extraction

Error Analysis

55

Misc 10%

Specific Labels 14%

Context 33%

Coreference 10%

Types 14%

Metonymy 19%

Conclusion• a modular deterministic system achieves good performance

56

Vinculum

http://github.com/xiaoling/vinculum

Conclusion• a modular deterministic system achieves good performance• a comprehensive evaluation over nine data sets

56

Vinculum



• CrossWikis provides better cond. prob.

56

Vinculum



• CrossWikis provides better cond. prob.• Fine-grained entity types are very useful

56

Vinculum



• CrossWikis provides better cond. prob.• Fine-grained entity types are very useful• Coreference and Coherence also improve the performance

56

Vinculum



• CrossWikis provides better cond. prob.• Fine-grained entity types are very useful• Coreference and Coherence also improve the performance

• http://github.com/xiaoling/vinculum

56

Vinculum


• a modular deterministic system achieves good performance • a comprehensive evaluation over nine data sets

• CrossWikis provides better cond. prob. • Fine-grained entity types are very useful • Coreference and Coherence also improve the performance

• http://github.com/xiaoling/vinculum

Conclusion

57

Thanks!

Questions?

Vinculum


Entity Type


Coreference

Coherence

Mention Extraction

Oracle Entity Types

58

F1

50

62.5

75

87.5

100

ACE MSNBC AIDA-D AIDA-T TAC09 TAC10 TAC10T TAC11 TAC12 Overall+NER (Gold) +FIGER (Gold)

Implementation Details

59

Component Implementation

Mention Extraction Stanford NER

Candidate Generation CrossWikis

Entity Type Prediction Fine-grained Entity Types

Coreference Stanford Coreference

Coherence NGD + relational triples

System Comparison

60

VINCULUM AIDA WIKIFIER

Mention Extraction NER NER NER, noun

phrasesCandidate Generation CrossWikis intra-Wikipedia intra-Wikipedia

Entity Types FIGER NER NER

Coreference representative mention - re-rank the

candidates

Coherence NGD, relational NGD NGD, relational

Learning deterministic trained on AIDA trained on Wiki

Error Analysis: Metonymy

• South Africa managed to avoid a fifth successive defeat in 1996 at the hands of the All Blacks …

• Prediction : South Africa

• Label : South Africa national rugby union team

61

Error Analysis: Entity Types

• Instead of Los Angeles International, for example, consider flying into Burbank or John Wayne Airport ...

• Prediction : Burbank, California

• Label : Bob Hope Airport

62

Error Analysis: Coreference

• It is about his mysterious father, Barack Hussein Obama, an imperious if alluring voice gone distant and then missing.

• Prediction : Barack Obama

• Label : Barack Obama Sr.

63

Error Analysis: Context

• Scott Walker removed himself from the race, but Green never really stirred the passions of former Walker supporters, nor did he garner outsized support “outstate”.

• Prediction : Scott Walker (singer)

• Label : Scott Walker (politician)

64

Documents

Design Challenges for Entity Linking - GitHub Pagesxiaoling.github.io/pubs/ling-tacl15-slides.pdf · 2018-05-18 · Design Challenges for Entity Linking Xiao Ling, Sameer Singh, Daniel