Look before you Hop:Conversational Question Answering over

Knowledge Graphs Using Judicious Context Expansion

Philipp Christmann, Rishiraj Saha Roy, Abdalghani Abujabal, Jyotsna Singh, Gerhard Weikum

Max Planck Institute for Informatics, Saarbrücken, Germany

CIKM 2019, Beijing

Question Answering over KGs

1

Which actor voiced the character Unicorn in The Last Unicorn? Mia Farrow

ØOne-shot information needØQuestion usually needs to be well-formed[Abujabal et al. (2018), Diefenbach et al. (2019), Huang et al. (2019)]

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

Schmendrick

voice actor

character role

Alan Arkin

performer

soundtrack album

The Last Unicorn Soundtrack

America

followed by Your move (album)

genre

Pop rock

performer

MiaFarrow

Conversational KG-QAWhich actor voiced the character Unicorn in The Last Unicorn?

…

And Alan Arkin was behind . . .?

So who performed songs?

Genre of this band?

By the way, who directed the movie?

Schmendrick

America

Folk rock

Jules Bass

Mia Farrow

2

Ø Information needs rarely one-shot Ø Natural mode of interaction

Conversational KG-QAWhich actor voiced the character Unicorn in The Last Unicorn?

And Alan Arkin was behind . . .?

So who performed songs?

Genre of this band?

By the way, who directed the movie?

3

Complete

Incomplete

Conversational KG-QA

And Alan Arkin was behind . . .?

So who performed songs?

Genre of this band?

By the way, who directed the movie?

4

ØAd hoc

Conversational KG-QA

And Alan Arkin was behind . . .?

So who performed songs?

Genre of this band?

By the way, who directed the movie?

5

ØAd hoc

ØUngrammatical

Conversational KG-QA

And Alan Arkin was behind . . .?

So who performed songs?

Genre of this band?

By the way, who directed the movie?

6

ØUngrammatical

ØAd hoc

Ø Information left out

Related Work

Question Completion [Kumar et. al (2017)]Ø Reformulation as complete, self-contained question

Large-scale benchmark CSQA [Saha et. al (2018)]Ø Question created semi-automatically using templatesØ Artificial conversation flow

CSQA–method and Dialog-to-Action [Guo et. al (2018)]Ø Seq-2-Seq learning methodsØTraining data sparse ØAd-hoc formulations key problem

7

Desiderata and ContributionsØ Large topic jumps in conversations are rare

Ø Conversations establish localized context in KG

Ø Harness underlying KG-connectivityØ Expand context with relevant entities and predicates in neighborhood

Ø CONVEX: CONVersational KG-QA using judicious context EXpansionØCompletely unsupervised!

Ø CONVEX works on top of any KG-QA system to handle conversations

8

Initial Context

9

Which actor voiced the character Unicorn in The Last Unicorn? Mia Farrow

Standalone KG-QA

MiaFarrow

NERD system

The Last Unicorn

The Unicorn

character role

voiceactor

Initial Context

10

And Alan Arkin was behind . . .?

Which actor voiced the character Unicorn in The Last Unicorn? Mia Farrow

How to expand the context?The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

Context Expansion

11

And Alan Arkin was behind . . .?

Which actor voiced the character Unicorn in The Last Unicorn? Mia Farrow

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

Neighborhood of Mia Farrow

Neighborhood of The Unicorn

Neighborhood of The Last Unicorn

Judicious Context Expansion

12

And Alan Arkin was behind . . .?

Which actor voiced the character Unicorn in The Last Unicorn? Mia Farrow

Do not expand with the complete neighborhood!

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

The Last Unicorn

character role{2}

voiceactor{1}

MiaFarrow

The Unicorn

Problem: Exploring Context Neighborhood

13

And Alan Arkin was behind . . .?

performer{1}

soundtrack album

The Last Unicorn Soundtrack

America

rock band

character role{1}

voiceactor{2}

Alan Arkin

Schmendrick

genre{1}

Folk rock

spouse

Frank Sinatra

present in work based on present in

work

The Last Unicorn (novel)

genre{6}

speculative fiction novel

present in work

genre{2}

fantasy film followed by

Your move (album)

genre{4}Pop rock

occupation

singer

Vocal jazz

genre{5}

song by

900 Miles

genre{7}

Folk

cast member

political thriller

genre{8}

Argo

class

Frontier Nodes

14

The Last Unicorn

character role{2}

voiceactor{1}

MiaFarrow

The Unicorn

And Alan Arkin was behind . . .?

performer{1}

soundtrack album

The Last Unicorn Soundtrack

America

rock band

character role{1}

voiceactor{2}

Alan Arkin

Schmendrick

genre{1}

Folk rock

spouse

Frank Sinatra

present in work based on present in

work

The Last Unicorn (novel)

genre{6}

speculative fiction novel

present in work

genre{2}

fantasy film followed by

Your move (album)

genre{4}Pop rock

occupation

singer

Vocal jazz

genre{5}

song by

900 Miles

genre{7}

Folk

cast member

political thriller

genre{8}

Argo

classDetermine Frontier nodes to describe anexpansion border

Context Graph

15

And Alan Arkin was behind . . .?

Which actor voiced the character Unicorn in The Last Unicorn? Mia Farrow

Expand graph accordingly!Schmendrick

voice actor

character role

Alan Arkin

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

Context Graph

16

Schmendrick

voice actor

character role

Alan Arkin

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

performersoundtrack album

The Last Unicorn Soundtrack

America

Which actor voiced the character Unicorn in The Last Unicorn?

Graph expanded with relevant facts only

Mia Farrow

So who performed songs?

And Alan Arkin was behind . . .?

Context Graph

17

Genre of this band?

So who performed songs?

And Alan Arkin was behind . . .?

Which actor voiced the character Unicorn in The Last Unicorn?

Schmendrick

voice actor

character role

Alan Arkin

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

performersoundtrack album

The Last Unicorn Soundtrack

America How to determine Frontier nodes?

Mia Farrow

Relevance to the Question

18

The Last Unicorn

character role{2}

voiceactor{1}

MiaFarrow

The Unicorn

performer{1}

soundtrack album

The Last Unicorn Soundtrack

America

rock band

character role{1}

voiceactor{2}

Alan Arkin

Schmendrick

genre{1}

Folk rock

spouse

Frank Sinatra

present in work based on present in

work

The Last Unicorn (novel)

genre{6}

speculative fiction novel

present in work

genre{2}

fantasy film followed by

Your move (album)

genre{4}Pop rock

occupation

singer

Vocal jazz

genre{5}

song by

900 Miles

genre{7}

Folk

cast member

political thriller

genre{8}

Argo

class

Genre of this band?

Word2vec similarity of node with question word

=> Maximum across question words

Relevance to the Question

19

The Last Unicorn

character role{2}

voiceactor{1}

MiaFarrow

The Unicorn

performer{1}

soundtrack album

The Last Unicorn Soundtrack

America

rock band

character role{1}

voiceactor{2}

Alan Arkin

Schmendrick

genre{1}

Folk rock

spouse

Frank Sinatra

present in work based on present in

work

The Last Unicorn (novel)

genre{6}

speculative fiction novel

present in work

genre{2}

fantasy film followed by

Your move (album)

genre{4}Pop rock

occupation

singer

Vocal jazz

genre{5}

song by

900 Miles

genre{7}

Folk

cast member

political thriller

genre{8}

Argo

class

Genre of this band?

1.0

1.0

1.0 1.01.0

1.0

1.0

0.9

0.60.3

0.4

0.20.1

0.3

0.4

0.3

0.3

0.1 0.2

0.3

0.4 0.2

0.4

0.2 0.5

0.60.6

0.4

0.2

0.8

0.4

Relevance to the Context

20

The Last Unicorn

character role{2}

voiceactor{1}

MiaFarrow

The Unicorn

performer{1}

soundtrack album

The Last Unicorn Soundtrack

America

rock band

character role{1}

voiceactor{2}

Alan Arkin

Schmendrick

genre{1}

Folk rock

spouse

Frank Sinatra

present in work based on present in

work

The Last Unicorn (novel)

genre{6}

speculative fiction novel

present in work

genre{2}

fantasy film followed by

Your move (album)

genre{4}Pop rock

occupation

singer

Vocal jazz

genre{5}

song by

900 Miles

genre{7}

Folk

cast member

political thriller

genre{8}

Argo

class

Genre of this band?

1.0

1.0

1.0 1.01.0

1.0

match (genre{n}) = 1.0

1.0

Relevance to the Context

21

The Last Unicorn

character role{2}

voiceactor{1}

MiaFarrow

The Unicorn

performer{1}

soundtrack album

The Last Unicorn Soundtrack

America

rock band

character role{1}

voiceactor{2}

Alan Arkin

Schmendrick

genre{1}

Folk rock

spouse

Frank Sinatra

present in work based on present in

work

The Last Unicorn (novel)

genre{6}

speculative fiction novel

present in work

genre{2}

fantasy film followed by

Your move (album)

genre{4}Pop rock

occupation

singer

Vocal jazz

genre{5}

song by

900 Miles

genre{7}

Folk

cast member

political thriller

genre{8}

Argo

class

Genre of this band?

1.0

1.0

1.0 1.01.0

1.0

match (genre{n}) = 1.0

genre{1} 1.0

Relevance to the Context

22

The Last Unicorn

character role{2}

voiceactor{1}

MiaFarrow

The Unicorn

performer{1}

soundtrack album

The Last Unicorn Soundtrack

America

rock band

character role{1}

voiceactor{2}

Alan Arkin

Schmendrick

genre{1}

Folk rock

spouse

Frank Sinatra

present in work based on present in

work

The Last Unicorn (novel)

genre{6}

speculative fiction novel

present in work

genre{2}

fantasy film followed by

Your move (album)

genre{4}Pop rock

occupation

singer

Vocal jazz

genre{5}

song by

900 Miles

genre{7}

Folk

cast member

political thriller

genre{8}

Argo

class

Genre of this band?

1.0

1.0

1.0 1.01.0

1.0

match (genre{n}) = 1.0

genre{1} 1.0

Distance to all entity nodes in the context

=> Weighted by turn they occurred in

KG priors

23

Genre of this band?

The Last Unicorn

character role{2}

voiceactor{1}

MiaFarrow

The Unicorn

performer{1}

soundtrack album

The Last Unicorn Soundtrack

America

rock band

character role{1}

voiceactor{2}

Alan Arkin

Schmendrick

genre{1}

Folk rock

spouse

Frank Sinatra

present in work based on present in

work

The Last Unicorn (novel)

genre{6}

speculative fiction novel

present in work

genre{2}

fantasy film followed by

Your move (album)

genre{4}Pop rock

occupation

singer

Vocal jazz

genre{5}

song by

900 Miles

genre{7}

Folk

cast member

political thriller

genre{8}

Argo

class

award received

Not relevant!match (RSH-Gold for Cult Band) = 0.61match (rock band) = 0.65

prox (RSH-Gold for Cult Band) = 0.85prox (rock band) = 0.85

RSH-Gold for Cult Band

KG priors

24

Genre of this band?

The Last Unicorn

character role{2}

voiceactor{1}

MiaFarrow

The Unicorn

performer{1}

soundtrack album

The Last Unicorn Soundtrack

America

rock band

character role{1}

voiceactor{2}

Alan Arkin

Schmendrick

genre{1}

Folk rock

spouse

Frank Sinatra

present in work based on present in

work

The Last Unicorn (novel)

genre{6}

speculative fiction novel

present in work

genre{2}

fantasy film followed by

Your move (album)

genre{4}Pop rock

occupation

singer

Vocal jazz

genre{5}

song by

900 Miles

genre{7}

Folk

cast member

political thriller

genre{8}

Argo

class

award received

Not relevant!

match (RSH-Gold for Cult Band) = 0.61match (rock band) = 0.65

prox (RSH-Gold for Cult Band) = 0.85prox (rock band) = 0.85

RSH-Gold for Cult Band

Prioritize the more frequent/prominent entities and predicates

=> Normalize the value with maximum frequency

Frontier Score

25

Matching similarity

match (candidate c)

Context relevance

prox (candidate c)

KG priors

prior (candidate c)

With hyperparameters h1, h2, h3

Frontier Nodes

26

Candidate Matchgenre{1} 1.00

genre{2} 1.00

… …

folk rock band 0.89

RSH-Gold for Cult Band

0.87

fantasy film 0.36

… …

Candidate Proxgenre{1} 0.91

folk rock band 0.86

RSH-Gold for Cult Band

0.86

… …

genre{2} 0.34

fantasy film 0.36

… …

Candidate KG priors… …

genre{1} 0.56

genre{2} 0.56

… …

folk rock band 0.34

… …

RSH-Gold for Cult Band

0.01

Matching similarity Context relevance KG priors

Fagin’s Threshold Algorithm to retrieve top-kranked nodes according to frontier score

Frontier Nodes

27

ØOutput of Fagin’s Threshold AlgorithmÞ Top-ranked candidates according to Frontier score

Schmendrick

voice actor

character role

Alan Arkin

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

performersoundtrack album

The Last Unicorn Soundtrack

America genre{1} Folk rock

folk rock bandclass

Frontier nodes

Genre of this band?

Look before you Hop!

28

Genre of this band?

Schmendrick

voice actor

character role

Alan Arkin

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

performersoundtrack album

The Last Unicorn Soundtrack

America genre{1} Folk rock

folk rock bandclass

Schmendrick

voice actor

character role

Alan Arkin

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

performersoundtrack album

The Last Unicorn Soundtrack

America genre{1} Folk rock

folk rock bandclass

Answer to the Question?

29

Ø Distance to Frontier nodesØ Weighted by the frontier scoreØ distance_F

Ø Distance to all nodes in context graph XØ Weighted by the turn they occurred inØ distance_X

=> Explicit part

=> Implicit part

Genre of this band?

Answer Detection

30

Folk rock

Ø Top-ranked node according to answer_score

Genre of this band?

Schmendrick

voice actor

character role

Alan Arkin

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

performersoundtrack album

The Last Unicorn Soundtrack

America genre{1} Folk rock

folk rock bandclass

Answering Steps

1. Define expansion borderÞ Determine most relevant nodes in neighborhood of contextÞ Frontier nodes

2. Expand context according to frontier nodes3. Detect answer in expanded graph

31

Experimental Dataset: ConvQuestions

32

Ø 11,200 distinct conversationsØ 5 utterances per conversation

Ø Initial question + 4 follow-up questions

Ø Domains: Books, Movies, Music, TV Series, SoccerØ Gathered via crowdsourcing

Benchmark Properties

33

Ø Realistic benchmarkØ Questions created by humans from Amazon Mechanical TurkØ In topic of their choice

Ø Natural flow of conversationsØ Conversations were not interleavedØ Order of utterances was not permuted

ConvQuestions Examples

34

What is the name of the second book? Books

Ordinal questions

ConvQuestions Examples

35

Joins

The director’s first wife? Movies

What is the name of the second book? Books

ConvQuestions Examples

36

Incomplete cues

The director’s first wife? Movies

What is the name of the second book? Books

Music First album?

ConvQuestions Examples

37

The director’s first wife? Movies

What is the name of the second book? Books

Music First album?

How many creators has the TV series with less episodes?TV Series

Comparatives

ConvQuestions Examples

38

The director’s first wife? Movies

What is the name of the second book? Books

Music First album?

How many creators has the TV series with less episodes?

Did they win the Super Cup the previous year?

TV Series

Soccer

Temporal questions

Experimental Setup

39

Ø Underlying KGØ Wikidata

Ø Standalone KG-QA systems Ø QAnswer, Platypus, Naïve, Oracle

Ø MetricsØ P@1, MRR, Hit@5

Ø Hyperparameters h1,...,h5 tuned on small dev set

Experimental Setup

40

Ø BaselinesØ Star modelØ Chain model

Ø Question Completion

Ø Dialog-to-Action (Guo et al., NeurIPS 2018)

0.000

0.050

0.100

0.150

0.200

0.250

QAnswerQuestion CompletionQAnswer + CONVEXQAnswer + StarQAnswer + ChainDialog-to-Action

Main Results

41MRR

QAnswer can not handle incomplete questions

0.000

0.050

0.100

0.150

0.200

0.250

QAnswerQuestion CompletionQAnswer + CONVEXQAnswer + StarQAnswer + ChainDialog-to-Action

Main Results

42MRR

Question Completion helps QAnswer

Main Results

43MRR

CONVEX increases performance

0.000

0.050

0.100

0.150

0.200

0.250

QAnswerQuestion CompletionQAnswer + CONVEXQAnswer + StarQAnswer + ChainDialog-to-Action

Main Results

44

MRR

Star Model is intuitive

but oversimplifies

0.000

0.050

0.100

0.150

0.200

0.250

QAnswer

Question Completion

QAnswer + CONVEX

QAnswer + StarQAnswer + Chain

Dialog-to-Action

Main Results

45MRR

Chain Model performs worse

0.000

0.050

0.100

0.150

0.200

0.250

QAnswerQuestion CompletionQAnswer + CONVEXQAnswer + StarQAnswer + ChainDialog-to-Action

Main Results

46MRR

D2A not able to deal with complexity of ConvQuestions

0.000

0.050

0.100

0.150

0.200

0.250

QAnswerQuestion CompletionQAnswer + CONVEXQAnswer + StarQAnswer + ChainDialog-to-Action

Conclusion

47

Ø CONVEXØ First unsupervised systemØ Enables any standalone KG-QA with

conversational supportØ Based on judicious context expansion

Ø ConvQuestionsØ First realistic benchmark on Conversational KG-QAØ 11,200 conversations from 5 domains

Data and Code:qa.mpi-inf.mpg.de/convex

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

Conclusion

48

Ø CONVEXØ First unsupervised systemØ Enables any standalone KG-QA with

conversational supportØ Based on judicious context expansion

Ø ConvQuestionsØ First realistic benchmark on Conversational KG-QAØ 11,200 conversations from 5 domains

Data and Code:qa.mpi-inf.mpg.de/convex

Schmendrick

voice actor

character role

Alan Arkin

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

Conclusion

49

Ø CONVEXØ First unsupervised systemØ Enables any standalone KG-QA with

conversational supportØ Based on judicious context expansion

Ø ConvQuestionsØ First realistic benchmark on Conversational KG-QAØ 11,200 conversations from 5 domains

Data and Code:qa.mpi-inf.mpg.de/convex

Schmendrick

voice actor

character role

Alan Arkin

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

performersoundtrack album

The Last Unicorn Soundtrack

America

Conclusion

50

Ø CONVEXØ First unsupervised systemØ Enables any standalone KG-QA with

conversational supportØ Based on judicious context expansion

Ø ConvQuestionsØ First realistic benchmark on Conversational KG-QAØ 11,200 conversations from 5 domains

Data and Code:qa.mpi-inf.mpg.de/convex

Schmendrick

voice actor

character role

Alan Arkin

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

performersoundtrack album

The Last Unicorn Soundtrack

America genre{1} Folk rock

folk rock bandclass

Conclusion

51

Ø CONVEXØ First unsupervised systemØ Enables any standalone KG-QA with

conversational supportØ Based on judicious context expansion

Ø ConvQuestionsØ First realistic benchmark on Conversational KG-QAØ 11,200 conversations from 5 domains

Data and Code:qa.mpi-inf.mpg.de/convex

Schmendrick

voice actor

character role

Alan Arkin

The Last Unicorn

character role

voiceactor

MiaFarrow

The Unicorn

performersoundtrack album

The Last Unicorn Soundtrack

America genre{1} Folk rock

folk rock bandclass

Thank you!Thanks for SIGIR student travel grant!