Traversing Knowledge Graphs in Vector Space

7/23/2019 Traversing Knowledge Graphs in Vector Space

1/149

Traversing Knowledge Graphs inVector Space

Kelvin Guu, John Miller, Percy Liang


2/149

Knowledge graphs


3/149

Knowledge graphs

What languages are spoken by people in Portugal?


4/149

Knowledge graphs

portugal/location/language



5/149

Knowledge graphs



portugal


6/149

Knowledge graphs



portugal

fernando_pessoa

jorge_sampaio


7/149

Knowledge graphs

french

portuguese

english



portugal

fernando_pessoa

jorge_sampaio


8/149

Knowledge graphs

french

portuguese

english

fernando_pessoa

jorge_sampaio

portugallocation


9/149

Knowledge graphs

french

portuguese

english

fernando_pessoa

jorge_sampaio

portugal

(fernando_pessoa, location, portugal)

location


10/149

Knowledge graphs

french

portuguese

english

fernando_pessoa

jorge_sampaio

portugal

(fernando_pessoa, location, portugal)

location

fact = edge = triple


11/149

Knowledge graphs

french

portuguese

english

fernando_pessoa

jorge_sampaio

portugallocation

93.8%of persons from Freebase have no placeof birth, and 78.5%have no nationality [Min et al, 2013]


12/149

Knowledge graphs

french

portuguese

english

fernando_pessoa

jorge_sampaio

portugallocation


strength:compositionality


13/149

Knowledge graphs

french

portuguese

english

fernando_pessoa

jorge_sampaio

portugallocation


weakness:

incompleteness

strength:compositionality


14/149

Vector space models

fernando

pessoa

portugal

barackobama

united

states


15/149

Vector space models

fernandopessoa

portugal

barackobama

united

states

location

location


16/149

Vector space models

fernandopessoa

portugal

barackobama

united

states

location

location

large # facts > low-dimensional space


17/149

Vector space models

fernandopessoa

portugal

barackobama

united

states

location

locationcompression > generalization

large # facts > low-dimensional space


18/149

Vector space models

Tensor factorization [Nickel et al., 2011]

Neural Tensor Network [Socher et al., 2013]

TransE [Bordes et al., 2013]

Universal Schema [Riedel et al., 2013]

General framework + comparison [Yang et al.,2015]

Compositional embedding of paths [Neelakantan etal., 2015]


19/149

Vector space models

Tensor factorization [Nickel et al., 2011]

Neural Tensor Network [Socher et al., 2013]


Universal Schema [Riedel et al., 2013]

General framework + comparison [Yang et al.,2015]

Compositional embedding of paths [Neelakantan etal., 2015]

strength:handle incompleteness


20/149


21/149

Graph databases Vector space models

Compositionalqueries

Handleincompleteness


22/149




This talk


23/149




This talk


24/149




This talk

PART I

PART II


25/149

Outline


26/149

Outline

PART I


27/149

Outline

PART I

Interpret many existing vector space models as each

implementing a traversal operator.


28/149

Outline

PART I




29/149

Outline

PART I



Proposehow all these models can be generalized to answer

path queries.


30/149

Outline

PART I




path queries.


31/149

Outline

PART I




path queries.

PART II


32/149

Outline

PART I




path queries.

PART II

Showthat when we train these models to answer path queries,they actually become better at predicting missing facts.


33/149

Outline

PART I




path queries.

PART II



34/149

Outline

PART I




path queries.

PART II


Explainwhy this new path-based training procedurehelps.


35/149


36/149

Path queries




37/149

Path queries

{portugal}portugal


38/149

Path queries

{portugal}portugal / location

{fernando_pessoa,

jorge_sampaio,

vasco_da_gama}


39/149

Path queries


/ language{fernando_pessoa,

jorge_sampaio,

vasco_da_gama}

{portuguese,

spanish,

english}


40/149

Path queries in vector space



jorge_sampaio,

vasco_da_gama}

{portuguese,

spanish,

english}


41/149




jorge_sampaio,

vasco_da_gama}

{portuguese,

spanish,

english}

sparse

vectors


42/149




jorge_sampaio,

vasco_da_gama}

{portuguese,

spanish,

english}

sparse

vectors


43/149




jorge_sampaio,

vasco_da_gama}

{portuguese,

spanish,

english}

sparse

vectorsx ->


44/149




jorge_sampaio,

vasco_da_gama}

{portuguese,

spanish,

english}

sparse

vectorsx -> x ->


45/149




jorge_sampaio,

vasco_da_gama}

{portuguese,

spanish,

english}

sparse

vectorsx -> x ->

dense,

low-dim

vectors

x -> x ->


46/149




jorge_sampaio,

vasco_da_gama}

{portuguese,

spanish,

english}

sparse

vectorsx -> x ->

dense,

low-dim

vectors

x -> x ->

set vectors!


47/149


{portugal}portugal / location / language

{fernando_pessoa,

jorge_sampaio,

vasco_da_gama}

{portuguese,

spanish,

english}

sparse

vectorsx -> x ->

dense,

low-dim

vectors

x -> x ->

traversaloperators


48/149


{portugal} {fernando_pessoa,

jorge_sampaio,

vasco_da_gama}

{portuguese,

spanish,

english}

sparse

vectorsx -> x ->

dense,

low-dim

vectors

x -> x ->

membership operator:dot product

portugal / location / language

xenglish

xenglish


49/149


q = s / r1 / r2 / / rk


50/149


q = s / r1 / r2 / / rk

x>

sWr1Wr2 . . .W rk


51/149


q = s / r1 / r2 / / rk

x>

sWr1Wr2 . . .W rk

is tan answer to q?


52/149


q = s / r1 / r2 / / rk

x>

sWr1Wr2 . . .W rk

is tan answer to q?

xtscore(q, t) =


53/149


q = s / r1 / r2 / / rk

x>

sWr1Wr2 . . .W rk

is tan answer to q?

xtscore(q, t) =


54/149

Training examples:

Margin:

Objective:

Algorithm:

Training


55/149

Training examples:

Margin:

Objective:

Algorithm:

Training

(q, t)


56/149

Training examples:

Margin:

Objective:

Algorithm:

Training

(q, t)

margin (q, t, t0) = score (q, t) score (q, t0)


57/149

Training examples:

Margin:

Objective:

Algorithm:

Training

(q, t)


nX

i=1

X

t0N(qi)

[1margin (qi, ti, t0)]+


58/149

Training examples:

Margin:

Objective:

Algorithm:

Training

(q, t)


nX

i=1

X

t0N(qi)


SGD

T i i


59/149

Training examples:

Margin:

Objective:

Algorithm:

Training

(q, t)


nX

i=1

X

t0N(qi)


SGD

Next:generalize existing models

E i i d l


60/149

Existing vector space models

Tensor factorization [Nickel et al., 2011] Neural Tensor Network [Socher et al., 2013]


Universal Schema [Riedel et al., 2013] General framework + comparison [Yang et al.,

2015]

E i i d l


61/149


E i i d l


62/149


score (subject,predicate, object)

E i ti t d l


63/149



score (obama, nationality, united states)

core (obama,nationality, germany)

= HIGH

= LOW

E i ti t d l


64/149





= HIGH

= LOW

score (obama,nationality, france)?

E i ti t d l


65/149





= HIGH

= LOW

score (obama,nationality, france)?

special case ofanswering

path queries

E i ti t d l


66/149


r

s

t

Predict whether the triple (s, r, t)is true

N th d l


67/149

New path query models

r3r1 r2

r4

Predict whether a path existsbetween sand t

s

t

N th d l


68/149

New path query models

r3r1 r2

r4


s

t

single-hop > multi-hop

G li i ti d l


69/149

Original model

(single edge)

Generalize existing models

New model

(path)

G li i ti d l


70/149

Original model

(single edge)


New model

(path)

Bilinear

(Nickel+, 2012)



71/149

Original model

(single edge)


New model

(path)

x>

sWrxt

Bilinear

(Nickel+, 2012)



72/149

Original model

(single edge)


New model

(path)

x>

sWrxt x

>

sWr1Wr2 . . .W rkxt

Bilinear

(Nickel+, 2012)



73/149

Original model

(single edge)


New model

(path)

x>

sWrxt x

>

sWr1Wr2 . . .W rkxt

Bilinear

(Nickel+, 2012)



74/149

Original model

(single edge)


New model

(path)

x>

sWrxt x

>

sWr1Wr2 . . .W rkxt

Bilinear

(Nickel+, 2012)



75/149

Original model

(single edge)


New model

(path)

x>

sWrxt x

>

sWr1Wr2 . . .W rkxt

Bilinear

(Nickel+, 2012)

TransE

(Bordes+, 2013)



76/149

Original model

(single edge)


New model

(path)

x>

sWrxt x

>

sWr1Wr2 . . .W rkxt

Bilinear

(Nickel+, 2012)

TransE

(Bordes+, 2013) kxs + wr xtk

2

2



77/149

Original model

(single edge)


New model

(path)

x>

sWrxt x

>

sWr1Wr2 . . .W rkxt

Bilinear

(Nickel+, 2012)

TransE

(Bordes+, 2013) kxs + wr1 . . .+ wrk xtk

2

2kxs + wr xtk2

2



78/149

Original model

(single edge)


New model

(path)

x>

sWrxt x

>

sWr1Wr2 . . .W rkxt

Bilinear

(Nickel+, 2012)

TransE


2

2kxs + wr xtk2

2



79/149

Original model

(single edge)


New model

(path)

x>

sWrxt x

>

sWr1Wr2 . . .W rkxt

Bilinear

(Nickel+, 2012)

TransE


2

2kxs + wr xtk2

2

More generally

(??, 2015)



80/149

Original model

(single edge)


New model

(path)

x>

sWrxt x

>

sWr1Wr2 . . .W rkxt

Bilinear

(Nickel+, 2012)

TransE


2

2kxs + wr xtk2

2

More generally

(??, 2015)M (Tr(xs), xt)



81/149

Original model

(single edge)


New model

(path)

x>

sWrxt x

>

sWr1Wr2 . . .W rkxt

Bilinear

(Nickel+, 2012)

TransE


2

2kxs + wr xtk2

2

More generally

(??, 2015)M (Trk(. . .Tr1(xs)), xt)M (Tr(xs), xt)



82/149

Original model

(single edge)


New model

(path)

x>

sWrxt x

>

sWr1Wr2 . . .W rkxt

Bilinear

(Nickel+, 2012)

TransE


2

2kxs + wr xtk2

2

More generally



83/149



84/149

Original model

(single edge)


New model

(path)

x>

sWrxt x

>

sWr1Wr2 . . .W rkxt

Bilinear

(Nickel+, 2012)

TransE


2

2kxs + wr xtk2

2

More generally


answer path queries

a newway to trainexisting models

Old: single-edge training


85/149

Old: single-edge training

r

s

t

Predict whether the triple (s, r, t)is true

New: path training


86/149

New: path training

r3r1 r2

r4


s

t


87/149

Outline


88/149

Outline

PART I




path queries.

PART II

Showthat when we train these models to answer path queries,

they actually become better at predicting missing facts.

Explainwhy this new path-based training procedurehelps.

Experiments


89/149

Experiments

PATHSINGLE-

EDGE

PATH

SINGLE-EDGE

TRAINING

TASK


90/149


91/149

Experiments


92/149

Experiments

PATH PATH-1

PATH

PATH-1

TRAINING

TASK

portugal/location/language?

(pessoa, language, english)?

Train and test distributions match


93/149

Train and test distributions match

PATHSINGLE-

EDGE

PATH

SINGLE-EDGE

TRAINING

TASK



Datasets


94/149

Datasets

[Chen et al, 2013] WordNet Freebase

Entities 39,000 75,000

Relations 11 13

Train edges 113,000 316,000

Test edges 11,000 24,000

Datasets


95/149

Datasets


Entities 39,000 75,000

Relations 11 13

Train edges 113,000 316,000

Test edges 11,000 24,000

(laugh, has_instance, giggle)

Datasets


96/149

Datasets


Entities 39,000 75,000

Relations 11 13

Train edges 113,000 316,000

Test edges 11,000 24,000


(snort, derivationally_related, snorter)

Datasets


97/149

Datasets


Entities 39,000 75,000

Relations 11 13

Train edges 113,000 316,000

Test edges 11,000 24,000


(snort, derivationally_related, snorter)

(abraham_lincoln, place_of_birth, kentucky)

Path training: way more data


98/149

Path training: way more data

~ WordNet Freebase

Single-edge training 113,000 316,000

Path training 2,000,000 6,000,000

Single-edge test 11,000 24,000

Path test 45,000 110,000

Path sampling


99/149

Path sampling

r3r1

r2r4

1. start at a random node in the knowledge graph2. traverse a random sequence of relations3. The final destination marked as one answer to the query.

s

t

Several models that we generalize


100/149

g

Evaluation metric


101/149

Evaluation metric


Evaluation metric


102/149

Evaluation metric


1. klingon2. portuguese3. latin

4. javascript5. chinese

Evaluation metric


103/149

a uat o et c


Quantile:percentage of negatives ranked aftercorrect answer (100 is best, 0 is worst)



Evaluation metric


104/149


Quantile:percentage of negatives ranked aftercorrect answer (100 is best, 0 is worst)

75%



Experiments


105/149

p

PATHSINGLE-

EDGE

PATH

SINGLE-EDGE

TRAINING

TASK




106/149

Results: path querying


107/149

p q y g

Freebas

e

50

60

70

80

90

100

Bilinear Bilinear-Diag TransE

Edge Path Edge Path Edge Path

WordNet

50

6070

80

90

100



Why does path training help?


108/149

y p g p



109/149

Train and test distributions match:If you test onpath queries, you should train on path queries, not

single edges.

y p g p



110/149


single edges.

Why should single-edge training be so bad at

path queries?

y p g p



111/149


single edges.

Why should single-edge training be so bad at

path queries? Path query training cuts down on cascading errors

y p g p

Cascading errors


112/149

g

Who is Tad Lincolns grandparent?

Cascading errors


113/149

g


Cascading errors


114/149

g


pathtraining

Example


115/149

p


116/149

PATHSINGLE-

EDGE

PATH

SINGLE-EDGE

TRAINING

TASK



Results: single-edge task


117/149

Results: single-edge task


118/149

Freebas

e

50

60

70

80

90

100



WordNet

50

6070

80

90

100




119/149

PATHSINGLE-

EDGE

PATH

SINGLE-EDGE

TRAINING

TASK



Why does path training helpon the single edge task?


120/149

on the single-edge task?



121/149


Paths in the graph can help infer other edges.



122/149



Lao et al, 2010 (Path Ranking Algorithm)



123/149




Nickel et al, 2014



124/149




Nickel et al, 2014

Neelakantan et al, 2015



125/149


Paths in the graph can help infer other edges. Lao et al, 2010 (Path Ranking Algorithm)

Nickel et al, 2014

Neelakantan et al, 2015spouse

child

A B

C



126/149



Nickel et al, 2014


child

A B

C

child?



127/149



Nickel et al, 2014


child

A B

C

child?

spouse (A,B) child (B,C) = child (A,C)

Why does path training helpknowledge base completion?


128/149

knowledge base completion?



129/149


If you cannot model paths correctly, you cannotuse them to infer edges.



130/149


If you cannot model paths correctly, you cannotuse them to infer edges.

PATH SINGLE-EDGE

PATH

SINGLE-EDGE

TRAINING

TASK

Path training helps learn Horn clauses


131/149



132/149

parent (A,B) place of birth (B,C) = nationality (A,C)High-precision



133/149

parent (A,B) place of birth (B,C) = nationality (A,C)

High-precision

location (A,B) borders (B,C) = nationality (A,C)

Low-precision



134/149


High-precision


Low-precision

child (A,B) gender (B,C) = nationality (A,C)

Zero-coverage



135/149


High-precision


Low-precision

child (A,B) gender (B,C) = nationality (A,C)

Zero-coverage

WparentWplace of birth Wnationality


136/149

To recap


137/149


Compositional

queries



138/149

Connections

and

speculative thoughts

RNNs


139/149

RNNs


140/149

Repeated application of traversal operators can bethought of as implementing a recurrent neuralnetwork.

RNNs


141/149


Trk . . . Tr2 Tr1(xs)

RNNs


142/149



Initial hidden state: xs

Inputs: r1, r2, , rk

RNNs


143/149



Initial hidden state: xs

Inputs: r1, r2, , rk

[Neelakantan et al, 2015], [Graves et al, 2013]

Matrix factorization


144/149

x x

W

E

ET

[Nickel et al, 2013], [Bordes et al, 2011]

Matrix factorization


145/149

x x

W

E

ET

3

x x

WW

Graph-like data


146/149

All kinds of datasets are graphs

finished_giving

open domain relations

kelvin

his_talk

okay

was_considered

[Riedel et al, 2013], [Fader et al, 2011], [Banko et al 2007]

Graph-like data


147/149


the boy happily ate ice cream

the child enjoyed dessert

the child hated dessert

entailment

negation

textual entailment

[Bowman et al, 2015], [Dagan et al, 2009]

Graph-like data


148/149


0.1

word co-occurrences

king

throne

queen

0.4

[Levy, 2014], [Pennington, 2014], [Mikolov, 2013]


149/149

Thank you!

Documents

Traversing Knowledge Graphs in Vector Space