Case Study: Real-time Recommendations with a Graph Database

Realtime Recommendations

with a Graph Database

Andreas Kollegger

#neo4j @akollegger

Wednesday, August 24, 11

Andreas Kollegger --[known_as]--> @akollegger --[talks_about]--> #neo4j

What’s the plan?

๏Graph Database 101

๏Case Study: Viadeo

๏Finding new Friends

๏Recommending new Sites

๏tweet using #neo4j

๏I am @akollegger

NOSQL Categories

Data Complexity

NOSQL Categories

Data Complexity

Key-ValueStore

NOSQL Categories

Size BigTable

Clones

Data Complexity

Key-ValueStore

NOSQL Categories

Size BigTable

Clones

Data Complexity

Key-ValueStore

DocumentDatabases

NOSQL Categories

Size BigTable

Clones

Data Complexity

Key-ValueStore

DocumentDatabases

GraphDatabases

NOSQL Categories

Size BigTable

Clones

Data Complexity

Key-ValueStore

DocumentDatabases

GraphDatabases

NOSQL Categories

Size BigTable

Clones

Data Complexity

Key-ValueStore

DocumentDatabases

GraphDatabases

usecases

(this is still billionsof nodes &

relationships)

Graph DB 101(Reading a graph)

A graph database?

๏no: not for storing charts & graphs, or vector artwork

A graph database?

๏yes: for storing data that is structured as a graph

A graph database?

• remember linked-list, tree?

A graph database?

• graphs are the generalized connected data structure

A graph database?

๏whiteboard friendly - no data mangling required

A graph database?

๏whiteboard friendly - no data mangling required

๏“A traditional relational database may tell you the average salary of everyone in this conference,

but a graph database will tell you who is most likely to buy you a beer.”

Q: What are graphs good for?

๏Recommendations

๏Business intelligence

๏Social computing

๏Geospatial

๏MDM

๏Systems management

๏Genealogy

๏Time series data

๏Product catalogue

๏Web analytics

๏Scientific computing (especially bioinformatics)

๏ Indexing your slow RDBMS

๏And much more!

Q: What are graphs good for?

A: highly-connected data

G=(V,E)

๏A (Graph) -records data in- (Nodes)

๏A (Graph) -records data in- (Relationships)

๏(Nodes) are -organized by- (Relationships)

๏(Nodes & Relationships) -have- (Properties)

๏ Instead of a Data Model, you have a Data Graph

๏Structure is defined by the Relationships you create

records data in

Nodes Relationships

Properties

organized by

records data in

Query a graph with a traversal

๏A (Traversal) -navigates- a (Graph)

๏A (Traversal) -identifies- (Paths)

๏(Paths) -order- (Nodes)

๏Traversals reveal localized data

๏Find data that is relevant to other data

Nodes Relationships

organized by

Traversal

identifies

navigatesGraph

records data in records data in

Indexes look-up Nodes or Relationships

๏An (Index) -maps to either-

(Nodes or Relationships)

๏An (Index) is a -special- (Traversal)

๏ Indexes find starting points for a traversal

๏Can be queried with simple constraints

Nodes Relationships

Properties

organized by

maps from

maps to either maps to either

Traversalspecial

Neo4j is a Graph Database

๏A (Graph Database) -manages-

a (Graph) and related (Indexes)

๏A Graph Database:

• Full ACID Transactions

•High-Availability (read scaling)

• 32 Billion Nodes, 32 Billion Relationships, 64 Billion Properties

• Server with REST API

•Or, Embeddable as a Java Library

Graph Indexes

GraphDatabase

manages manages

lookup

Neo4j Features

๏Neo4j is stable

• In 24/7 operation since 2003

๏Neo4j is under active development

• open source, with a vibrant community

๏High performance graph operations

• high performance? like what? ...

Example: Social Network “path exists”๏~1,000 persons

๏average 50 friends per person

๏pathExists(a,b) limited to depth 4

๏caches warmed up to eliminate disk I/O

# persons query time

Relational database

1,000 2000ms

1,000 2ms

1,000,000 2ms

EmilAndreas

MarcusGabriella

Madeline

Joanne

Relational database

1,000 2000ms

1,000 2ms

1,000,000 2ms

EmilAndreas

MarcusGabriella

Madeline

Joanne

Relational database

1,000 2000ms

1,000 2ms

1,000,000 2ms

EmilAndreas

MarcusGabriella

Madeline

Joanne

Relational database

1,000 2000ms

1,000 2ms

1,000,000 2ms

EmilAndreas

MarcusGabriella

Madeline

Joanne

Case Study: Viadeo

Case Study: Viadeo๏Viadeo - 35 million user professional network

๏the past tech

• recommendations running on MySQL as a batch job

๏the past tech

• took an hour, then success happened.. and it took two days

๏the past tech

๏now using Neo4j

๏the past tech

๏now using Neo4j

• recommendations run in real-time

๏the past tech

๏now using Neo4j

•what does a recommendation look like?

๏the past tech

๏now using Neo4j

๏the past tech

๏now using Neo4j

๏Check out: http://www.youtube.com/watch?v=sLlbtoCfciE

A closer look at recommendations

Recommendations 2 ways...

๏Gremlin - graph scripting

•an algorithmic approach

๏Cypher - declarative querying

•a “pattern matching” approach

๏Stig - functional graph query

•ok, not yet...16

Gremlin

Gremlin๏a graph scripting DSL

๏part of the TinkerPop stack

๏try it out:

• g - the graph itself

๏try it out:

• g.v(0) - node 0

๏try it out:

• g.v(0) - node 0

• g.v(0).in - nodes connected to Node 0

๏try it out:

• g.v(0) - node 0

• g.v(1).out(“KNOWS”) - nodes connected by “KNOWS”

๏try it out:

• g.v(0) - node 0

๏details about Gremlin: https://github.com/tinkerpop/gremlin

๏try it out:

• g.v(0) - node 0

๏details about Gremlin: https://github.com/tinkerpop/gremlin

๏props to @twarko for creating it

Cypher

Cypher๏a pattern-matching query language (SQL-ish)

๏developed in-house

๏try it out:

• start a=(0) return a - get node 0

๏try it out:

• start a=(1) match (a)-->(b) return b - traverse from node 1

๏try it out:

• start a=(1) match (a)--()--(c) return c - friends of friends

๏try it out:

๏details about Cypher:

๏try it out:

๏details about Cypher:

• http://docs.neo4j.org/chunked/stable/cypher-query-lang.html

The Social Graphas a Data Graph

The Usual Social Graph, is also data

name:Andreas

employer:Neo Technology

worked_at

body: presenting at NoSQL Now! like right now!

status

url: http://www.amazon.com/Nikon-14-24mm-2-8G-Nikkor-Angle/dp/B000VDCTCI/

body: mocking up a simplified social graph

status

url: http://www.imdb.com/title/tt1475582/

employer:Xilinx

worked_at

The Usual Social Graph, is also data

name:Andreas

worked_at

status

status

employer:Xilinx

worked_at

with the Usual “status tweets”

name:Andreas

worked_at

status

status

employer:Xilinx

worked_at

with the Usual “status tweets”

name:Andreas

worked_at

status

status

employer:Xilinx

worked_at

and the Usual Suspects

name:Andreas

worked_at

status

status

employer:Xilinx

worked_at

and the Usual Suspects

name:Andreas

worked_at

status

status

employer:Xilinx

worked_at

and the Usual Business

name:Andreas

worked_at

status

status

employer:Xilinx

worked_at

and the Usual Business

name:Andreas

worked_at

status

status

employer:Xilinx

worked_at

and the Usual Likes

name:Andreas

worked_at

status

status

employer:Xilinx

worked_at

and the Usual Likes

name:Andreas

worked_at

status

status

employer:Xilinx

worked_at

Making Friends

My Expanded Ego Network

name:Andreas

Me & My Friends

name:Andreas

Me & My Friends

name:Andreas

Friends of my Friends

name:Andreas

name:Andreas

name:Andreas

Friend of a Friend of a Friend?

name:Andreas

name:Andreas

name:Andreas

Whatever.

Query for Friends of Friends

Query for Friends of Friends// Gremlin - andreas to friends to their friends, counted

Query for Friends of Friends// Gremlin - andreas to friends to their friends, countedandreas.out('KNOWS').out('KNOWS').name.groupCount(m)

// Cypher - match from andreas to friends to their friends

// Cypher - match from andreas to friends to their friendsstart andreas=(1) match (andreas)-[:KNOWS]->(friends)

-[:KNOWS]->(foafs) return foafs

// result: Emil, Stephen, Allison (& Peter?)

Is that good enough?

Ranking my Friends’ Friends (intuitively)

name:Andreas

name:Andreas

name:Andreas

name:Andreas

name:Andreas

Query for Ranked extended FoaF

Query for Ranked extended FoaF// Gremlin - a bit more complex

Query for Ranked extended FoaF// Gremlin - a bit more complex// - requiring “memory” and weights

Query for Ranked extended FoaF// Gremlin - a bit more complex// - requiring “memory” and weights// - weights decrease with each “hop” outward

Query for Ranked extended FoaF// Gremlin - a bit more complex// - requiring “memory” and weights// - weights decrease with each “hop” outward// - extended hop, to include Stephen

Query for Ranked extended FoaF// Gremlin - a bit more complex// - requiring “memory” and weights// - weights decrease with each “hop” outward// - extended hop, to include Stephenandreas.out('KNOWS').aggregate(x).out('KNOWS').except(x).name

.groupCount(m){it}{it+1.0}.back(2)

.groupCount(m){it}{it+1.0}.back(2).out('KNOWS').except(x).name.groupCount(m){it}{it+0.5}

==>Emil=2.0

==>Emil=2.0==>Allison=2.0

==>Emil=2.0==>Allison=2.0==>Stephen=1.0

==>Emil=2.0==>Allison=2.0==>Stephen=1.0==>Tiberius=0.5

Like, like this

What else might I like?

name:Andreas

likes:+1

url: http://theoatmeal.com/comics/state_web_summer/

url: http://browsertoolkit.com/fault-tolerance.png

url: http://lensbaby.com/optics-pinhole.php

likes:+1

likes:+2

likes:+1

likes:+2

likes:+3

What else might I like?

name:Andreas

likes:+1

url: http://theoatmeal.com/comics/state_web_summer/

url: http://browsertoolkit.com/fault-tolerance.png

url: http://lensbaby.com/optics-pinhole.php

likes:+1

likes:+2

likes:+1

likes:+2

likes:+3

Query for Things to Like

Query for Things to Like// Cypher - websites my friends like

Query for Things to Like// Cypher - websites my friends likestart andreas=(1) match (andreas)-[:KNOWS]->(friend)

-[:LIKES]->(website) return website.url

+------------------------------------------------+

+------------------------------------------------+| website.url |

+------------------------------------------------+| website.url |+------------------------------------------------+

+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png |

+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 |

+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 || http://lensbaby.com/optics-pinhole.php |

+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 || http://lensbaby.com/optics-pinhole.php || http://theoatmeal.com/comics/state_web_summer/ |

+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 || http://lensbaby.com/optics-pinhole.php || http://theoatmeal.com/comics/state_web_summer/ || http://www.imdb.com/title/tt0133093 |

+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 || http://lensbaby.com/optics-pinhole.php || http://theoatmeal.com/comics/state_web_summer/ || http://www.imdb.com/title/tt0133093 || http://browsertoolkit.com/fault-tolerance.png |

+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 || http://lensbaby.com/optics-pinhole.php || http://theoatmeal.com/comics/state_web_summer/ || http://www.imdb.com/title/tt0133093 || http://browsertoolkit.com/fault-tolerance.png |+------------------------------------------------+

+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 || http://lensbaby.com/optics-pinhole.php || http://theoatmeal.com/comics/state_web_summer/ || http://www.imdb.com/title/tt0133093 || http://browsertoolkit.com/fault-tolerance.png |+------------------------------------------------+6 rows, 4 ms

Query for Things to Like// ordered by average rating

Query for Things to Like// ordered by average ratingstart andreas=(1) MATCH (andreas)-[:KNOWS]->(friend)

-[l:LIKES]->(website) return website.url, AVG(l.plus)

-[l:LIKES]->(website) return website.url, AVG(l.plus) ORDER BY AVG(l.plus) DESC

+--------------------------------------------------------------+

+--------------------------------------------------------------+| website.url | avg(l.plus) |

+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+

+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 |

+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 || http://www.imdb.com/title/tt0436992 | 2.0 |

+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 || http://www.imdb.com/title/tt0436992 | 2.0 || http://www.imdb.com/title/tt0133093 | 2.0 |

+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 || http://www.imdb.com/title/tt0436992 | 2.0 || http://www.imdb.com/title/tt0133093 | 2.0 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 |

+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 || http://www.imdb.com/title/tt0436992 | 2.0 || http://www.imdb.com/title/tt0133093 | 2.0 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 || http://browsertoolkit.com/fault-tolerance.png | 1.0 |

+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 || http://www.imdb.com/title/tt0436992 | 2.0 || http://www.imdb.com/title/tt0133093 | 2.0 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 || http://browsertoolkit.com/fault-tolerance.png | 1.0 |+--------------------------------------------------------------+

+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 || http://www.imdb.com/title/tt0436992 | 2.0 || http://www.imdb.com/title/tt0133093 | 2.0 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 || http://browsertoolkit.com/fault-tolerance.png | 1.0 |+--------------------------------------------------------------+5 rows, 5 ms

Query for Things to Like// ordered by number of ratings, then average rating

Query for Things to Like// ordered by number of ratings, then average ratingstart andreas=(1) MATCH (andreas)-[:KNOWS]->(friend)

-[l:LIKES]->(website) return website.url, AVG(l.plus), count(*) ORDER BY count(*) DESC, AVG(l.plus) DESC

+-------------------------------------------------------------------------+

+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |

+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+

+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 |

+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 || http://lensbaby.com/optics-pinhole.php | 3.0 | 1 |

+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 || http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 |

+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 || http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 |

+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 || http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 | 1 |

+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 || http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 | 1 |+-------------------------------------------------------------------------+

+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 || http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 | 1 |+-------------------------------------------------------------------------+5 rows, 7 ms

Query for Things to Like// ordered, grouped, rated higher than 1

Query for Things to Like// ordered, grouped, rated higher than 1start andreas=(1) MATCH (andreas)-[:KNOWS]->(friend)

-[l:LIKES]->(website)

-[l:LIKES]->(website) WHERE l.plus>1

return website.url, AVG(l.plus), count(*)

return website.url, AVG(l.plus), count(*) ORDER BY count(*) DESC, AVG(l.plus) DESC

+-----------------------------------------------------------------+

+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |

+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-----------------------------------------------------------------+

+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-----------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 | 1 |

+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-----------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 |

+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-----------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 |

+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-----------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 |+-----------------------------------------------------------------+

+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-----------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 |+-----------------------------------------------------------------+3 rows, 91 ms

What else would you recommend? ๏Continue to elaborate the graph, and the algorithm

•more variables reveal more nuances in results

๏Rank influence of each user

• PageRank, Centrality, pick-an-algorithm

•Modulate weights by user rank

Questions?

๏learn more at http://neo4j.org

๏join the mailing list

๏join a meetup, or start one and we’ll come to you

Case Study: Real-time Recommendations with a Graph Database

Technology

Graph Database with Cassandra

Graph Database and Neo4j

TigerGraph: A Native MPP Graph Database · 2019-01-25 · We introduce TigerGraph, a new graph database product by the homonymous company. TigerGraph is a native parallel graph database,

Oracle Spatial and Graph in Oracle Database 19c...Title Oracle Spatial and Graph in Oracle Database 19c Author Oracle Corporation Subject Oracle Spatial and Graph in Oracle Database

Database normalization using dependencies graph fotache_m_strimbei_c__ibima2008

Neo4j Graph Database, from PHP

sones graph database in the cloud

Introduction to Graph database - cse.hcmut.edu.vn

Do I need a Graph Database?

Graph database

Graph database in sv meetup

Neo4J: Graph Database - People Neo4J: Graph Database Giuseppe Burtini, Graeme Douglas and Yipin Guo February 24, 2013 Giuseppe Burtini, Graeme Douglas and Yipin Guo Neo4J: Graph Database

Graph database super star

TigerGraph: A Native MPP Graph Database

Lightning-Talk: Ramen->Graph->Database, neo4j_fdw

Application of graph databases and graph theory … a labelled property graph-based model using the world leading graph database Neo4J 1 as a graph database framework

Graph Database System Neo4j

Visualize your graph database

OrientDB: graph database in practice - · PDF fileYou will learn In short 3 WHAT Graph Graph database WHY Real world example HOW With PHP With relational database Best practice

Introduction to Graph Database