Upload
sergio-gomez-villamor
View
11.443
Download
0
Embed Size (px)
Citation preview
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Technical Seminar
Sergio Gómez
December, 2011
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Introduction
Basic Concepts
Database construction
Query database
Loaders
Script loaders
Tips & tricks
Index
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Index
Introduction
Basic Concepts
Database construction
Query database
Loaders
Script loaders
Tips & tricks
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Introduction
Graph database
Graph databases focus on the structure of the model. Nodes and edges instead of tables. Relationships are first-class citizens.
Explicit in the model.
DEX is a programming librarywhich allows to manage agraph database.
Very large datasets. High performance
query processing.
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
IntroductionDex Definition
Persistent and temporary graph management programming library.
Data model: Typed and attributed directed multigraph.
Typed: Node and edge instances belong to a type (label). Attributed: Node and edge instances may have attribute
values. Directed: Edge can be directed or undirected. Multigraph: Multiple edges between two nodes.
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
IntroductionGraph Model
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Introduction
Basic Concepts
Database construction
Query database
Loaders
Script loaders
Tips & tricks
Index
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Basic Concepts
Java library public API Private native
dynamic library Automatically
loaded
System requirements: Java Runtime Environment, v1.5 or higher. Operative system:
Windows, MacOSX, Linux 32 and 64 bits
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Basic Concepts
Dex Session1
N
Graph
1
1
Persistent DB
Objects
1
N
Database
1
N
Set of OIDs
Dexjava Class Diagram
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Basic ConceptsMain methods
Dex
open(filename) Databasecreate(filename) Databaseclose()
DatabasenewSession() Session
Session
getGraph() Graphclose()
Graph
newNodeType(name) intnewEdgeType(name) intnewNode(type) longnewEdge(type) longnewAttribute(type, name) intsetAttribute(oid, attr, value)getAttribute(oid, attr) value
select(type) Objectsselect(attr, op, value) Objectsexplode(oid, type) Objectsneigbors(oid, type) Objects
ObjectsIterator
hasNext() booleannext() long
Objects
add(long)exists(long)
copy(objs)union(objs)intersection(objs)difference(objs)
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Index
Introduction
Basic Concepts
Database construction
Query database
Loaders
Script loaders
Tips & tricks
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Database construction Graph
Dex: Loads library and manages graph db instances. Database: Manages a graph db instance. Session: Manages queries and temporary data.
Nodes & Edges Type:
Dex identifier (integer) Public identifier (string)
Instance: DEX identifier (long) – OID belongs to a type
Attributes Attribute:
DEX identifier (int) public identifier (string) Scope: type or global Temporary (per Session) or persistent
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Database construction
Database Dex#create(String path, String alias)Creates a new graph database instance.Returns the Database instance to manage a new persistent graph.
Database Dex#open(String path, bool read)Opens an existing graph database instance.Read-only mode.Returns the Database instance to manage the persistent graph.
Session Database#newSession()Initiates a new user Session.
Graph Session#getGraph()Gets the Graph instance which represents the graph data.
Create a graph database
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Database construction
import com.sparsity.dex.gdb.*;…Dex dex = new Dex(new DexConfig());Database db = dex.create(“C:/image.dex”, “graphdb”);Session s = db.newSession();……s.close();db.close();dex.close();
Create a graph database example
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Database construction
int Graph#newNodeType(String name)Creates a new node type with the given unique name.Returns the Dex node type identifier.
long Graph#newNode(int nodeType)Creates a new node belonging to the given node type. Returns the Dex object identifier.
Add nodes
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Database construction
int Graph#newEdgeType(String name, bool directed,bool neighbors)
Creates a new edge type with the given unique name. Directed or undirected edge type.
Create neighbor-index or not. Returns the Dex edge type identifier.
int Graph#newRestrictedEdgeType(String name, int srcNodeType, int dstNodeType, bool neighbors)
Creates a new directed edge type with the given unique name.(Integrity restriction) Source and destination of the edge
instances are restricted to the given node types.Create neighbor-index or not.Returns the Dex edge type identifier.
Add edges
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Database construction
int Graph#newEdge(int edgeType, long tail, long head)Creates a new edge belonging to the given edge type. Tail is the source and head is the target.Returns the Dex edge identifier.
Add edges
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
…Graph g = s.getGraph();
int person = g.newNodeType(“PERSON”);long p1 = g.newNode(person);long p2 = g.newNode(person);long p3 = g.newNode(person);
int friend = g.newEdgeType(“FRIEND”, false, false);
long e1 = g.newEdge(friend, p1, p2);long e2 = g.newEdge(friend, p2, p3);
int loves = g.newEdgeType(“LOVES”, true, false);
long e3 = g.newEdge(loves, p1, p3);…
Add nodes and edges example
Database construction
p1
p2
p3
p1
p2
p3
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Database construction
class ValueEncapsulates a value and its domain (data type).Use them to set and get attribute values for the objects.
int Graph#newAttribute(int type, String name, DataType dt, AttributeKind kind)Creates a new attribute with the given unique name for the given node or edge type. Returns the Dex attribute identifier.
“dt” can be: Boolean, Integer, Long, Double, String, Text, Timestamp, OID.
“kind” can be:Basic: Just set and get operations are allowed.Indexed: Select operations are allowed as well as set and get operations.Unique: As indexed. Unique integrity restriction: no two objects with the same value for the attribute but NULL.
Manage attributes
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Database construction
Graph#setAttribute(long oid, int attr, Value v)Sets the given Value for the given attribute to the given object identifier. Given attribute must be defined for the object’s type. Value ‘s data type must match attribute’s data type or NULL.
Graph#getAttribute(long oid, int attr, Value v)Gets the Value for the given attribute and for the given object identifier. Given attribute identifier must be defined for the object’s type.
Manage attributes
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Database construction
…int name = g.newAttribute(person, “NAME”,
String, Unique);int age = g.newAttribute(person, “AGE”,
Integer, Indexed);Value v = new Value();
g.setAttribute(p1, name, v.setString(“JOHN”));g.setAttribute(p1, age, v.setInteger(18));g.setAttribute(p2, name, v.setString(“KELLY"));g.setAttribute(p3, name, v.setString(“MARY"));…
int since = g.newAttribute(friend, “SINCE”, Integer, Indexed);
g.setAttribute(e1, since, v.setInt(2000));g.setAttribute(e2, since, v.setInt(1995));…
JOHN18
KELLY
MARY
1995
2000JOHN18
KELLY
MARY
Manage attributes example
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Database construction
…int phones = g.newEdgeType("phones“, true, true);int when = g.newAttribute(phones, "when",
String, Indexed);
long e4 = g.newEdge(phones, p1, p3);g.setAttribute(e4, when, v.setString("4pm")));
long e5 = g.newEdge(phones, p1, p3);g.setAttribute(e5, when, v.setString("5pm"));
long e6 = g.newEdge(phones, p3, p2);g.setAttribute(e6, when, v.setString("6pm"));…
g.getAttribute(p1, name, v);System.out.println(v);g.getAttribute(e5, when, v);System.out.println(v);g.getAttribute(e4, when, v);System.out.println(v);
1995
2000JOHN18
KELLY
MARY
4pm5pm
6pm
Manage attributes example
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Index
Introduction
Basic Concepts
Database construction
Query database
Loaders
Script loaders
Tips & tricks
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Query database
Manage node and edge types
int Graph#findType(String name)Returns the Dex type identifier for the given type name.
Type Graph#getType(int type)Returns the metadata for the given Dex type identifier.
TypeList Graph#findTypes()Returns the list of all existing Dex type identifiers.
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Query database
Manage attributes
int Graph#findAttribute(int type, String name)Returns the Dex attribute identifier for the given Dex type identifier and attribute name.
Attribute Graph#getAttribute(int attr)Returns the metadata for the given Dex attribute identifier.
AttributeList Graph#findAttributes(int type)Returns the list of all existing Dex attribute identifiers for the given Dex type identifier.
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Query database
Objects
class ObjectsUnordered set of OIDs for large collections.Implements Set<Long>, Iterable<Long>.
boolean Objects#add(long oid)Adds the given OID to the collection.Returns true if added, false if the OID was already into the collection.
boolean Objects#exists(long oid)Returns true if the given OID exists into the collection, false otherwise.
boolean Objects#remove(long oid)Removes the given OID from the collection.Returns true if removed or false if the OID was not into the collection.
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Query database
Objects
long Objects#union(Objects objs)this = this UNION objsReturns the new size of the collection.
long Objects#intersection(Objects objs)this = this INTERSECTION objsReturns the new size of the collection.
long Objects#difference(Objects objs)this = this DIFFERENCE objsReturns the new size of the collection.
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Query database
Objects Graph#select(int t)Retrieves object identifiers belonging to the given node or edge type.
Objects Graph#select(int attr, Condition c, Value v)
Retrieves object identifiers which satisfy the condition for the given Value.“c” can be:
Equal, NotEqual, GreaterEqual, GreaterThan, LessEqual,LessThan, Between.Also, for String attributes: Like, LikeNoCase, RegExp.
long Graph#findObject(int attr, Value v)Randomly retrieves an object identifier which has the given value for the given attribute (or Objects.InvalidOID if not found).Useful for unique attributes.
Retrieve data
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Query database
Objects Graph#explode(long oid, int edgeType, EdgesDirection dir)
Retrieves out-going or in-going edges (or both) from or to the given node identifier and for the given edge type.“dir” can be:
Ingoing, Outgoing, Any.
Objects Graph#neighbors(long oid, int edgeType, EdgesDirection dir)
Retrieves neighbor nodes to the given node identifier which can be reached through the given edge type and direction.“dir” can be:
Ingoing, Outgoing, Any.
Navigation
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Query database
…Graph g = s.getGraph();Objects persons = g.select(person);ObjectsIterator it = persons.iterator();while (it.hasNext()) {
long p = it.next();dbg.getAttribute(p, name, v);String name = v.toString();
}it.close();persons.close();…
JOHN18
KELLY
MARY
JOHN18
KELLY
MARY
1995
4pm
5pm
6pm
2000
Retrieve data example
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Query database
…Objects objs1 = g.select(when, GreaterThan, “5pm”);// objs1 = { e5, e6 }Objects objs2 = g.explode(p1, phones, Outgoing);// objs2 = { e4, e5 }objs1.intersection(objs2);// objs1 = { e5, e6 } ∩ { e4, e5 } = { e5 }…objs1.close();objs2.close();…
JOHN18
KELLY
MARY
JOHN18
KELLY
MARY
1995
4pm
5pm
6pm
2000
Navigation & Objects operations example
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Index
Introduction
Basic Concepts
Database construction
Query database
Loaders
Script loaders
Tips & tricks
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Loaders Package com.sparsity.dex.io
NodeTypeLoader Requires a RowReader
CSVReader … or write your own implementation.
Creates a node instance for each row and sets its attributes with the values within each column.
EdgeTypeLoader Requires a RowReader, too. Creates an edge instance for each row and sets its
attributes with the values within each column. Two special columns to identify source and target
nodes for the edge.
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Index
Introduction
Basic Concepts
Database construction
Query database
Loaders
Script loaders
Tips & tricks
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Script loaders
(CREATE|USE) GDB alias INTO ‘filename‘
CREATE NODE node_type_name "(“[attribute_name(INTEGER|LONG|DOUBLE|STRING|BOOLEAN|TIMESTAMP|TEXT)[INDEXED|UNIQUE|BASIC], ...]
")“
CREATE [UNDIRECTED] EDGE edge_type_name[FROM node_type_name TO node_type_name] "(“
[attribute_name(INTEGER|LONG|DOUBLE|STRING|BOOLEAN|TIMESTAMP|TEXT)[INDEXED|UNIQUE|BASIC], ...]
") [MATERIALIZE NEIGHBORS]"
Schema definition
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Script loaders
LOAD NODES ‘file_name’ [LOCALE loc]COLUMNS attribute_name [alias_name], …INTO node_type_name[IGNORE (attribute_name|alias_name), …][FIELDS
[TERMINATED char][ENCLOSED char][ALLOW_MULTILINE [max]]]
[FROM num][MAX num][MODE (ROWS|COLUMNS [SPLIT [PARTITIONS num]])]
Load nodes
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Script loaders
LOAD EDGES ‘file_name’ [LOCALE loc]COLUMNS attribute_name [alias_name], …INTO node_type_name[IGNORE (attribute_name|alias_name), …]WHERETAIL (attribute_name|alias_name) = node_type_name.attribute_nameHEAD (attribute_name|alias_name) = node_type_name.attribute_name[FIELDS
[TERMINATED char][ENCLOSED char][ALLOW_MULTILINE [max]]]
[FROM num][MAX num][MODE (ROWS|COLUMNS [SPLIT [PARTITIONS num]])]
Load edges
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Script loaders
create gdb WIKIPEDIA into 'wikipedia.dex' create node TITLES (
ID int unique, 'TEXT' string, NLC string, TITLE string indexed
) create node IMAGES (
ID int unique, NLC string, FILENAME string indexed
)create edge REFS from TITLES to TITLES (
NLC string, "TEXT" string, TYPE string
) materialize neighborscreate undirected edge IMGS
Examples
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Script loaders
use gdb WIKIPEDIA into 'wikipedia.dex' load nodes 'images.csv'
columns ID, NLC, FILENAME into IMAGES from 2 max 10000
load edges 'references.csv' columns NLC, 'TEXT', TYPE, FROM F, TO T into REFS ignore F, T where
tail F = TITLES.ID head T = TITLES.ID
fieldsterminated ‘|’enclosed ‘”’allow_multiline
mode columns split partitions 3
Examples
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Index
Introduction
Basic Concepts
Database construction
Query database
Loaders
Script loaders
Tips & tricks
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Tips & tricks
Index or not? Attributes:
Attributes used at select operations must be indexed.
Optionally, index once the attribute has been created/loaded.
Neighbors: It is recommended to index those
edge types used at neighbors operations.
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Tips & tricks
String attributes String
Maximum length = 2047. Indexed or not.
Select [==, !=, >, >=, <, <=, Like, LikeNoCase, RegExp]
Text (Character large object) Unlimited length. Not Indexed.
Just get and set. Streaming read and write operations.
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Tips & tricks
Others: DB cross-platform format.
32 – 64 bits, OS independent. Just take into account platform endianness.
Read only mode. Configuration:
com.sparsity.dex.gdb.DexConfig Set the maximum memory usage.
0 means unlimited. License.
No license means evaluation version.
Dex G
rap
h D
ata
base
http://www.sparsity-technologies.com
Thanks for yourattention
Any questions?
SPARSITY-TECHNOLOGIESJordi Girona, 1-3, Edifici K2M
08034 [email protected]
http://www.sparsity-technologies.com