GraphConnect Europe 2016 - Enterprise Data Integration with a new JDBC Driver for Neo4j 3.0 - Lorenzo Speranzoni

/ 50

LONDON, 26 April 2016

1

LORENZO SPERANZONI | @inserpio CEO @ LARUS Business Automation - GCE2016 BRONZE SPONSOR

GraphConnect Europe 2016

Explore the new COUCHBASE and JDBC

integrations for Neo4j 3.0

/ 50LONDON, 26 April 2016GraphConnect Europe 2016

EXPLORE THE NEW COUCHBASE & JDBC INTEGRATIONS FOR NEO4J 3.0

AGENDA

✦ Let’s say Hi ( “Ciao” !! )

✦ Neo4j - Couchbase Connector

✦ JDBC Driver with BOLT for Neo4j 3.0

2



AGENDA




3



WELCOME AND

THANKS FOR COMING TO THIS LIGHTNING TALK

4



5

[:WORKS_FOR { role: ‘CEO’ }]

Venice

[:BRONZE_SPONSOR]

Lorenzo Speranzoni

LARUS Business Automation

[:BASED_IN][:CERTIFIED_ON]

[:PARTNER_OF]



6

… this is a typical day in Venice George Clooney Wedding

BY THE WAY … … NOT THIS ONE DURING …

#*@!#!!



7

2016

Neo4j Contributors

20152011

First Spikes ( with neo4j 1.x !! )

2013

LARUS’ TRIP WITH NEO4J (quite a long and cool story)



8

CONSULTING

[:DELIVERS]

TRAININGS

SUPPORT

[:DELIVERS]

[:DELIVERS] [:FOR]

[:ON]

[:ON]

[:ADO

PTED

]



AGENDA




9



The new Neo4j - Couchbase Connector

10



KEY ASPECTS

✦ BI-DIRECTIONAL

✦ CONFIGURABLE

11



https://developer.spotify.com/web-api/console/get-album

12

CA

SE S

TUDY




https://developer.spotify.com/web-api/console/get-album [ RESPONSE ]

13

CA

SE S

TUDY




BASIC JSON 2 GRAPH CONVERSION RULES

PLEASE NOTE.

✦ Most of the following approaches come from William Lion’s (@lionwj)MongoDB - Neo4j Connector (http://neo4j.com/blog/neo4j-doc-manager-polyglot-persistence-mongodb/)

✦ Don’t miss William’s session “NoSQL Polyglot Persistence: Tools and Integrations” here at GCE2016 (4:30PM), for many more examples about managing multiple databases.

14

https://twitter.com/lionwj

http://neo4j.com/blog/neo4j-doc-manager-polyglot-persistence-mongodb/



PRIMITIVE ATTRIBUTES …

15

album_type: “U2” href: “https://api.spotify.com/v1/albums/586Z…” id: “586ZRfgsIckfcKvHVcGM4V” name: “The Joshua Tree” popularity: 53 release_date: “1987-03-10” release_date_precision: “day” type: “album” uri: “spotify:album:586ZRfgsIckfcKvHVcGM4V”

ALBUM

BECOME NODE’S ATTRIBUTES



ARRAY OF PRIMITIVE ATTRIBUTES …

16

album_type: “U2” available_markets: [ “MX”, “US” ] href: “https://api.spotify.com/v1/albums/586Z…” id: “586ZRfgsIckfcKvHVcGM4V” name: “The Joshua Tree” popularity: 53 release_date: “1987-03-10” release_date_precision: “day” type: “album” uri: “spotify:album:586ZRfgsIckfcKvHVcGM4V”

ALBUM

STILL BECOME NODE’S ATTRIBUTES



OBJECT ATTRIBUTES . . .

17

href: “https://api.spotify.com/v1/artists/51B…” id: “51Blml2LZPmy7TTiAg47vQ” name: “U2” type: “artist” uri: “spotify:album:51Blml2LZPmy7TTiAg47vQ”

ARTIST

BECOME CONNECTED NODES

name: “The Joshua Tree”

ALBUM

[ :ALBUM_ARTIST ]



ARRAY OF OBJECTS ATTRIBUTES …

18

ARTIST

AGAIN BECOME CONNECTED NODES

ALBUM disc_number: 1 duration_ms: 337973 href: “https://api.spotify.com/v1/tracks/3zq…” id: “3zqTlJtp7KTPrNUKNahvAo” name: “Where The Streets Have No Name” uri: “spotify:album:3zqTlJtp7KTPrNUKNahvAo” track_number: 1 type: “track”

ITEM

TRACKS

ITEM

2ITEM

3ITEM

11

ITEM

4

[ :TRA

CKS_IT

EMS ]



Neo4j - Couchbase Connector

COMPONENTS (and a bit of noisy code)

19



Neo4j - Couchbase Connector Components

20



1. Couchbase Mutation Listener public class CouchbaseEvent {

/** * Wrapped {@link CouchbaseMessage}. */

private CouchbaseMessage message;}

public class CouchbaseEventFilter {

/** * Returns true if event is a {@link MutationMessage}. */

public static boolean accept(final CouchbaseEvent couchbaseEvent) { return couchbaseEvent.getMessage() instanceof MutationMessage; }}

21



1. Couchbase Mutation Listener public class CouchbaseEventHandler implements EventHandler<CouchbaseEvent> {

/** * Handles {@link CouchbaseEvent}s that come into the response RingBuffer. */ @Override

public void onEvent(final CouchbaseEvent event, final long sequence, final boolean endOfBatch) throws Exception { if (CouchbaseEventFilter.accept(event)) { MutationMessage mutationMessage = (MutationMessage) event.getMessage(); ByteBuf jsonDocument = mutationMessage.content(); DocumentTransformer<String> remoteNeo4jTransformer = new RemoteNeo4jTransformer();

remoteNeo4jTransformer.transform(mutationMessage.key(), “COUCHBASE_DOCUMENT", asString(jsonDocument)); } } }

22



1. Couchbase Mutation Listener public class CouchbaseMutationListener extends Thread {

/** * Builds and starts the event consumer. */

public void buildDisruptor() { Disruptor<CouchbaseEvent> disruptor = new Disruptor<CouchbaseEvent>(new CouchbaseEventFactory(), 32, Executors.newCachedThreadPool()); disruptor.handleExceptionsWith(new CouchbaseExceptionLogger()); disruptor.handleEventsWith(new CouchbaseEventHandler()); this.couchbaseEventBuffer = disruptor.start(); }

/** * Connect this dispatcher to Couchbase. */

public void connect() { this.couchbaseClusterFacade.send(new SeedNodesRequest(couchbaseNodes)).timeout(2, TimeUnit.SECONDS).toBlocking().single(); this.couchbaseClusterFacade.send(new OpenBucketRequest(this.couchbaseBucket, couchbasePassword)).timeout(2, TimeUnit.SECONDS).toBlocking().single(); }

}

23



2. Neo4j JSON Loader public class JsonDocument {

private String id; private String type; private String content; private JsonMappingStrategy mappingStrategy; private List<JsonObjectDescriptor> objectDescriptors;}

public enum JsonMappingStrategy {

ATTRIBUTE_BASED(“attribute”), DOMAIN_DRIVEN(“domain”), STORED_PROCEDURE(“stored procedure”),}

24

public class JsonObjectDescriptor { private String entityName; private List<String> uniqueKeyAttributes; private String typeAttribute;}



2. Neo4j JSON Loader @Path("/")public class JsonLoaderRestController {

private final GraphDatabaseService graphDatabaseService;

public JsonLoaderRestController(@Context GraphDatabaseService graphDatabaseService) { this.graphDatabaseService = graphDatabaseService; }

@POST @Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })

public Response loadJSON(JAXBElement<JsonDocument> jsonDocumentWrapper) {

JsonDocument jsonDocument = jsonDocumentWrapper.getValue(); JsonLoaderService loaderService = new DefaultJsonLoaderService(this.graphDatabaseService); QueryStatistics queryStatistics = loaderService.save(jsonDocument); return Response.ok(queryStatistics).build(); }}

neo4j-server.properties org.neo4j.server.thirdparty_jaxrs_classes=it.larusba.integration.neo4j.jsonloader.rest=/jsonloader

25



2. Neo4j JSON Loader

public class DefaultJsonLoaderService implements JsonLoaderService {

@Override public JsonLoaderStatistics save(JsonDocument jsonDocument) throws JsonParseException, JsonMappingException, IOException {

JsonTransformer<String> jsonTransformer = JsonTransformerFactory.getInstance(jsonDocument.getMappingStrategy()); String cypher = jsonTransformer.transform(jsonDocument);

try (Transaction tx = this.graphDatabaseService.beginTx()) {

Result result = this.graphDatabaseService.execute(cypher); jsonLoaderStatistics = new JsonLoaderStatistics(result.getQueryStatistics()); tx.success(); }

return jsonLoaderStatistics; }

26



3. Neo4j Mutation Listener public class Neo4jEventListener implements TransactionEventHandler<Void> {

private GraphDatabaseService db;

public Neo4jEventListener(GraphDatabaseService db) { this.db = db; }

public void afterCommit(TransactionData data, Void state) { List<JsonDocument> jsonDocuments = buildJSONDocument(data); if (CollectionUtils.isNotEmpty(jsonDocuments)) { CouchbaseJSONLoader.upsert(jsonDocuments); } }}

27



4. Couchbase JSON Loader public class CouchbaseJSONLoader

public static void upsert(List<JsonDocument> jsonDocuments) {

Cluster cluster = CouchbaseCluster.create(); Bucket defaultBucket = cluster.openBucket();

for (JsonDocument jsonDocument : jsonDocuments) { defaultBucket.upsert(jsonDocument); }

cluster.disconnect(); }}

28



Neo4j - Couchbase Connector

HOW ABOUT THE DOMAIN?

WOULDN’T BE COOL TO BE ABLE TO TRAIN THE “JSON LOADER” WITH THE BUSINESS DOMAIN

YOUR WORKING ON?

29



HOW ABOUT THESE INTERESTING ATTRIBUTES ?

30



31

WHY DON’T WE USE “ID” ATTRIBUTES TO UNIQUELY IDENTIFY NODES?

We could use them on MERGE statements: MERGE (album:ALBUM {id: ‘586ZRfgsIckfcKvHVcGM4V’}) MERGE (track:TRACK {id: ‘3zqTlJtp7KTPrNUKNahvAo’})



32

AND WHY DON’T WE USE “TYPE” ATTRIBUTES AS NODES’ LABELS?

MERGE (album:ALBUM {id: ‘586ZRfgsIckfcKvHVcGM4V’}) MERGE (track:TRACK {id: ‘3zqTlJtp7KTPrNUKNahvAo’})



DESCRIBE YOUR DOMAIN i.e. instruct how the JSON 2 GRAPH transformer should behave List<JsonObjectDescriptor> jsonObjectDescriptors = new ArrayList<JsonObjectDescriptor>();

jsonObjectDescriptors.add(new JsonObjectDescriptor("Album", Arrays.asList("id"), “type”));jsonObjectDescriptors.add(new JsonObjectDescriptor("Artists", Arrays.asList("id"), “type"));jsonObjectDescriptors.add(new JsonObjectDescriptor("Images", Arrays.asList("url"), null));jsonObjectDescriptors.add(new JsonObjectDescriptor("Tracks", Arrays.asList("href"), null));jsonObjectDescriptors.add(new JsonObjectDescriptor("Items", Arrays.asList("id"), “type”));

JsonDocument jsonDocument = new JsonDocument("U2TheJoshuaTreeAlbum", "Album", spotifyTheJoshuaTreeAlbumDocument, JsonMappingStrategy.DOMAIN_DRIVEN, jsonObjectDescriptors);

33

key type json strategy descriptors



DOMAIN DRIVER JSON TRANSFORMER public class DomainDrivenJsonTransformer implements JsonTransformer<String> {

@Override public String transform(JsonDocument jsonDocument) throws JsonParseException, JsonMappingException, IOException {

34



35



36

“DOMAIN DRIVEN” TRANSFORMERALLOWS YOU TO RE-USE“U2” and all other NODES

… SO THAT YOU GET

A REAL BENEFIT FROM RELATIONSHIPS



AGENDA




37



38

The new JDBC Driver with BOLT

Neo4j 3.0

APPLICATION

JDBC

EMIL’S OPENING KEYNOTE



Neo4j JDBC Driver: A-TEAM

39

Project Leader

HTTP Protocol

Tester

BENOIT SIMARD

STEFAN ARMBRUSTER

MICHAEL HUNGER

Dev & Project LeaderRALF BECHER



KEY ASPECTS

✦ MAKE USE OF THE NEW B.O.L.T. PROTOCOL

✦ MODULAR DESING (bolt, http, in-memory, file)

✦ HEAVILY TESTED

✦ FOCUS ON PERFORMANCE

40



41

1. Make use of the new BOLT protocol

Neo4j 3.0

NEO4J-JDBC

BOLT module

neo4j-java-driver

APPLICATION

https://github.com/neo4j/neo4j-java-driver

https://github.com/neo4j/neo4j-java-driver



42

2. Modular design

Neo4j 3.0

NEO4J-JDBC BOLT

HTTP

FILE

IN-MEMORY



3. Heavily tested

✦ ~300 unit tests

✦ ~50 integration tests

43

for ~60 classes



4. Focus on performance ✦ OLD: 30’20’’

✦ NEW: 17’11’’

44

30.000 CSV rows

WOW! ALMOST HALF TIME



45

INTEGRATE NEO4J WITH ALL YOUR FAVOURITE TOOLS AND PRODUCTS

Neo4j 3.0



Minimum Viable Snippet // Make sure Neo4j Driver is registeredClass.forName(“it.larusba.neo4j.jdbc.bolt.BoltDriver”);

// ConnectConnection con = DriverManager.getConnection(“jdbc:bolt://localhost”);

// Querytry (Statement stmt = con.createStatement()) { ResultSet rs = stmt.executeQuery(“MATCH (n:User) RETURN n.name”); while (rs.next()) { System.out.println(rs.getString(“n.name”)); }}

con.close();

46



47

RC1

Neo4j Contrib

M5M3 M4

Neo4j 3.0 JDBC Driver RELEASE PLAN

HTTP PROTOCOL java.sql.CallableStatement

FULLY RETRO-COMPATIBILITY

BOLT PROTOCOL

early autumn

PRODUCTION READY

TODAY AT GCE2016

early summer



HAVE FUN WITH US

✦ https://github.com/larusba/neo4j-jdbc

✦ https://github.com/larusba/neo4j-couchbase-connector

✦ https://github.com/larusba/neo4j-json-loader

48

https://github.com/larusba/neo4j-jdbc

https://github.com/larusba/neo4j-couchbase-connector

https://github.com/larusba/neo4j-json-loader



COME AND TALK WITH THESE GUYS!!! THEY DID ALL THIS STUFF & THEY’RE IN THE ROOM :-)

✦ Mauro Roiter (@MauroRoiter) ✦ Marco Falcier (@mfalcier) ✦ Alberto D’Este (@ziotobia) ✦ Gianmarco Laggia (@glaggia)

49

https://twitter.com/MauroRoiter

https://twitter.com/mfalcier

https://twitter.com/ziotobia

https://twitter.com/glaggia

/ 50

LONDON, 26 April 2016

50

LORENZO SPERANZONI | @inserpio CEO @ LARUS Business Automation - GCE2016 BRONZE SPONSOR

GraphConnect Europe 2016

THANK YOU

Technology

GraphConnect Europe 2016 - Enterprise Data Integration with a new JDBC Driver for Neo4j 3.0 - Lorenzo Speranzoni