Upload
neo4j-the-fastest-and-most-scalable-native-graph-database
View
87
Download
1
Embed Size (px)
Citation preview
PowerPoint Presentation
Building a Scalable Platform for Sharing 500 Million PhotosWouter Crooy & Ruben HeusinkveldSolution Architect & Technical Lead, Albumprinter
Ruben1
Wouter CrooySolution Architect, Albumprinter @wcrooy
Ruben2
Ruben HeusinkveldTechnical Lead, Albumprinter @rheusinkveld
Ruben3
Who are weWouter Crooy Solution ArchitectRuben Heusinkveld Technical LeadNeo4j Certified Professionals
Ruben
At Albelli we want to inspire people to relive and share lifes moments by easily creating beautiful personalized photo products.Vision: To brighten up the world by bringing peoples moments to life.
Albumprinter is a Cimpress company. The most known brand here in the US for Cimpress is Vistaprint. Im sure youve all know it.Albumprinter is based in Amsterdam, The Netherlands.We have multiple consumer brands to serve the European marketAlbumprinter aquired FotoKnudsen in June 2014
4
The photo organizerDeliver well organized, easy to use and secure storage for all your imagesEase the process of selecting photos for creating photo productsStarted as part of a R&D Skunk works project
Ruben
Goal: Deliver well organized, easy to use and secure storage for all your imagesBuild by team of 5 (1 designer, 1 frontend developer, 1 quality engineer, and Wouter and myself focusing on the backend)
5
The photo organizer
RubenLaunched June of this yearAvailable on all devices6
The photo organizer
RubenPhotos are automatically grouped together into events7
The photo organizer
Ruben:Easy to share photos with friends or publicly if you wantPrivately via invites8
The photo organizer from photos to products
Ruben:The photos can be used to create any product like a photo book, calendar or wall decor9
The photo organizer demo
https://minnebanken.no
Ruben:The photos can be used to create any product like a photo book, calendar or wall decor10
The challenge
Wouter11
The challengeReplace legacy system with the new photo organizerMove 1.3 PB of photos from on premise to cloud storageAnalyze & organize all photos (511 million)Data cleansing while importingUsing the same technology / architecture during import and afterAbility to add features while importingCore of the systems are built in .NET
WouterNot uploading duplicates12
The importHard deadlineFactory closing that holds the data center with all photosStarted 1st of AprilMinimum processing of 150 images / second~500 queries / second to Neo4jUp to 700 EC2 instances on AWS
Wouter13
How we did itMicro servicesCommand Query Responsibility Segregation (CQRS)ClusterMultiple write nodesSingle master read only nodesHAProxyCypher only via REST interface.NET Neo4jClient
Wouter14
Architecture
WouterIn Neo4j we only store the metadata. The actual photos are stored in Amazon Simple Storage Service (S3).15
Why we choose Neo4jClose to domain modelNot an ordinary (relational) databaseLooking for relations between photos/usersScalableFlexible schema Natural / fluent queriesACID / data consistency
Wouter16
The design
Ruben17
Graph model
Ruben18
Graph model
Ruben19
Graph model
Ruben
20
Our Neo4j databaseMore than 1 billion nodes4.1 billion properties2.6 billion relationsTotal store size of 863 GB
RubenFor all those photos this resulted in:More than 1 billion nodes4.1 billion properties2.6 billion relationsTotal store size of 863 GB
21
Command Query Responsibility SegregationSeperation between writing and reading dataDifferent model between Query and Command APIIndependent scaling
RubenI know its really ambitious to explain CQRS within 2 slides. But I would still like to explain why and how it could work with Neo4j.
Events sourcing. Double update to db and cache. In our case we used a cache update/flush on certain rules. Pro: Less work, database is to large for cache.Con: Not always reliable cache source. 22
Bumps and Solutions
Wouter23
CQRS Seperate Reads & WritesNo active event publishing in placeSpecific scenarios for updating / writing dataAbility to create seperate model for read and writeUpdates (pieces) the user graphRequires reliable and consistent readScale out -> overloading locking of (user) graphAfter importLow performance scenarios -> cache with lower update priority
WouterNeo4j in its core is very capable of handling CQRS interfaces. Since youre not updating a table but (parts) of the graph. Due to its ACID nature is should also be able to make sure there are no race-conditions. But since this archicture allows to massively scale out that does not always match the capebilities of a ACID DB. Especially in the cases where the writes are more occuring then the reads.
Make sure the read is consistentIn our situation, CQRS is extra complex since we have a ordered crawler (5+ steps) which also does the writes. But the crawler(s) and query api are still allowed to do reads.
https://www.infoq.com/news/2015/05/cqrs-advantageshttp://udidahan.com/2011/04/22/when-to-avoid-cqrs/http://udidahan.com/2009/12/09/clarified-cqrs/http://udidahan.com/2010/08/31/race-conditions-dont-exist/
See also consistent read solution. In cases were we dont need to have consistsent read we can use the case. 24
Read after write consistencyAll reads should contain the very latest and most accurate dataReplication delay between serversSplit on consistency
Article by Aseem Kishore:https://neo4j.com/blog/advanced-neo4j-fiftythree-reading-writing-scaling/
WouterRead fastly outnumber writes in our application as for many applications.Split on consistency, not read vs. writeTrack user last write time for read after write consistencyMonitor and tune slave lag, via push/pull configsStick slaves by user for read after read consistencyhttps://neo4j.com/blog/advanced-neo4j-fiftythree-reading-writing-scaling/
Credits to Aseem Kishore and his team at FiftyThree for sharing this on the conference last year.
25
Graph lockingConcurrency challengeScale-out => more images from the same userManage the inputHigh spread of user/image combinationPrevent concurrent analysis of multiple images from the same user:GET /db/manage/server/jmx/domain/org.neo4j/instance%3Dkernel%230%2Cname%3DLocking
Wouter
Mainly during the importing of photos
{"description" : "org.neo4j.kernel.info.LockInfo","type" : "org.neo4j.kernel.info.LockInfo","value" : [{"name" : "description","description" : "description","value" : "ExclusiveLock[\nClient[1] waits for []]"}, {"name" : "resourceId","description" : "resourceId","value" : "2612184871"}, {"name" : "resourceType","description" : "resourceType","value" : "RELATIONSHIP"}]}26
Batch insert vs single insertCypher CSV import per 1000 recordsPrevent locking caused by concurrency issues
Wouter27
No infinite scale outFind the sweet spot for the amount of cluster nodes+1 nodes => more replications updates => higher load on write master
Wouter28
TimelineWere looking for photos which should belong to each other based on date-taken. Moving from full property scan to graph walking via the timeline. For large collection 75% less DB-hitsWalking the timeline if looking for photos within a certain timeframeLess photos to evaluate for property scan (SecondsSinceEpoch)Works perfectly for year, month, day selections
Wouter29
.NET & Rest interfaceCustom headers to REST Cypher endpoint (Filtered by HaProxy)To route to multiple write serversSticky session per userCustom additions to .NET Neo4jclientManaging JSON resultset
Wouter30
Graph design considerationsProperty scan(User) full-graph-scanDifferentiating propertyCreate nodeNo path/clustered indexes. (yet.. )
Making changes to the schema. For 550+ million nodes
Wouter31
Graph design improvementsProperty searchmatch (u:User { Id: 001"}) 2812 db hits
Node/Relationship searchmatch (u:User { Id: "001"})-[:HasFavourites]-(f:Favourites) 13 db hitsdbms.logs.query.* (dont forget to enable parameters resolving)Our alternative: Integrate with Kibana / Elasticsearchhttps://neo4j.com/docs/operations-manual/current/reference/
WouterDB hits increase when the number of photos increases if you do the property search32
The future
Wouter?33
The futureNeo4j 3.xBoltDataminingProcedures / APOC
Wouter?34
Thats a wrapWouter?35