Upload
thiago-alencar
View
550
Download
0
Embed Size (px)
Citation preview
Data Sync on iOS with Couchbase Mobile
(Or, building better UX / Apps with distributed databases and data synchronisation techniques)
http://ti.eng.br @thiago_sjcThiago Faria Alencar
February 17, 2016HERE Maps @ Berlin
What do you mean with User Experience anyway ?
Combination of technology to delivery the best user experience. That’s why people switch / opt to services: think about Blacklane, to Immobile scout, Soundcloud.. What can
you offer at the end of the day? Is it the best possible?
-Blacklane :)-Amazon-Airbnb
-Facebook-Spotify
-EtcDon’t “own" things, but redefine how user
experience music, transport,..
What do you mean with User Experience anyway ?
A little (real life) story
The coffee sales man in the middle of nowhere
Back in 2007..
Let’s go back in time< 1935 2007
Instant coffee for internal Brazilian
market.. let’s build a system for sales …
Product’s website at: www.cafesaojose.com.br
A website & mini ERP
A website & mini ERP
Basic LAMP Setup(Linux + Apache + Mysql + Php)
Good old times.. right?
Using the systemPedralva - Minas Gerais
Network coverage (as of today)
Oops, no connection? all data in the server miles away..
Back to 2016… (last weekend)
same old problems….
Transit routes (online-only feature)
Clear found/searched route, hop in the train
Ups, inside subway now..
1 min later.. hm.. couldn’t we show the past/found route?
Other examples
Spotify - offline advertising!
‘Hobby' projects (telemetry)
IoT
Even the most ‘connected' applications will experience network issues from time to time -> make them operate offline as much as possible
Network coverage getting better …
but as of today still far from ideal..
A data locality/expiration issue
Can we do better?
hell yes.
Recognise the problem..
“Git-like” for databases?
“A distributed database is a database in which portions of the database are stored on multiple computers
within a network.”
“Users have access to the portion of the database at their location so that they can access the data relevant to their
tasks without interfering with the work of others.”
Turns out, a similar set of features needed on of distributed databases (computer clusters) is desired to have at the end nodes
(mobile devices) > Same fundamental design decisions.
Analysing DB designs
Document / NoSql databases vs. relational databases
scalability / flexibility
CAP Theorem : set of basic requirements that describe any distributed system (not
just storage/database systems).
Consistency - All the servers in the system will have the same data so anyone using the system will get the same copy regardless of which server answers their request.
CAP Theorem
Availability - The system will always respond to a request (even if it's not the latest data or consistent across the
system or just a message saying the system isn't working).
CAP Theorem
Partition Tolerance - The system continues to operate as a whole even if individual servers fail or can't be
reached.
Document / NoSql databases vs. relational databases
Document / NoSql databases vs. relational databases
RDMS are more or less ‘all or nothing’, transaction-based, atomic operations oriented.
Distributed databases relaxes this ‘atomic' requirement in order to provide better scalability. It turns out this feature is useful for offline mobile apps as well as we’ll see when compared to relation traditional databases.
Document / NoSql databases vs. relational databases
‘NoSQL databases' improves scaling by relaxing consistency in favor of availability and partition tolerance.
A bit or market research..
Find the right tool to do the job
Intro to Couchbase • Open source• Cross-platform (really)• Open protocols• Backed by company, but small ‘lock-in’• Easy to get started• Small foot-print on client• Built-in sync handling• No ‘migration-hell’ (JSON docs)
How to solve the data sync problem?
Database per-user doesn’t scale
Instead, share a single database
Compare this to a “repository” in git: each subset of data would be like a “repo”.. not all users will have
access to all data (e.g. ACL with gitolite)
The ‘traditional' way of exchanging data..
Encapsulate all of it in a SSL connection and everything will be fine (that is, while your connection
works).
What is happening here?
Notice how in the request – response model, data is “routed” by the different API end-points: URI path will trigger a specific request handler in the server.
How such problems are solved during sync?
• Authentication• Authorization• Data validation / routing
How such problems are solved during sync?
• Data routing:
When another user connects to the same backend, we want only the documents belonging to him to be synchronised. The term used for this in Couchbase Mobile is data replication.
How such issues are resolved during sync?
That’s when Couchbase data orchestration features kicks in:
Not to depend on an active connection, what each data represents and the respective routing are described in the document’s content itself.
-Optimal solution sits neither at one or the other end of the spectrum - normally you need a mix of things : so normal http reqs still useful!
How such issues are resolved during sync?
For iOS, called ‘Couchbase Lite’
-Available for many other platforms.-Advantages of being a JSON database (no migration hell)-Sync gateway ‘like the git merge'
Document: the primary entity stored in a database is called a document instead of a “row” or “record”.
Views: ‘denormalised tables/indexes ’; dynamically generated from JSON objects
Queries: the action of looking up results from a view’s index
Some terminology in Couchbase Lite land..
(Roots in Map/Reduce programming model)
123456
CBLManager *manager = [CBLManager sharedInstance]; NSError *error;CBLDatabase* _database = [manager databaseNamed: @"conektapp_db" error: &error]; if (error) { NSLog(@"error getting database %@",error); }
NSDictionary* properties = @{@"user_id": @"joe", @"type": @"contactInfo", @"contactData": @“+49610234192"};
CBLDocument* document = [database documentWithID: [NSString stringWithFormat: @"%@:%@", type, user_id]];NSError* error;
if (![document putProperties: properties error: &error]) { [self handleError: error];}
Saving an object in Couchbase Lite
( For questions on this see more detailed articles at http://ti.eng.br )
123456
- (CBLQuery*) queryContactInfoFromUsername:(NSString*) user{ //1- createView CBLView * contactInfoView = [self.database viewNamed: @"contactDataByUsername"]; [contactInfoView setMapBlock: MAPBLOCK({ if ([doc[@"type"] isEqualToString: @"contactInfo"]) { if (doc[@"user_id"]) emit(doc[@"user_id"], doc[@"contactData"]); } }) version: @"2"]; }
Retrieve an object in Couchbase Lite
123456
- (CBLQuery*) queryContactInfoFromUsername:(NSString*) user{ //1- createView CBLView * contactInfoView = [self.database viewNamed: @"contactDataByUsername"]; [contactInfoView setMapBlock: MAPBLOCK({ if ([doc[@"type"] isEqualToString: @"contactInfo"]) { if (doc[@"user_id"]) emit(doc[@"user_id"], doc[@"contactData"]); } }) version: @"2"]; //2 - make the query CBLQuery* query = [contactInfoView createQuery]; NSLog(@"Querying username: %@", user); query.startKey = user; query.endKey = user; return query;}
Retrieve an object in Couchbase Lite
123456
- (CBLQuery*) queryContactInfoFromUsername:(NSString*) user{ //1- createView CBLView * contactInfoView = [self.database viewNamed: @"contactDataByUsername"]; [contactInfoView setMapBlock: MAPBLOCK({ if ([doc[@"type"] isEqualToString: @"contactInfo"]) { if (doc[@"user_id"]) emit(doc[@"user_id"], doc[@"contactData"]); } }) version: @"2"]; //2 - make the query CBLQuery* query = [contactInfoView createQuery]; NSLog(@"Querying username: %@", user); query.startKey = user; query.endKey = user; return query;}//run, enumerateCBLQuery *contactQuery = [myObject queryContactInfoFromUsername: @"thiago"];CBLQueryEnumerator* result = [contactQuery run: &error];for(CBLQueryRow* row in result){ NSLog(@"Found document: %@", row.document);}
123456
NSURL* url = [NSURL URLWithString: @“http://yourapp.com/sync_gateway/"];
CBLReplication *push = [database createPushReplication: url];CBLReplication *pull = [database createPullReplication: url];push.continuous = pull.continuous = YES; [pull setCookieNamed:@"SyncGatewaySession" withValue:sessionValue path:nil expirationDate:nil secure:NO];[push setCookieNamed:@"SyncGatewaySession" withValue:sessionValue path:nil expirationDate:nil secure:NO]; NSLog(@"startSync");[pull start];[push start];
Getting replication running.. client side
The server-side
Sync gateway configuration
Located in the sync gateway, the 'sync function' is responsible for document authorisation, validation, routing.
The server-side
Sync gateway configuration123456789101112
{ "interface":":4984", "adminInterface":":4985", "log":["REST"], "databases":{ "sync_gateway":{ "server":"http://localhost:8091", "bucket":"app_bucket", "sync":`function(doc) {channel(doc.channels);}` } }}
sync function, not much defined for now.. will grow according to your application’s needs
When the sync function kicks in?
More on the sync function..
function(doc, oldDoc){
requireUser(oldDoc.owner);channel(doc.channel);access(doc.members, doc.roomID);
}
Validation
More on the sync function..
function(doc, oldDoc){
requireUser(oldDoc.owner);channel(doc.channel);access(doc.members, doc.roomID);
}
Routing
More on the sync function..
function(doc, oldDoc){
requireUser(oldDoc.owner);channel(doc.channel);access(doc.members, doc.roomID);
}
AccessControl
About the concept of “Channels”
App server tags Documents on entry; Docs are indexed by channel for replication.
Summary
Data scalability: Replicate only docs relevante to respective user.
Access Control: Replicate only documents visible to this user
Data Validation: Replicate only documents that contains valid information
Syncing events.. (change notifications)
Web Bonus
Web
Others
Bonus
Web
Others
Bonus
Thank you :)
Finally we have a way to automate data synchronisation!! Please use it! :)
And let’s shift our designs / apps away from the old "LAMP style” into something that actually works in real life
scenarios also when 'on the go'..
Q&A
One more slide to show how a more ‘elaborated' sync function may look like..
function (doc, oldDoc) { if (doc._deleted) { // Only editors with write access can delete documents: requireRole("role:editor"); requireUser(oldDoc.writers); // Skip other validation because a deletion has no other properties: return; } // Required properties: if (!doc.title || !doc.creator || !doc.channels || !doc.writers) { throw({forbidden: "Missing required properties"}); } else if (doc.writers.length == 0) { throw({forbidden: "No writers"}); } if (oldDoc == null) { // Only editors can create documents: requireRole("role:editor"); // The 'creator' property must match the user creating the document: requireUser(doc.creator) } else { // Only users in the existing doc's writers list can change a document: requireUser(oldDoc.writers); // The "creator" property is immutable: if (doc.creator != oldDoc.creator) { throw({forbidden: "Can't change creator"}); } } // Finally, assign the document to the channels in the list: channel(doc.channels);}