Webinar- Couchbase Mobile Deep Dive

Couchbase  Mobile    Deep  Dive

Jens  Al4e  

jens@couchbase.com

V1.0 - 6/17/14

©2009  Helder  da  Rocha

Architecture  of  Couchbase  Lite  and  &  Sync  Gateway

• Document  data  store  Mul?-­‐Version  Concurrency  Control  (MVCC)  

• Replica=on  protocol  Conflict  handling  

• Querying/filtering  Map/reduce  views  

• Client  APIs  REST  

Na?ve

Document  Store

Document  Database

• Persistent  map  from  IDs  →  documents  

• where  documents  are  JSON  objects  

• …plus  aLachments  Extra  set  of  proper?es  mapping  name  →  {binary  data,  MIME  type}  

• …with  mul=ple  revisions

Mul=-­‐Version  Concurrency  Control

• Each  document  has  a  revision  history  

• Each  revision’s  ID  has  a  genera=on  number  and  a  digest  e.g.  12-­‐c8c993737bf3e800b0a3d2e864c2039d  

• Dele=ons  are  just  special  revisions  “_deleted”:  true  

• Every  doc  update  must  specify  its  parent  revision  ID

Sequence  Index

• Every  revision  added  gets  a  sequence  number  

• Sequence  numbers  are  assigned  consecu=vely  

• Local  to  a  single  database  

• Documents  are  indexed  by  sequence  number  Abstract  ?meline  of  the  database’s  history  

Allows  asking  “what  changed  since  sequence  ____?”

Replica=on

What  Replica=on  Does

• Transfers  changed  documents  from  a  source  to  a  target  database  

• Incremental  

• Asynchronous  

• Peer-­‐to-­‐peer  

• Semi-­‐op=mized  Doesn’t  send  any  revision  that  the  target  already  has  

Doesn’t  send  any  adachment  version  that  the  target  already  has  

But  does  send  en?re  JSON  docs  &  adachments  (no  delta  compression)

Replica=on  Conflicts

• Conflicts  are  inevitable,  per  the  CAP  Theorem  

• Replica=ng  conflic=ng  revision  histories  produces  a  tree  

• The  document  model  allows  for  conflicts  A  conflict  is  a  valid  state  of  a  document  

• Conflict  resolu=on  is  the  app’s  job  Only  the  app  needs  to  care  about  resolving  conflicts  

App  can  use  any  logic  it  wants  to  pick  winners  or  merge  changes  

Resolu?on  creates  new  revisions  that  are  replicated

Jens — This is too

confusing. We really

need a diagram

here! Thx.

Replica=on  Conflicts  Cause  Branches

1-­‐abcd 2-­‐c124

1-­‐abcd 2-­‐c124 3-­‐beef 1-­‐abcd 2-­‐c124 3-­‐fade

1-­‐abcd 2-­‐c124

3-­‐beef

3-­‐fade

4-­‐dead

4-­‐feed

Resolving  a  Conflict

1-­‐abcd 2-­‐c124

3-­‐beef

3-­‐fade

Replica=on  Conflicts

• Conflicts  are  inevitable,  per  the  CAP  Theorem  

• Replica=ng  conflic=ng  revision  histories  produces  a  tree  

• The  document  model  allows  for  conflicts  A  conflict  is  a  valid  state  of  a  document  

• Conflict  resolu=on  is  the  app’s  job  Only  the  app  needs  to  care  about  resolving  conflicts  

App  can  use  any  logic  it  wants  to  pick  winners  or  merge  changes  

Resolu?on  creates  new  revisions  that  are  replicated

Replica=on  and  Peer-­‐to-­‐Peer

• It’s  more  than  just  push/pull  to  a  server  

• Any  two  databases  can  replicate  if  they  can  connect  Couchbase  Lite’s  “Listener”  component  allows  connec?ons  

• Arbitrary  topologies  (trees,  meshes)  are  supported

Replica=on  Protocol

• Extension  of  REST  API  

• One  ac=ve  partner  (client),  one  passive  (server)  

• Ac=ve  partner  can  push  or  pull  

• State  (checkpoint)  is  stored  redundantly  on  both  sides

Pull  Algorithm

• Verify  local  &  remote  checkpoints  

• Request  the  changes  since  checkpointed  remote  sequence  →  array  of  {sequence,  doc-­‐id,  rev-­‐id}  tuples  

• Determine  which  document  revisions  do  not  exist  locally  

• Download  leaf  revision(s)  of  each  such  document  Including  revision  history  

Skipping  unmodified  adachments  

• Update  the  checkpoints  

• In  con1nuous  mode,  keep  monitoring  remote  changes

Push  Algorithm

• Verify  local  &  remote  checkpoints  

• Send  IDs  of  revisions  made  since  checkpoint’s  local  sequence  Server  responds  with  the  subset  that  it  doesn’t  have  

• Upload  the  leaf  revision(s)  of  each  such  document  Including  revision  history  

Skipping  unmodified  adachments  

• Update  checkpoints  

• In  con1nuous  mode,  keep  monitoring  local  changes

Note to self: verify against https://github.com/couchbase/couchbase-lite-ios/wiki/Replication-Algorithm

Filtered  Replica=on

• Implemented  in  Couchbase  Lite,  CouchDB,  PouchDB,  etc.  

• App-­‐defined  filter  func=on  can  accept  or  skip  each  outgoing  revision  

• Always  runs  on  source  side  of  replica=on  Push:  on  client  side  

Pull:  on  server  side  

• Simple,  but  not  super  scaleable

Channel-­‐Based  Replica=on

• Implemented  by  Couchbase  Sync  Gateway  

• Documents  are  tagged  with  channel  IDs  

• User  accounts  have  access  to  sets  of  channels  

• Documents  are  indexed  by  channel  

• Pull  replica=ons  efficiently  look  up  docs  in  available  channels

Tell people to watch

the Sync Gateway

talk next week for

more details!

Map/Reduce  Views

Anatomy  of  a  View

• A  view  belongs  to  a  database,  has  a:  Name  

Map  func?on  

Reduce  func?on  (op?onal)  

Persistent  index  

Checkpoint  sequence  

• Views  exist  in:  Couchbase  Lite  

Couchbase  Server  

not  Sync  Gateway  

Also  CouchDB,  PouchDB,  etc.FYI: Last week’s iOS

webinar had better

slides explaining this

stuff.

Indexing  a  View

• For  each  document  updated  since  checkpoint  sequence:  1. Remove  all  index  rows  previously  generated  by  that  document  

2. Call  map  func?on  with  current  document  proper?es  as  parameter  

3. Each  ?me  emit()  is  called,  add  {key,  value,  docID}  tuple  to  the  index  

• Save  the  current  sequence  number  as  the  checkpoint  

• The  keys  in  the  index  are  sorted  as  JSON

Querying  a  View

• Is  its  checkpoint  sequence  the  database’s  latest  sequence?  If  not,  update  the  index  (as  per  previous  slide)  

This  can  be  suppressed  by  sejng  query’s  IndexUpdateMode  

• Fetch  a  range  of  {key,  value,  docID}  tuples  from  the  index  Honoring  startKey,  endKey,  skip,  limit,  reverse

Client  APIs

REST  API  —  The  Classic

• Originally  defined  by  Apache  CouchDB  

• Supported  by  Sync  Gateway  (with  slight  differences)  Server  side  of  replica?on  

API  for  app  servers  

• Supported  by  Couchbase  Lite  Listener  For  JavaScript  app  code  (e.g.  PhoneGap)  

Server  side  of  peer-­‐to-­‐peer  replica?on

REST  API

• Document  access  (CRUD):  PUT  /  GET  /  DELETE  

/dbname/docid  

• Query  a  view:  POST  /dbname/_design/name/_view/name  

• Replicate:  POST  /dbname/_replicate

Na=ve  APIs  by  Plalorm

• Objec=ve-­‐C:  iOS,  Mac  OS  X  

• Java:  Android,  Linux  

• C#:  Windows,  Windows  Mobile,  Mono          BETA  

• Common  API  design  across  languages  We  have  a  single  carefully  designed  language-­‐neutral  API  spec  

Matching  class  names,  method  names,  seman?cs  

Slight  differences,  to  remain  idioma?c

Point them to the

“Development with

Android / iOS”

webinars.

Na=ve  API:  Key  Classes

• Manager  

• Database  

• Document  

• View  

• Query  

• Replica=on

Oops, that’s all I wrote. Stall for time — maybe answer some questions?

THE  END

www.couchbase.com/webinars  

developer.couchbase.com/mobile/