Couchbase Integration: iPhone/Android/Web

Catherine Woolgar23/03/2015

Who We Are - Brief History (1)● Founded a company called PhoneAPlumber 2013

o Automated call menu for capturing customer leads and sending them to Tradesmen via text

● Lessons we learnt:o Customers do not like automated call menus (generally)o Tradesmen have an issue with taking payments and can amass bad debts

(10% of sole traders take card payments)o Tradesmen can be disorganised - receipts on the dashboard, paper written

invoices, verbal quotes, disorganised or non existent accounts or several people in the office working on processing jobs from pieces of paper and payments supplied by workers, leads manually or verbally disseminated

● Lead generation market is crowded / well served but Tradesmen and in particular Sole Traders and small businesses were not being served

● Created a small ios/Android apps and Web Dashboard for Tradesmen to take payments via GoCardless

Who We Are - Brief History (2)● Pivoted our business and started developing SmartTrade App as a 1 stop solution

for Tradesmeno Took initial feedback and experience from working with the Tradesmen we

were working with to create SmartTrade App 2.0● Where we are now:

o We are backed by UK Tech and public market investors: EC1 Capital, Zeus and Novum

o Targeted users are: Mobile Workers, Sole Traders and Small Business Ownerso Our focus is largely on distribution deals with some of the UK and Europe’s

leading Merchants Organisationso Current release SmartTrade App 3.0 available on Google Play, Amazon,

iTunes, www.smarttradeapp.com

What Does SmartTrade App Do?● Secure instant card capture with SmartPhone or

Tablet means no expensive hardware unlike current card reader providers

● Cheaper than current Payment providers such as IZettle, Payleven with option to pass through fees to customers

● Also offers over the phone processing unlike other card providers on the Web Dashboard

● Better cash-flow for merchants with 2 day card clearance Q2 2015

● No merchant account or trading history needed unlike current card reader providers which translate to weeks of delay and reject

Office In Your PocketFrom Job Leads

● Receive leads from back office or from 3rd party Job Lead Providers

● Via Merchant Website to app● Developers connect via our

APIIncreases Merchant sales

Leads to an Active Job● Send Quotes and see them

approved by customer● Send Invoices with pre-filled

costs● Add notes to a job and a

customer● Plan visits● Keep Customer details in 1

place (sync with contact lists ios and Android

● Paperless paperworkNo more paper. A new SME efficiency

Through To Payments● Multiple payment options: Credit and Debit

Card, Cash and PayLinks● Also offers 'Over The Phone' processing

unlike other card providers via the Web Dashboard

● System allows you to do as little or as much as you like

Liberates any SME accepting cards

Even Accounting & Tax● Easy Bookkeeping ● Complimentary back

office web management● Future integrations with

Sage, Xero..Removes need for Book-keeper

● Co-marketing through Accounting and SME Partner Channels such as Training Colleges

A new efficiency for Accountants

SmartTrade App Space

Current Version: Technology (1)● Python Django Web Application ● Django closely developed with MYSQL server● Celery package is used for time consuming operations i.e. sending emails / push

notification (rabbitmq backend)● pusher.com for dashboard notifications and parse.com for mobile notifications● Published an API using Tastypie (Django package) used in mobile applications and

available for use by our partners and other developerso JSON REST API (via HTTPS)

● AWS SES (email service)

● Digital Ocean servers

● Ansible for deployment to multiple services

● Native iOS app, native Android app (data models and memory storage)

● ElasticSearch for full text search through database

Current Version: Technology (2)● Twistd based persistent process to make additional handling of documents coming

from mobiles (using _changes API of sync-gateway) to guaranteed unique Job and Invoice numbers in the system o The process is supported by supervisord and is restarted in case of

disconnection

Why Couchbase?

Why Couchbase?

Why Couchbase?● Elastic scalability in conjunction with global deployment, i.e. it is easy to add new

nodes in different geo-locations which are then seamlessly synchronizedo Fast replication

Why Couchbase? - The Killer FeatureData mobility: offline mobile applications — necessary for the man in the van

And as an added bonus: real time updates pushed to mobile applications — my app is alive no more pull to update and progress bars!!

The Killer Feature (cont)● No more backend calculations:

o Now all financial calculations are performed offline on apps by extracting them to a C++ library which has Objective-C wrapper for iOS JNI / NDK (Java) wrapper for Android Python binding for server side calculations Emscripten Javascript wrapper for real time calculation functionality in

web browsers

And Lastly...● Well Supported and documented along with being used by larger

companies: Zynga, BskyB, eBay etc.● Great company, great people with awesome tech we love using

Couchbase Integration

Data Flow

Web App

Couchbase server

Sync-gateway

Mobile App(Couchbase Lite DB)

Web Browser

Create / Update

Query Views / Search

(read-only)

Get entire documents

1. CRUD operations are performed via sync-gateway API for all apps (also for Web App as the sync gateway ignores documents saved directly via the Couchbase server)

2. A revision number must be provided to save existing documents via sync-gateway.

3. Search (read-only access) performed directly via Couchbase API on the Couchbase server - retrieves UIDS for documents enabling creation of data necessary for front end views (Web App)

4. Entire documents retrieved from sync-gateway which contain revision numbers saving individual documents

Sync based on channels defined in documents

(Basic Auth / HTTPS)

Auth API requests

Authentication for Mobile

Web App(MYSQL DB)

Sync-gateway

Mobile AppSync

Signup / Signin API requests

Sign Up

Server1. Create Web App user record (MYSQL)2. Generate Web App API key (MYSQL)3. Create sync-gateway user using Web App API key

as password (sync-gateway)4. Create some user related documents (sync-

gateway)5. Return Web App API key.

Mobile6. Setup synchronization using username and

received API key7. Wait for synchronisation of the user documents

(optional)

Sign In

Server1. Check credentials (MYSQL)2. Return Web App API key

Mobile3. Setup synchronization using username and

received API key4. Wait for synchronisation of the user documents -

we wait for the Account document to be synced

Why use MYSQL still?● Django supports MYSQL natively● MYSQL DB not going to be synchronised with

mobile DB● It works very well

Create user on signup

What we learnt: Backend (1)Sync-gateway

1. If project involves sync-gateway then reading of documents must be done using sync-gateway API (_all_docs?include_docs=true). Otherwise you don’t have revision numbers and can not save documents.

2. View queries still can be done using very handy client library functions (Couchbase API).

Unit-testing & Development3. Consider stale=false parameter for querying views (created on Couchbase server) in your unit

tests. Create and immediate read of a document is typical for unit-tests, which can otherwise fail as the index is not updated.

4. It can be necessary for website development: new records won’t appear in the lists.5. Create a separate bucket for testing purposes - have a separate database for your testing purposes. 6. Consider vagrant to run Couchbase / sync-gateway on your local machine if it is not in the list of

supported platforms: MAC-OS 10.10, Ubuntu 14.x, etc.7. Couchbase can run on latest Ubuntu without problems. At least it can be used for development.8. Data filtering should be done on client side where possible ie. when the data set is “not large”:

a. less views, easier to develop / debug (with TDD)

What we learnt: Backend (2)

OtherDjango-models and other utilities such as a library for accessing the Sync-Gateway using Python will be available open-source soon on GitHub for anyone interested

What we learnt: Mobile● If you are using Couchbase in an existing app and swapping over from using REST

API number of things to consider:o Couchbase models are great in ios but for quick integration use your existing

data models especially if you use JSON data (dictionaries / collections) to create an instance of your data class CBLDocument has a “properties” property which returns a dictionary /

JSON format data (ios)

o Views are great although querying is a little inflexible right now in Couchbase SDK We needed to filter 6 properties in our Job List - Couchbase views have a

limit of 5 columns which can be mapped in the view Complex queries are tricky and we ended up relying on our old code base

What we learnt: Mobile (2)● If your UITableViews are complex and require multiple filters / sorting: for fast

integration use native features rather than go for CBLLiveQuery as a datasource with Couchbase tableso We use CBLQuery and create arrays of data which is quick enough with the

amount of data we need to display● CRM App Sample was a good basis to learn the functionality● Logging out of the application: we drop the database and re-create it on log in

o Considering in a future release to drop the database on login if the user credentials are different from the last time they logged in or are signing up for another account

● Using listeners makes life easy - data can be updated in real timeo however printing to the console can kill performance especially if you are

dealing with large data sets and you are listening to change on CBLDatabaseo Large number of listeners also can slow down your application

What we learnt: Mobile (3)● Give special attention to the cases when document properties are set to null or do

not exist in the document at all. As it’s a great source of bugs and performance problems o Warnings triggered by your application or Couchbase framework can slow the

app performance as they are written to the debug console● Android Apps: if the operating system needs to alloc memory to other apps, it can

stop your activities resulting in database and datasource instances which are no longer availableo write getters to better handle access to documents which check whether the

instances exist and re-build where required

Where We Are Now

Current Tasks● QA of entire system (final stages)● Conflict management - we use a soft delete on documents and using

Couchbase merge functionality until business requirements are provided● Using N1ckel for querying data● Integrating the ElasticSearch plugin for search functionality on the Web

Dashboard● 4.0 Couchbase Integration Release date: middle of April

Multi User version to follow in May /June● Use of channels to manage job deployment to members of an account● Dashboard UI additions and enhancements for job deployment