Couche Base par Tugdual Grall

Wednesday, March 20, 13

Win some cool stuff, send a mail to

[email protected]

with NORMANDYJUG in the subject.


mailto:[email protected]


Technical Evangelist

twi0er: @tgrallemail: [email protected]

Tugdual “Tug” Grall

Introduc)on to NoSQLwith Couchbase

Normandy JUG -‐ March 19th 2013




About me

• Tugdual “Tug” Grall

-‐ Couchbase-‐ Technical Evangelist

-‐ eXo-‐ CTO

-‐ Oracle-‐ Developer/Product Manager

-‐ Mainly Java/SOA

-‐ Developer in consul)ng firms

•Web

-‐ @tgrall

-‐ hQp://blog.grallandco.com-‐ tgrall• NantesJUG co-‐founder

• Pet Project :

• hQp://www.resultri.com


INTRO TO NOSQL


RDBMS ARE NOT ENOUGH?


Growth is the New Reality

• Instagram gained nearly 1 million users overnight when then expanded to Android


Draw Something -‐ Social Game

35 million monthly active users in 1 month !!!


http://blog.couchbase.com/preparing-massive-growth-revisited

http://blog.couchbase.com/preparing-massive-growth-revisited

By contrast....

The Simpson’s : Tapped OutDaily Active Users (Millions)


http://www.macworld.com/article/1165708/ea_stumbles_again_pulls_simpsons_game.html














































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































How do you take this growth?

RDBMS is good for many thing, but hard to scale

RDBMS Scales UpGet a bigger, more complex server

Users

Applica@on Scales OutJust add more commodity web servers

Users

System CostApplica)on Performance

Rela@onal Database

Web/App Server Tier


Won’t scale beyond this point


Scaling out RDBMS

• Run Many SQL Servers

•Data could be shared

-‐ Done by the applica)on code

• Caching for faster response )me

Web/App Server Tier

Memcached Tier

MySQL Tier


NoSQL Technology Scales Out

Scaling out fla;ens the cost and performance curves

NoSQL Database Scales OutCost and performance mirrors app @er

Users

NoSQL Distributed Data Store

Web/App Server Tier

Applica@on Scales OutJust add more commodity web servers

Users


Applica)on Performance System Cost


A New Technology?

Building new database to answer the following requirements

•No schema required before inser)ng data

•No schema change required to change data format

• Auto-‐sharding without applica)on par)cipa)on

•Distributed queries

• Integrated main memory caching

•Data synchroniza)on ( mul)-‐datacenter)

DynamoOctober 2007

CassandraAugust 2008

BigtableNovember 2006

VoldemortFebruary 2009

Very few organiza@ons want to (fewer can) build and maintain database soQware technology.But every organiza@on building interac@ve web applica@ons needs this technology.


What Is Biggest Data Management Problem Driving Use of NoSQL in Coming Year?

Lack of flexibility/rigid schemas

Inability to scale out data

Performance challenges

Cost All of these Other

49%

35%

29%

16% 12% 11%

Source: Couchbase Survey, December 2011, n = 1351.


NO SQL TAXONOMIES


NoSQL CatalogKey-‐Value

Memcached

Membase

Redis

Data Structure Document Column Graph

MongoDB

Couchbase Cassandra

Cache

(mem

ory on

ly)

Database

(mem

ory/disk)

Neo4j


NoSQL CatalogKey-‐Value

Memcached

Membase

Redis

Data Structure Document Column Graph

MongoDB

Couchbase Cassandra

Cache

(mem

ory on

ly)

Database

(mem

ory/disk)

Neo4j

HBase InfiniteGraph

Coherence


Hadoop ?


ClouderaHortonworks

Mapr

OperaIonal vs. AnalyIc Databases

CouchbaseMongoDB

CassandraHbase

AnalyCcDatabases

Get insights from data

Real-‐Cme, InteracCve Databases

Fast access to data

NoSQL


COUCHBASE


Couchbase Server Core Principles

Easy Scalability

Consistent High Performance

Always On 24x365

Grow cluster without applicaôn changes, without down^me with a single click

Consistent sub-‐millisecond read and write response ^mes with consistent high throughput

No down^me for so`ware upgrades, hardware maintenance, etc.

Flexible Data Model

JSON document model with no fixed schema.

JSONJSONJSON

JSONJSON

PERFORMANCE


Couchbase 2.0 New Features

JSON support Indexing and Querying

Cross data center replication

Incremental Map Reduce


Couchbase Handles Real World Scale


Sub^tleCouchbase Server 2.0 Architecture

Heartbeat

Process m

onito

r

Glob

al singleton supe

rviso

r

Confi

gura)o

n manager

on each node

Rebalance orchestrator

Nod

e he

alth m

onito

r

one per cluster

vBucket state and

replica)

on m

anager

hdpRE

ST m

anagem

ent A

PI/W

eb UI

HTTP8091

Erlang port mapper4369

Distributed Erlang21100 -‐ 21199

Erlang/OTP

storage interface

Couchbase EP Engine

11210Memcapable 2.0

Moxi

11211Memcapable 1.0

Memcached

New Persistence Layer

8092Query API

Que

ry Engine

Data Manager Cluster Manager


Couchbase Server 2.0 Architecture

New Persistence Layer

storage interface

Couchbase EP Engine

11210Memcapable 2.0

Moxi

11211Memcapable 1.0

Object-‐level Cache

Disk Persistence

8092Query API

Que

ry Engine

HTTP8091

Erlang port mapper4369

Distributed Erlang21100 -‐ 21199

Heartbeat

Process m

onito

r

Glob

al singleton supe

rviso

r

Confi

gura)o

n manager

on each node

Rebalance orchestrator

Nod

e he

alth m

onito

r

one per cluster

vBucket state and

replica)

on m

anager

hdp

REST m

anagem

ent A

PI/W

eb UI

Erlang/OTP

Server/Cluster Management & Communica@on

(Erlang)

RAM Cache, Indexing & Persistence Management

(C & V8)

The Unreasonable Effectiveness of C by Damien Katz


http://damienkatz.net/2013/01/the_unreasonable_effectiveness_of_c.html

http://damienkatz.net/2013/01/the_unreasonable_effectiveness_of_c.html

Apache 2.0Open Source Project

hQps://github.com/couchbase/

hQps://github.com/couchbaselabs/

hQp://review.couchbase.org/Gerrit:


https://github.com/couchbase/

https://github.com/couchbase/

https://github.com/couchbaselabs/

https://github.com/couchbaselabs/

http://review.couchbase.org/

http://review.couchbase.org/

SETTING UP TO DEVELOP


Install Couchbase Server 2.0

Ubuntu

RedHat

Mac OS X

Windows

or build from sources


Official SDKs

www.couchbase.com/develop

Clojure

Python

Ruby

libcouchbase

Go


http://www.couchbase.com/develop

http://www.couchbase.com/develop

Client SDKs

Couchbase ClientApp Server

make connec)

on

receive top

ologyCouchbase TopologyUpdate


COUCHBASE OPERATIONS


Write OperaIon

33 2Managed Cache

Disk Que

ue

Disk

Replicaôn Queue

App Server

Couchbase Server Node

Doc 1Doc 1

Doc 1

To other node


Basic OperaIons

COUCHBASE SERVER CLUSTER

• Docs distributed evenly across servers

• Each server stores both ac@ve and replica docsOnly one doc ac)ve at a )me

• Client library provides app with simple interface to database

• Cluster map provides map to which server doc is onApp never needs to know

• App reads, writes, updates docs

• Mul@ple app servers can access same document at same @me

READ/WRITE/UPDATE

ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc

SERVER 1

ACTIVE

Doc 4

Doc 7

Doc

Doc

Doc

SERVER 2

Doc 8

ACTIVE

Doc 1

Doc 2

Doc

Doc

Doc

REPLICA

Doc 4

Doc 1

Doc 8

Doc

Doc

Doc

REPLICA

Doc 6

Doc 3

Doc 2

Doc

Doc

Doc

REPLICA

Doc 7

Doc 9

Doc 5

Doc

Doc

Doc

SERVER 3

Doc 6

APP SERVER 1

COUCHBASE Client LibraryCLUSTER MAP

COUCHBASE Client LibraryCLUSTER MAP

APP SERVER 2

Doc 9


• get (key)– Retrieve a document

• set (key, value)– Store a document, overwrites if exists

• add (key, value)– Store a document, error/excepôn if exists

• replace (key, value)– Store a document, error/excepôn if doesn’t exist

• cas (key, value, cas)– Compare and swap, mutate document only if it hasn’t changed while execu^ng this operaôn

Store & Retrieve OperaIons


Atomic Counter OperaIonsThese operaôns are always executed in order atomically.

• set (key, value)– Use set to iniâlize the counter

• cb.set(“my_counter”, 1)

• incr (key)– Increase an atomic counter value, default by 1

• cb.incr(“my_counter”) # now it’s 2

• decr (key)– Decrease an atomic counter value, default by 1

• cb.decr(“my_counter”) # now it’s 1


Mental Adjustments

• In SQL we tend to want to avoid hilng the database as much as possible– Even with caching and indexing tricks, and massive improvements over the years, SQL s^ll gets bogged down by complex joins and huge indexes, so we avoid making database calls

• In Couchbase, get’s and set’s are so fast they are trivial, not bo0lenecks, this is hard for many people to accept at first; Mul^ple get statements are commonplace, don’t avoid it!


JSON Document Structuremeta{

“id”: “u::[email protected]”,“rev”: “1-‐0002bce0000000000”,“flags”: 0,“expira@on”: 0,“type”: “json”

}

document{

“uid”: 123456,“firstname”: “jasdeep”,“lastname”: “Jaitla”,“age”: 22,“favorite_colors”: [“blue”, “black”],“email”: “[email protected]”

}

Meta Informa@on Including Key

All Keys Unique and Kept in RAM

Document Value

Most Recent In Ram And Persisted To Disk


DEMONSTRATION


Add Nodes to Cluster

• Two servers addedOne-‐click opera@on

• Docs automa@cally rebalanced across clusterEven distribu)on of docsMinimum doc movement

• Cluster map updated

• App database calls now distributed over larger number of servers

REPLICA

ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc 4

Doc 1

Doc

Doc

SERVER 1

REPLICA

ACTIVE

Doc 4

Doc 7

Doc

Doc

Doc 6

Doc 3

Doc

Doc

SERVER 2

REPLICA

ACTIVE

Doc 1

Doc 2

Doc

Doc

Doc 7

Doc 9

Doc

Doc

SERVER 3 SERVER 4 SERVER 5

REPLICA

ACTIVE

REPLICA

ACTIVE

Doc

Doc 8 Doc

Doc 9 Doc

Doc 2 Doc

Doc 8 Doc

Doc 5 Doc

Doc 6

READ/WRITE/UPDATE READ/WRITE/UPDATE

APP SERVER 1

COUCHBASE Client Library

CLUSTER MAP


CLUSTER MAP

APP SERVER 2


User Configured Replica Count = 1


Fail Over Node

REPLICA

ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc 4

Doc 1

Doc

Doc

SERVER 1

REPLICA

ACTIVE

Doc 4

Doc 7

Doc

Doc

Doc 6

Doc 3

Doc

Doc

SERVER 2

REPLICA

ACTIVE

Doc 1

Doc 2

Doc

Doc

Doc 7

Doc 9

Doc

Doc

SERVER 3 SERVER 4 SERVER 5

REPLICA

ACTIVE

REPLICA

ACTIVE

Doc 9

Doc 8

Doc Doc 6 Doc

Doc

Doc 5 Doc

Doc 2

Doc 8 Doc

Doc

• App servers accessing docs

• Requests to Server 3 fail

• Cluster detects server failedPromotes replicas of docs to ac)veUpdates cluster map

• Requests for docs now go to appropriate server

• Typically rebalance would follow

Doc

Doc 1 Doc 3

APP SERVER 1


CLUSTER MAP


CLUSTER MAP

APP SERVER 2

User Configured Replica Count = 1



Indexing and Querying


ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc

SERVER 1

REPLICA

Doc 4

Doc 1

Doc 8

Doc

Doc

Doc

APP SERVER 1


CLUSTER MAP


CLUSTER MAP

APP SERVER 2

Doc 9

• Indexing work is distributed amongst nodes

• Large data set possible

• Parallelize the effort

• Each node has index for data stored on it

• Queries combine the results from required nodes

ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc

SERVER 2

REPLICA

Doc 4

Doc 1

Doc 8

Doc

Doc

Doc

Doc 9

ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc

SERVER 3

REPLICA

Doc 4

Doc 1

Doc 8

Doc

Doc

Doc

Doc 9

Query


DEMONSTRATION


Cross Data Center ReplicaIon (XDCR)

RAM CACHE

Doc 1

Doc 2

Doc

Doc

Doc

Doc

Doc

Doc

Doc

Doc

Doc

SERVER 1

Doc 6

DISK

RAM CACHE

Doc 1

Doc 2

Doc

Doc

Doc

Doc

Doc

Doc

Doc

Doc

Doc

SERVER 2

Doc 6

DISK

RAM CACHE

Doc 1

Doc 2

Doc

Doc

Doc

Doc

Doc

Doc

Doc

Doc

Doc

SERVER 3

Doc 6

DISK

Couchbase ClusterWest Coast Data Center

RAM CACHE

Doc 1

Doc 2

Doc

Doc

Doc

Doc

Doc

Doc

Doc

Doc

Doc

SERVER 1

Doc 6

DISK

RAM CACHE

Doc 1

Doc 2

Doc

Doc

Doc

Doc

Doc

Doc

Doc

Doc

Doc

SERVER 2

Doc 6

DISK

RAM CACHE

Doc 1

Doc 2

Doc

Doc

Doc

Doc

Doc

Doc

Doc

Doc

Doc

SERVER 3

Doc 6

DISK

Couchbase ClusterEast Coast Data Center


Map FuncIon


DEMONSTRATION


ElasIc Search Adaptor

• Elastic Search is good for ad-hoc queries and faceted browsing• Our adapter is aware of changing Couchbase topology• Indexed by Elastic Search after stored to disk in Couchbase

Elas@cSearch


I’m Excited to See What You Build,Q & A

Contact me on Twider@tgrall

Contact me by [email protected]

Learn More About Design PadernsCouchbaseModels.com

Senng up for Ruby on RailsCouchbaseOnRails.com

Couchbase Docswww.couchbase.com/docs/index-‐full.html

Couchbase Forumswww.couchbase.com/forums

IRC#couchbase#libcouchbase




http://www.couchbase.com/docs/index-full.html

http://www.couchbase.com/docs/index-full.html

Win some cool stuff, send a mail to

[email protected]

with NORMANDYJUG in the subject.




Q&A
