Couchbase by Tugdual Grall

Lorraine JUG - 19-JUN-2013

Friday, June 21, 13

Elsass JUG - 20-JUN-2013

Send a mail to:[email protected]

ElsassJUG in the subject...

Join the French Couchbase Meetuph3p://www.meetup.com/Couchbase-‐France/

Friday, June 21, 13

mailto:[email protected]


http://www.meetup.com/Couchbase-France/



Introduc)on to NoSQL with Couchbase

Tugdual ‘Tug’ Grall@tgrall

Technical Evangelist

Friday, June 21, 13


About me...

• Tugdual “Tug” Grall Couchbase

• Technical Evangelist

eXo

• CTO

Oracle

• Developer/Product Manager

Mainly Java/SOA Developer in consul)ng firms

• Web

@tgrall

hLp://blog.grallandco.com

tgrall

NantesJUG co-‐founder

Pet Project :

hLp://www.resultri.com

Friday, June 21, 13

http://blog.grallandco.com

http://blog.grallandco.com

http://www.resultri.com

http://www.resultri.com


Agenda

• Why NoSQL ?

• NoSQL Landscape

• Document Design

• Use Cases

• Couchbase Server 2.0

Friday, June 21, 13


$30B Database Market Being Disrupted

<50%?

2027

95%RelaVonal

Technology

2012

All new database growth will be NoSQL

RelaVonal Technology



NoSQLTechnology

Other

Friday, June 21, 13


Three Macro Trends Driving DisrupVon

Big Data SaaS/Cloud CompuHngBig Users

Friday, June 21, 13


Big DataHigh Data Variety and Velocity

Source: IDC 2011 Digital Universe Study (hLp://www.emc.com/collateral/demos/microsites/emc-‐digital-‐universe-‐2011/index.htm)

Trillions of G

igabytes (ZeL

abytes)

0

0.50

1.00

1.50

2.00

2000 2006 2011

Unstructured and Semi-‐Structured Data

Structured Data

Text, Log Files, Click Streams, Blogs, Tweets, Audio, Video, etc.

More Flexible Data Model Required

Friday, June 21, 13


RDBMS are not Enough?

Friday, June 21, 13


Growth is the New Reality

• Instagram gained nearly 1 million users overnight when then expanded to Android

Friday, June 21, 13


Draw Something Viral Growth

191715131197533/12826242220181614121082/6

Draw Something by OMGPOPDaily Ac)ve Users (millions)

21

2

4

6

8

10

12

14

16

Friday, June 21, 13


RDBMS is good for many thing, but hard to scale

RDBMS Scales UpGet a bigger, more complex server

Users

ApplicaVon Scales OutJust add more commodity web servers

Users

System CostApplica)on Performance

RelaHonal Database

Web/App Server Tier


Won’t scale beyond this point

How do you take this growth?

Friday, June 21, 13


Web/App Server Tier

Memcached Tier

MySQL Tier

Scaling out RDBMS

• Run Many SQL Servers

• Data could be sharded

Done by the applica)on code

• Caching for faster response Vme

Friday, June 21, 13


Scaling out fla3ens the cost and performance curves

NoSQL Database Scales OutCost and performance mirrors app Ver

Users

NoSQL Distributed Data Store

Web/App Server Tier

ApplicaVon Scales OutJust add more commodity web servers

Users


Applica)on Performance System Cost

NoSQL Technology Scales Out

Friday, June 21, 13


DynamoOctober 2007

CassandraAugust 2008

BigtableNovember 2006

VoldemortFebruary 2009

Very few organizaHons want to (fewer can) build and maintain database so]ware technology.But every organizaHon building interacHve web applicaHons needs this technology.

A New Technology?

• Building new database to answer the following requirements No schema required before inser)ng data

No schema change required to change data format

Auto-‐sharding without applica)on par)cipa)on

Distributed queries

Integrated main memory caching

Data synchroniza)on ( mul)-‐datacenter)

Friday, June 21, 13


Lack of flexibility/rigid schemas

Inability to scale out data Performance challenges Cost All of these Other

49%

35%

29%

16%12% 11%

What Is Biggest Data Management Problem Driving Use of NoSQL in Coming Year?

Source: Couchbase Survey, December 2011, n = 1351.

Friday, June 21, 13


NoSQL CatalogKey-‐Value

Memcached

Membase

Redis

Data Structure Document Column Graph

MongoDB

Couchbase Cassandra

Cache

(mem

ory on

ly)

Database

(mem

ory/disk) Neo4j

HBase InfiniteGraph

Coherence

Friday, June 21, 13


OperaVonal vs. AnalyVc Databases

ClouderaHortonworks

Mapr

CouchbaseMongoDB

CassandraHbase

AnalyHcDatabases

Get insights from data

Real-‐Hme, InteracHve Databases

Fast access to data

NoSQL

Friday, June 21, 13


Hadoop

Friday, June 21, 13


Use Cases

Friday, June 21, 13


Use CasesKey Value • Session Management

• User Profile/Preferences• Shopping Cart

Document • Event Logging• Content Management • Web Analy)cs• E-‐Commerce Applica)on

Columns • Event Logging• Content Management• Counters

Graph • Connected Data / Social Networks• Rou)ng, Dispatch• Recommenda)ons based on Social Graph

Friday, June 21, 13


Data Models

Friday, June 21, 13


RelaVonal vs Document Data Model

RelaHonal data model Document data modelCollec)on of complex documents witharbitrary, nested data formats and

varying “record” format.

Highly-‐structured table organiza)on with rigidly-‐defined data formats and

record structure.

C1 C2 C3 C4

JSONJSON

JSON

{

}

Friday, June 21, 13


The RelaVonal Approach to Storing Data

Rela)onal databases were not designed with clusters in mind, which is why people have cast around for an alterna)ve. Storing aggregates as fundamental units makes a lot of sense for running on a cluster.

hLp://mar)nfowler.com/bliki/AggregateOrientedDatabase.html

Friday, June 21, 13

http://martinfowler.com/bliki/AggregateOrientedDatabase.html

http://martinfowler.com/bliki/AggregateOrientedDatabase.html


Document Database by Comparison

o::1001{uid: “ji22jd”,customer: “Ann”,line_items: [

{ sku: 0321293533, quan: 2, unit_price: 48.0 },{ sku: 0321601912, quan: 1, unit_price: 39.0 },{ sku: 0131495054, quan: 1, unit_price: 51.0 }

],payment: { type: “Amex”, expiry: “04/2001”,

last5: 12345 }}

Easy to distribute dataMakes sense to applica)on programmers

Friday, June 21, 13


Objects Serialized to JSON and Back

User Object

string uid

string firstname

string lastname

int age

array favorite_colors

string email

u::[email protected]{“uid”: 123456,“firstname”: “John”,“lastname”: “Smith”,“age”: 22,“favorite_colors”: [“blue”, “black”],“email”: “[email protected]”}

User Objectstring uid

string firstname

string lastname

int age

array favorite_colors

string email

u::[email protected]{“uid”: 123456,“firstname”: “john”,“lastname”: “smith”,“age”: 22,“favorite_colors”: [“blue”, “black”],“email”: “[email protected]”}

set()

get()

Friday, June 21, 13








Schema Changes

V1

Speaker

Robin Johnson

Tug Grall

John Zablo

Event Info

ID Title Speaker_id

Frank Weigel

1 Paris JUG rbin

2 QCON UK tgrall

3 NoSQL CLN johnz

4 Devoxx fweigel

ID Name

rbin

tgrall

johnz

fweigel

Friday, June 21, 13

Speaker

Robin Johnson

Tug Grall

John Zablo

Event Info

ID Title Speaker_id

Frank Weigel

1 Paris JUG rbin

2 QCON UK tgrall

3 NoSQL CLN johnz

4 Devoxx fweigel

ID Name

rbin

tgrall

johnz

fweigel


Schema Changes

V2 A talk could be done by one or more speakers!

1.Create a new table2.“Prepare” the Data (move to new table, delete column)3.Change the constraints and foreign keys4.Change the applicaZon code5.Hope that your DBA is in a good day

Friday, June 21, 13


Schema Changes

V1

event:1{type: “event”,title: “Paris JUG”,speaker: “rbin”}




event:1{ type: “event”, title: “Paris JUG”, speaker: “rbin”}

user:rbin{ type: “user”, name: “Robin Johnson”,}





user:rbin{ type: “user”, name: “Robin Johnson”, id: “rbin”}

Friday, June 21, 13


Schema Changes

V2





event:1{ type: “event”, title: “Paris JUG”, speaker: “rbin”}






user:rbin{ type: “user”, name: “Robin Johnson”, id: “rbin”}

A talk could be done by one or more speakers!

Friday, June 21, 13


Schema Changes

V2





event:1{ type: “event”, title: “Paris JUG”, speakers: {“rbin”, “tgrall”}}






user:rbin{ type: “user”, name: “Robin Johnson”, id: “rbin:}

1.Change the App1.Update the data (‘when you want’)2.Change your “index” to query the new data

You can do it in a “lazy” way ..

Friday, June 21, 13


QuesVons ? Concerns ?

• Data DuplicaVon?

• Schema ?

• TransacVon ?

• DBA ?

• Backup/Restore ?

• ...?

Friday, June 21, 13


Couchbase ServerNoSQL Document Database

Friday, June 21, 13


Easy Scalability Consistent High Performance

Always On 24x365

Grow cluster without applica)on changes, without down)me with a single click

Consistent sub-‐millisecond read and write response )mes with consistent high throughput

No down)me for sorware upgrades, hardware maintenance, etc.

Flexible Data Model

JSON document model with no fixed schema.

Couchbase Server

Friday, June 21, 13


Open Source Project

Friday, June 21, 13


Couchbase Handles Real World Scale

Friday, June 21, 13


Heartbeat

Process m

onito

r

Glob

al singleton supe

rviso

r

Confi

gura)o

n manager

on each node

Rebalance orchestrator

Nod

e he

alth m

onito

r

one per cluster

vBucket state and

replica)

on m

anager

hbp

REST m

anagem

ent A

PI/W

eb UI

HTTP8091

Erlang port mapper4369

Distributed Erlang21100 -‐ 21199

Erlang/OTP

storage interface

Couchbase EP Engine

11210Memcapable 2.0

Moxi

11211Memcapable 1.0

Memcached

New Persistence Layer

8092Query API

Que

ry Engine

Data Manager Cluster Manager

Couchbase Server Architecture

Friday, June 21, 13


Heartbeat

Process m

onito

r

Glob

al singleton supe

rviso

r

Confi

gura)o

n manager

on each node

Rebalance orchestrator

Nod

e he

alth m

onito

r

one per cluster

vBucket state and

replica)

on m

anager

hbp

REST m

anagem

ent A

PI/W

eb UI

HTTP8091

Erlang port mapper4369

Distributed Erlang21100 -‐ 21199

Erlang/OTP

storage interface

Couchbase EP Engine

11210Memcapable 2.0

Moxi

11211Memcapable 1.0

Memcached

New Persistence Layer

8092Query API

Que

ry Engine

Couchbase Server Architecture

Friday, June 21, 13


Couchbase Opera)ons

Friday, June 21, 13


33 2

Single node -‐ Couchbase Write Opera)on

Managed Cache

Disk Que

ue

Disk

Replica)on Queue

App Server

Couchbase Server Node

Doc 1Doc 1

Doc 1

To other node

Friday, June 21, 13


COUCHBASE SERVER CLUSTER

Basic Opera)on

• Docs distributed evenly across servers

• Each server stores both acZve and replica docsOnly one server ac)ve at a )me

• Client library provides app with simple interface to database

• Cluster map provides map to which server doc is onApp never needs to know

• App reads, writes, updates docs

•MulZple app servers can access same document at same Zme

User Configured Replica Count = 1

READ/WRITE/UPDATE

ACTIVE

SERVER 1

ACTIVE

Doc 4

Doc 7

Doc

Doc

Doc

SERVER 2

Doc 8

ACTIVE

Doc 1

Doc 2

Doc

Doc

Doc

REPLICA

Doc 4

Doc 1

Doc 8

Doc

Doc

Doc

REPLICA

Doc 6

Doc 3

Doc 2

Doc

Doc

Doc

REPLICA

Doc 7

Doc 9

Doc 5

Doc

Doc

Doc

SERVER 3

Doc 6

Doc 5

Doc 2

Doc

Doc

DocDoc 9

APP SERVER 1

COUCHBASE Client Library

CLUSTER MAP


CLUSTER MAP

APP SERVER 2

Friday, June 21, 13


Add nodes to the cluster

• Two servers addedOne-‐click operaZon

• Docs automaZcally rebalanced across clusterEven distribu)on of docsMinimum doc movement

• Cluster map updated

• App database calls now distributed over larger number of servers

REPLICA

ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc 4

Doc 1

Doc

Doc

SERVER 1

REPLICA

ACTIVE

Doc 4

Doc 7

Doc

Doc

Doc 6

Doc 3

Doc

Doc

SERVER 2

REPLICA

ACTIVE

Doc 1

Doc 2

Doc

Doc

Doc 7

Doc 9

Doc

Doc

SERVER 3 SERVER 4 SERVER 5

REPLICA

ACTIVE

REPLICA

ACTIVE

Doc

Doc 8 Doc

Doc 9 Doc

Doc 2 Doc

Doc 8 Doc

Doc 5 Doc

Doc 6

READ/WRITE/UPDATE READ/WRITE/UPDATE

APP SERVER 1

COUCHBASE Client Library COUCHBASE Client Library

CLUSTER MAP CLUSTER MAP

APP SERVER 2


Friday, June 21, 13


Fail Over Node

REPLICA

ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc 4

Doc 1

Doc

Doc

SERVER 1

REPLICA

ACTIVE

Doc 4

Doc 7

Doc

Doc

Doc 6

Doc 3

Doc

Doc

SERVER 2

REPLICA

ACTIVE

Doc 1

Doc 2

Doc

Doc

Doc 7

Doc 9

Doc

Doc

SERVER 3 SERVER 4 SERVER 5

REPLICA

ACTIVE

REPLICA

ACTIVE

Doc 9

Doc 8

Doc Doc 6 Doc

Doc

Doc 5 Doc

Doc 2

Doc 8 Doc

Doc

• App servers accessing docs

• Requests to Server 3 fail

• Cluster detects server failedPromotes replicas of docs to ac)veUpdates cluster map

• Requests for docs now go to appropriate server

• Typically rebalance would follow

Doc

Doc 1 Doc 3

APP SERVER 1


CLUSTER MAP


CLUSTER MAP

APP SERVER 2


Friday, June 21, 13


Client SDK

www.couchbase.com/develop

Clojure

Python

Ruby

libcouchbase

Go

Friday, June 21, 13

http://www.couchbase.com/develop

http://www.couchbase.com/develop


Demo Time

Friday, June 21, 13


Indexing and Querying


ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc

SERVER 1

REPLICA

Doc 4

Doc 1

Doc 8

Doc

Doc

Doc

APP SERVER 1


CLUSTER MAP


CLUSTER MAP

APP SERVER 2

Doc 9

• Indexing work is distributed amongst nodes

• Large data set possible

• Parallelize the effort

• Each node has index for data stored on it

• Queries combine the results from required nodes

ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc

SERVER 2

REPLICA

Doc 4

Doc 1

Doc 8

Doc

Doc

Doc

Doc 9

ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc

SERVER 3

REPLICA

Doc 4

Doc 1

Doc 8

Doc

Doc

Doc

Doc 9

Query

Friday, June 21, 13


Map FuncVon

Friday, June 21, 13


Cross Data ReplicaVon

US DATA CENTER EUROPE DATA CENTER ASIA DATA CENTER

ReplicaZon ReplicaZon

ReplicaZon

• Data close to users

• MulVple locaVons for disaster recovery

• Independently managed clusters serving local data

Friday, June 21, 13


www.couchbase.com/download

Couchbase Server

Friday, June 21, 13


Q & A

Meetuph;p://meetup.com/Couchbase-‐France/

Contacts@tgrall / [email protected]

Friday, June 21, 13

http://meetup.com/Couchbase-France/

http://meetup.com/Couchbase-France/




Thank [email protected]

@tgrall

Join the French Couchbase Meetuph3p://www.meetup.com/Couchbase-‐France/

Friday, June 21, 13



Lorraine JUG - 19-JUN-2013

Friday, June 21, 13