Three Things You Need to Know About Data Modeling: Couchbase Connect 2015

THREE (OR MORE) THINGS YOU NEED TO KNOW ABOUT DATA MODELINGMatthew Revell, Developer Advocate, Couchbase

WE’RE STILL LEARNING

©2015 Couchbase Inc. 3

©2015 Couchbase Inc. 4

Three query methods

Application layer computation Off-load computation

Predictable queries

Key-value: pre-computed answers Views

Ad-hoc queriesN1QL and key-value with manual indexes

N1QL and views

KEY-VALUE

©2015 Couchbase Inc. 6

Key-value

Pre-compute answers asynchronously Store object state Choose when to embed data and when to refer

to it Design your keys well

PRE-COMPUTED ANSWERS

©2015 Couchbase Inc. 8

We’re used to asking questions

©2015 Couchbase Inc. 9

Key-value look-ups give us a library of answers

42

©2015 Couchbase Inc. 10

Answer oriented databases

http://martinfowler.com/bliki/AggregateOrientedDatabase.html

©2015 Couchbase Inc. 11

Airline bookings

EMBED, OR REFER?

How much should we denormalise?

©2015 Couchbase Inc. 13

An ecommerce order

©2015 Couchbase Inc. 14

The same order in a document

©2014 Couchbase, Inc. ©2015 Couchbase Inc. 15

Embedded v referred

15

©2015 Couchbase Inc. 16

When to embed

You should embed data when: Speed trumps all else

Slow moving data

No duplication

Application layer can keep multiple copies of same data in sync

©2015 Couchbase Inc. 17

When to refer

You should refer to data when: Consistency is a priority

The data has large growth potential

KEY DESIGN

©2015 Couchbase Inc. 19

Three ways to build a key

Key design is as important as document design. There are three broad types of key:

Human readable/deterministic: e.g. an email address

Computer generated/random: e.g. UUID

Compound: e.g. UUID with a deterministic portion

©2015 Couchbase Inc. 20

Human readable/deterministic

public class user {

private String name;private String email;private String streetAddress;private String city;private String country;private String postCode;private String telephone;private Array orders;private Array

productsViewed;}

{ "name": "Matthew Revell", "address": "11-21 Paul Street", "city": "London", "postCode": "EC2A 4JU", "telephone": "44-20-3837-9130", "orders": [ 1, 9, 698, 32 ], “productsViewed”: [8, 33, 99, 100] }

Key: matthew@couchbase.com

©2015 Couchbase Inc. 21

Random/computer genereated

{ "name": "Matthew Revell", "email": "matthew@couchbase.com", "address": "11-21 Paul Street", "city": "London", "postCode": "EC2A 4JU", "telephone": "44-20-3837-9130", "orders": [ 1, 9, 698, 32 ], “productsViewed”: [8, 33, 99, 100] }

Key: 1001

©2014 Couchbase, Inc. ©2015 Couchbase Inc. 22

Using counters to generate keys

22

Application

user_id = incr(“counter_key")

add(user_id, data)

Creating a new user

add(email_address, user_id)

Application

key = get("matthew@couchbase.com")

get(key)

Finding user by email address

©2015 Couchbase Inc. 23

Multiple look-up documents

u::count

1001

u::1001

{ "name": “Matthew Revell",

"facebook_id": 16172910,

"email": “matthew@couchbase.com”,

“password”: ab02d#Jf02K

"created_at": "5/1/2012 2:30am",

“facebook_access_token”: xox0v2dje20,

“twitter_access_token”: 20jffieieaaixixj }

fb::16172910

1001

nflx::2939202

1001

twtr::2920283830

1001

em::matthew@couchbase.com

1001

em::matthew@understated.co.uk

1001

uname::mrevell

1001

©2015 Couchbase Inc. 24

Compound keys

Compound keys are look-up documents with a predictable name.

It’s a continuation of the embedded versus referred data discussion.

©2015 Couchbase Inc. 25

Compound keys: example

u::1001

{ "name": "Matthew Revell", "email": "matthew@couchbase.com", "address": "11-21 Paul Street", "city": "London", "postCode": "EC2A 4JU", "telephone": "44-20-3837-9130", "orders": [ 1, 9, 698, 32 ], “productsViewed”: [8, 33, 99, 100]

}

©2015 Couchbase Inc. 26

Compound keys: example

u::1001

{ "name": "Matthew Revell", "email": "matthew@couchbase.com", "address": "11-21 Paul Street", "city": "London", "postCode": "EC2A 4JU", "telephone": "44-20-3837-9130", "orders": [ 1, 9, 698, 32 ]}

u::1001::productsviewed

{"productsList": [

8, 33, 99, 100]

}

©2015 Couchbase Inc. 27

Compound keys: example

u::1001

{ "name": "Matthew Revell", "email": "matthew@couchbase.com", "address": "11-21 Paul Street", "city": "London", "postCode": "EC2A 4JU", "telephone": "44-20-3837-9130", "orders": [ 1, 9, 698, 32 ]}

u::1001::productsviewed

{"productsList": [

8, 33, 99, 100]

}

p::8

{

id": 1,"name": "T-shirt","description": "Red Couchbase shirt","quantityInStock": 99,"image": "tshirt.jpg”

}

©2015 Couchbase Inc. 28

Compound keys: example

u::1001

{ "name": "Matthew Revell", "email": "matthew@couchbase.com", "address": "11-21 Paul Street", "city": "London", "postCode": "EC2A 4JU", "telephone": "44-20-3837-9130", "orders": [ 1, 9, 698, 32 ]}

u::1001::productsviewed

{"productsList": [

8, 33, 99, 100]

}

p::8

{

id": 1,"name": "T-shirt","description": "Red Couchbase shirt","quantityInStock": 99

}

p::8::img

“http://someurl.com/tshirt.jpg”

N1QL AND VIEWS!

The world beyond key-value

©2015 Couchbase Inc. 30

©2015 Couchbase Inc. 31

N1QL and Views

N1QL Views

Ad-hoc querying Predictable queries

Nested JSON in and unnested JSON out

Number crunching

Large growth clustersMulti-dimensional/geospatial queries

©2015 Couchbase Inc. 32

N1QL

Indexes Schema and documents types Keyspaces

©2015 Couchbase Inc. 33

Indexes

You always need at least one index PRIMARY index gives you all of the keys in the

bucket Secondary indexes enable ad-hoc querying at

speed

©2015 Couchbase Inc. 34

Indexes

GSI

versusVIEWS

©2015 Couchbase Inc. 35

Indexes

NEW Global Secondary Indexes Views

N1QL query to create a new indexJavascript map/reduce

Option to run dedicated indexing and query nodes

Queries always run on database nodes

ForestDB backed Couchstore backed

Supports multi-dimensional/geospatial queries

©2015 Couchbase Inc. 36

Maintaining schema and working with keyspaces JOINs work on primary keys and secondary keys They JOIN across and within keyspaces (for now, that

means buckets)

Airlines:

airline_24 ← Key (“primary key”){ "id": "24", "type": "airline", "name": "American Airlines", "iata": "AA", "icao": "AAL", "callsign": "AMERICAN", "country": "United States", "active": "Y"}

Routes:

route_5966 ← Key { "id": "5966", "type": "route", "airline": "AA", "airlineid": "airline_24", ← This is the foreign key "sourceairport": "MCO", "destinationairport": "SEA", "stops": "0", "equipment": "737", "schedule": [... ]}

©2015 Couchbase Inc. 37

Offloading computation to N1QL

N1QL allows you to choose which elements are returned at the top level. This is incredibly useful as it reduces the need to write complex parsing logic: Within the application. In Client side front end frameworks

(Angular/Backbone etc)SELECT a.name, s.flight, s.utc, r.sourceairport, r.destinationairport, r.equipment FROM `travel-sample` r UNNEST r.schedule s JOIN `travel-sample` a ON KEYS r.airlineid WHERE r.sourceairport='LHR' AND r.destinationairport='LAX' AND s.day=2 ORDER BY a.name

QUESTIONS!

OTHER SESSIONS!

IRFC: A design pattern for next gen business applicationsSteve Yen, 4.30pm ThursdayDev track

THANKS!

Matthew Revellmatthew@couchbase.com@matthewrevell

Get Started with Couchbase Server 4.0: www.couchbase.com/beta

Get Trained on Couchbase: training.couchbase.com