NoSQL

Tuesday, March 22, 2011

The Software Crisis

Writing correct, understandable, and verifiable computer programs is difficult.

Edsger Dijkstra

“as long as there were no machines, programming was no problem at all;

when we had a few weak computers, programming became a mild problem,

and now we have gigantic computers, programming has become an equally gigantic problem.”

The Software Crisis

IMSThe Hierarchical

Database

(1966)

Vern WattsTuesday, March 22, 2011

“A Relational Model for Large

Shared Databanks”

(1970)

Ted CoddTuesday, March 22, 2011

“In striving to make every user happy, a technology can

actually leave the majority unhappy.”

“Every good idea is generalized to its level of

inapplicability.”

(Peter Principle)

Jim Gray

“NoSQL” Reintroduced(2008)

Eric Evans

Total Cost of Ownership

• The price of a license

• The price of support

• The price of hardware

Oracle +/- 47k / CPU?Software update / support +/- 10k?

Internet Scale

• Massive data collections

• Huge number of requests

• Coming from geographic areas across the globe

• 24/7

Availability

Data Models

Column Oriented

key namedcolumn

namedcolumn

Column Family ≈ Table

Empty cells are cheap (sparse

table)

Can grow “indefinitely”

Schemaless No secundary

indexes

DatastoreService service = ...;Key key = KeyFactory.createKey(family, recordId);Entity entity = service.get(key);entity.getProperty(“firstname”);entity.getProperty(“surname”);

BigTable

Column Oriented + Super Columns

key namedcolumn

namedcolumn

Super Columns

Key Value Store

•Schemaless•Versioning

10110110

Kyoto CabinetDB db = new DB(...);db.set(“ws103177”, “Wilfred Springer <wilfredspringer@sun.com>”);db.get(“ws103177”);

1 mln records in 0.9 s

Graph Database

SPARQL

Document Store

Improved Indexing

<persons><person><name>Wilfred</name><surname>Springer</person>…</persons>

[{ "Name" : "Wilfred", "Surname" : "Springer"}, …]

Serverside Processing

Product

CustomerReviews

DetailPageURL

ItemAttributes

LargeImage

Height

MediumImageURL

Height

SalesRank

SimilarProducts

SmallImage

fs_author_terms

Height

Publisher

RelaseDate

Format

Binding

ProductGroup

ProductName

Studio

PublicationDate

Manufacterer

Amount

CurrencyCode

FormattedPrice

Content

Source

IsLinkSuppressed

TotalReviews ASIN

HelpfulVotes

Rating

Summary

Content

TotalVotes

Reviewer

CustomerId

EditorialReviews

Review

Languages

ListPrice

Author

fs_browsenodes

fs_keywords_terms

fs_version

ItemAttributes

Publisher

RelaseDate

Format

Binding

ProductGroup

ProductName

Studio

PublicationDate

Manufacterer

Amount

CurrencyCode

FormattedPrice

NameLanguages

ListPrice

Author

// find all productsdb.products.find() // find all products

// find products with 446 pages (slow)db.products.find({“ItemAttributes.NumberOfPages”: 446})

// find products with 446 pages (fast)db.products.ensureIndex({"ItemAttributes.NumberOfPages": 1}) db.products.find({“ItemAttributes.NumberOfPages”: 446})

Various Queries

Product ItemAttributes NumberOfPages

db.products.find( {"fs_keywords_terms": "java"}, {"ItemAttributes.Title" : 1})

Find books on “java”

ProductItemAttributes Title

fs_keywords_terms

!"#$!%&%'()*+,-$.!/$01234(/3((5/+/60(60**0!7

!8#,9*!%&%!8:;<-;%=>+?*,@%#A%BCDCE%=*+>A$%F$#,#>A&%;GC+,#+C9%HI#$*%J>G%;G>KGCLL*G@!

J@"?*MN>G$@",*GL@&%O!?*AA*,P!E!9!E!+C9D*G,!E!L#+PC*9!E!)!E!$>ACP>>!E!,+Q!E!#Q!E!@>+?*,@!E!)CDC!E!@*+>A$!E!*$#,#>A!E!QGC+,#+C9!E!KI#$*!E!QG>KGCLL*G@!R

db.products.find( {"fs_keywords_terms": "java"}, {"ItemAttributes.Title" : 1}).sort({“SalesRank”: -‐1})

... with the worst sales rank

ProductItemAttributes Title

fs_keywords_terms

!"#$!%&%'()*+,-$.!/$01234(/3((5/+/60(60**0!7

!8#,9*!%&%!8:;<-;%=>+?*,@%#A%BCDCE%=*+>A$%F$#,#>A&%;GC+,#+C9%HI#$*%J>G%;G>KGCLL*G@!

J@"?*MN>G$@",*GL@&%O!?*AA*,P!E!9!E!+C9D*G,!E!L#+PC*9!E!)!E!$>ACP>>!E!,+Q!E!#Q!E!@>+?*,@!E!)CDC!E!@*+>A$!E!*$#,#>A!E!QGC+,#+C9!E!KI#$*!E!QG>KGCLL*G@!R

db.products.group({ key: {"ItemAttributes.NumberOfPages": true }, cond: {}, initial: {count: 0}, reduce: function(obj,prev) { prev.count++ }})

Count books per #pages

db.runCommand({mapreduce: "DenormAggCollection",query: { filter1: { '$in': [ 'A', 'B' ] }, filter2: 'C', filter3: { '$gt': 123 } },map: function() { emit( { d1: this.Dim1, d2: this.Dim2 }, { msum: this.measure1, recs: 1, mmin: this.measure1, mmax: this.measure2 < 100 ? this.measure2 : 0 } );},reduce: function(key, vals) { var ret = { msum: 0, recs: 0, mmin: 0, mmax: 0 }; for(var i = 0; i < vals.length; i++) { ret.msum += vals[i].msum; ret.recs += vals[i].recs; if(vals[i].mmin < ret.mmin) ret.mmin = vals[i].mmin; if((vals[i].mmax < 100) && (vals[i].mmax > ret.mmax)) ret.mmax = vals[i].mmax; } return ret; },finalize: function(key, val) { val.mavg = val.msum / val.recs; return val; },out: 'result1',verbose: true});db.result1. find({ mmin: { '$gt': 0 } }). sort({ recs: -1 }). skip(4). limit(8);

SELECT Dim1, Dim2, SUM(Measure1) AS MSum, COUNT(*) AS RecordCount, AVG(Measure2) AS MAvg, MIN(Measure1) AS MMin MAX(CASE WHEN Measure2 < 100 THEN Measure2 END) AS MMaxFROM DenormAggTableWHERE (Filter1 IN (’A’,’B’)) AND (Filter2 = ‘C’) AND (Filter3 > 123)GROUP BY Dim1, Dim2HAVING (MMin > 0)ORDER BY RecordCount DESCLIMIT 4, 8

()*+,-./.01-230*2/4*5+123/6)-/,+55-./*+7/63/8-93/02/7:-/16,/;+2470*2</)-.+402=/7:-/30>-/*;/7:-/?*)802=/3-7@

A-63+)-3/1+37/B-/162+6559/6==)-=67-.@

C==)-=67-3/.-,-2.02=/*2/)-4*)./4*+273/1+37/?607/+2705/;02650>670*2@

A-63+)-3/462/+3-/,)*4-.+)65/5*=04@

D057-)3/:6E-/62/FGAHC470E-G-4*).I5**802=/3795-@

' C==)-=67-/;057-)02=/1+37/B-/6,,50-./7*/7:-/)-3+57/3-7</2*7/02/7:-/16,H)-.+4-@

& C34-2.02=J/!K/L-34-2.02=J/I!

G-E030*2/$</M)-67-./"N!NIN#IN'

G048/F3B*)2-</)048*3B*)2-@*)=

19OPQ A*2=*LRSQL Mongo

Availability versus

Consistency

CAP Theorem

Eric Brewer

Availability Consistency

PartitionTolerance

Pick two

Strong Consistency

0 value = "foo"

value = "bar"

value = "bar" value = "bar"

value = "bar"

After the update, any subsequent access will return the updated value.

Weak Consistency

1 value = "bar"

value = "bar" /"foo"

value = "bar" / "foo"

0 value = "foo"

The system does not guarantee that at any given point in the future subsequent access will return the updated value

Eventual Consistency

If no updates are made to the object, eventually all accesses will return the last updated value.

1 value = "bar"

value = "bar" value = "bar"

value = "bar"

0 value = "foo"

t ≥ 1

Session Consistency

Within the “session”, the system guarantees read-your-writes consistency

2 value = "foo"

Session 1

Session 2

1 value = "bar"

0 value = "foo"

2 value = "bar"

Read-your-writes Consistency

Process A, after updating a data item always access the updated value and never sees an older value

1 value = "bar"

0 value = "foo"

2 value = "bar"

Monotonic Read Consistency

If a process has seen a particular value for the object, any subsequent access will never return any previous values

0 value = "foo"

1 value = "foo"

2 value = "foo"

4 value = "bar"

value = "bar"

Eventual Consistentency in RDBMS

Eventual consistency is not just a property of NoSQL Solutions

APrimary Backup replica

Log shipping

No Strong Consistency in

Face Of...

Network Partitions

Awrites new value

replicates new value

reads new value

Network Partitions

Awrites new value

replicates new value

reads new value

Partition Tolerance

Awrites new value

fails to replicate

new value

reads old value

Partition Intolerance

failing attempt to write a new

fails to replicate

new value

How to do better?

N = 4R = 2

Proper Replication Factor

Optimizations

• Optimize read: R = 1, N = W

• Optimize write: W = 1, N = R

Consistent HashingKey KA

No free rideYou need to consider giving up on:

•Avoiding redundancy

•Referential integrity

•Strong consistency

•Ad hoc queries

•Joins

•Ease of reporting

•...

NoSQL Today

Resources

http://nosqltapes.com/

http://nosql-database.org/

http://nosqlsummer.org/

No SQLwspringer@xebia.com

NoSQL

Technology

NoSQL and Big Data Analytics at NOSQL NOW! 2013

How companies use NoSQL & Couchbase - NoSQL Now 2014

NOSQL - CRS4dassia.crs4.it/wp-content/uploads/2014/11/01_NOSQL.pdf · 2015-03-06 · NOSQL Origini e Significato NOSQL = NO a SQL NOSQL = Not Only SQL Il termine NOSQL fu introdotto

NoSQL Orientado à documentos. Apresentação NoSQL Problemas que levaram a utilização do NoSQL Quando utilizar NoSQL? Tipos Orientação à documentos MapReduce

How to Use NoSQL - nosqlroadshow.comnosqlroadshow.com/dl/NoSQL-Road-Show/slides/Howt... · How to Use NoSQL in Enterprise Java Applications Patrick Baumgartner NoSQL Roadshow | Zürich

NoSQL Smackdown!

NoSQL Oakland

Warum NoSQL?

Node.js + NoSQL

NoSQL: Graph Databases. Databases Why NoSQL Databases?

Oracle NoSQL Database – A Distributed Key-Value Store · HPTS, October 24, 2011 Agenda • Oracle and NoSQL • Oracle NoSQL Database Architecture • Oracle NoSQL Database Technical

Nosql seminar

INTRODUCTION TO MONGODB...INTRODUCTION TO MONGODB Intro to Databases / NOSQL INTUITION NoSQL vs. SQL SQL NoSQL NoSQL vs. SQL SQL NoSQL Table Based Documents, Key-Value pairs, Graph-based,

NoSQL features of Db2 (LUW) revisited€¦ · • Most NoSQL stores lack true ACID transactions Term “NoSQL” introduced 1998 by Carlo Strozzi (shell-interfaced RDBMS); ... NoSQL

SQL vs NoSQL: The NoSQL way

NoSQL 프로그래밍 : 한 권으로 끝내는 NoSQL 솔루션 활용법

NoSQL benchmarking

Bringing SQL to NoSQL: Rich, Declarative Query for NoSQL

Database NoSQL: CouchBase - ingegneria … · 3.1 Couchbase Lite ... sarà presentato in particolare CouchBase. Database NoSQL: CouchBase : 5 . Capitolo 1: NoSQL . 1.1 Perché NoSQL?

Wmware NoSQL