Prof. Dr. Stefan Edlich

NoSQL in der Cloud

nosqlberlin.de nosqlfrankfurt.de nosql powerdays

http://nosql-database.org

NoSQL is specialization!

• Big Data

• Massive Write Performance

• Fast KV Access

• Write Availability

• Flexible Schema (Migration) + Flexible Datatypes

• Easier maintainability, administration and operations

• No single point of failure

• Programmer ease of use

Theorie?!Map/Reduce Map/Reduce Nachfolger!

ACID / BASE & CAP P liegt in der Regel nie vor!

Consistent Hashing Basis skalierbarer K/V Stores

MVCC non blocking Vorteile

Vector Clocks [122:1][147:2|122:1]

[97:3|147:2|122:1]

Persistenz

Disk Full Memory Hybrid Pluggable

DatenmodellColumn Family DocumentDB Key / Value Graph

SchnittstellenREST Language

APIThrift/Avro

Map/Reduce Get/Put

Google Protocol Buffers

=>

• JSON

• Binary data transfer

• automatic RPC generation

• no code generation

• Client + Server tauschen Schema bei Änderung

unbedingt evaluieren!

Apache Avro!

Datenmodelle

Voldemort, Chordless, Scalaris, Dynamo / Dynomite

db4o, Versant, Objectivity, Gemstone, Progress, Mark Logic, EMC Momentum, Tamino, GigaSpaces, Hazelcast, Terracotta, …

Column Family

DocumentDBs

Key/ValueDBs

GraphDBs

andere

HBase Cassandra

SimpleDB

+ Skalierung = new node+ Community+ API- Replikation- Aufsetzen, Optimierung, Wartung

+ Skalierung = new node+ Replikation+ Konfiguration (r, w)- Dokumentation- Abfragen

+ stressfreie SaaS Lösung+ transparent scaling- UTF-8 String- Daten liegen bei Amazon+- kein tuning / config

Document Databases

any JS-Client

no Middleware!

DB+WebServer+evolving App

2.Runde+= 6,5 Mio $

- nicht normalisiert (Duplicates, Delete Orphans, ...)

- (konfigurierbare Zeit Crash anfällig) (Journaling)

- Eventually Consistent

- echte Skalierung nur über Sharding

- (noch nicht kill -9 fest)

EC2 Node 66 GB EC2 Node 66 GB

67 GBIndex Data

11 hours + 1 day off

+ nicht normalisiert+ Schema Agilität+ Doku exzellent+ Speed (MemMapped Files)+ Installation+save =28 sek!+ beliebige Indizes+ MapReduce+ Rich Query Language+ GridFS (statt HDFS)+ einfache Replizierung (Master-Slave / Replica Sets)

db.system.indexes.find();db.friends.getIndexes();db.friends.ensureIndex({friend: 1});db.friends.ensureIndex({friend: 1, zip: 1}); //compounddb.friends.find({friend: „Mario“, zip: „13755“}).explain();

Queries: age: {$gt: 10} food:{$all: [„pizza“, „noodles“]}$gt, $lt, $lte, $ne, $in, $nin, $mod, $all, $size, $exists, $type, , $or, $elem, $elemMatch, regexp, ...

NoSQL Query LockIn?!

Migrations Architektur-Pattern:

A) Blacklist

Sich veränderndes Schema

rename

try { ...} catch (FirstException | SecondException ex) { // newName = BlackList.checkName(OldName)}

B) „Rails“ Migration

new name

new name

new name

new name

old namenew name

(nicht wenn zu oft repliziert)

old namenew name

old namenew name

old namenew name

Duplikate = SpaceAktualität der Daten

„Pre-Joined“ Daten!„pre-computeD“

wachsende Daten raus oder Pre-SPACED

In die Cloud…

Clients

Config Serversmongos

ROUTER

Replica Set

Shard A Shard B Shard C

RAM+DISK+

POSSIBLE ARBITER

micro64 bit [extra | double | quadrupel] Large

Erfahrungen…

• RAID Konfigurationen (00,01,10,03,05, …)

• Journaling-Dateisysteme (ext4, xfs, …)• (Security) Ports, F-Deskriptoren,

Snapshots,…

www.mongodb.org/display/DOCS/Amazon+EC2

K/V-Stores+ sehr schnell > 100.000 /sek

+ konfigurierbarer Disc sync

+ API für eigene Anbindung

+ einfache Replikation

+ hash, list, set, sorted set,

messages

+ Installation

UNIX: 38 sek

Windows: 18 sek

- cloud-cluster erst in Version

3.*

Datenstrukturen abbilden ->

Sorted Set

memcached

API

• simply dynamic scaling (up & down)• scales linear• bullet proof by Zynga.com• limited membase protocol• Membase Tap (Protocol Interception)• Code-Node:

Membase in der Cloud

• Fertige RightScale & AMI templates

• Diverse Ports öffnen

• DNS Eintrag und keine verändernden IPs

• Master Node angeben• legt Quota für die Erben fest

• Backups für EBS

GraphDBs

Property Graph

player

Graph DBs in der Cloud• > N Milliarden Knoten? Sharding!

• aber meistens kein „predictable lookup“

• möglich nur bei Domain Specific Knowledge

• ausbalancierte DBs ohne sweet spots kaum möglich

• Access Patterns + Heuristiken (Insert Sharding / Runtime Sharding) => partitionierungs Algorithmen

• (HA) Neo4j Cache Sharding! • Multi-Master Cluster for Consistent

Routing

> 220 DBs

durchausfrustrierendes Consulting…

Data

Transactions

Performance

Queries

Architecture

other Non-Functional Requirements

Analyse your DataDomain-Data, Log-Data, Event-Data, Message-Data, critical Data, Business-Data, Meta-Data, temp Data, Session-Data, Geo Data, etc.

Data- / Storage-Model:relational, column-o, doc-alike, graphs, objects, etc.What Types / Type-System?Data-Navigation, Data Amount, Data Komplexity (Deep XML?)

ACID vs. BASE vs. Mixture?CAP decisions

Performance Dimension AnalysisLatency, Request behaviour, Throughput

Scale-Up vs Scale-Out

Distribution Architecturelocal, parallel, distributed / grid, service, cloud, mobile, p2p, …

Data Access Patternsread / write distribution, random / sequential, Access Design Patterns

Query RequirementsTypical queries, Tools, Ad-Hoc Queries, SQL / LINQ needed, Map/Reduce? …

Non Functional Requirements:Replication, Refactoring Frequency, DB-Support, Qualification / simplicity, Company restrictions, DB diversity (allowed?), Security, Safety / Backup & Restore, Crash Resistance, Licence…

NoSQL

FAZIT

Unbedingt RAM & SDD annehmen!

Gustavo

AlonsoLot‘s of >1 PT RAM DBs

in California!

Service, RAM, Cloud, Mobile

RethinkDB

SAP-Strategie?

DaaS Zeitalter

Alleine für MongoDB weit über 100 „Database-as-a-Service“ Provider!Amazon: SimpleDB, Hadoop, etc.

Viele clevere hybrid Lösungen!

CouchBase, Hadoop+MySQL

Database-aaS => best Mix!

OLAP

Availability

Ad Hoc Query

kritische

Daten

unkritische

Daten

(View, Domain, Stamm, Meta, Log, …)by Couch, MongoDB, Redis, Membase,

…

Management

Analytics

Zahlungsdaten, persönliche Daten, …by classic RDBMS, Vertica, VoltDB,

Database.com, GenieDB, … Hadoop* BI

OLAP BI

Dwight Merriman (10gen)

Links

• nosql-database.org

• nosqltapes.com

• mynosql.com

.com

Diskussion!

Thanks for listening!

http://edlich.de

Schutzpatent

funktionale

(graph)

Dekomposition?

Oder…

Group By

Use Case:Aggregate

pi -> 1015 -> 1000 cluster

Programmierung top! Programmierung nervt!

Nur `large data indexing`

Starke Konkurrenz: Stratosphere (TUB), ePic, SwissBox, etc.

„A giant step back!

Imcompatible, missing features,

not new, …“ Stonebraker

herrlich

paralellisierbar

Map

Reduce

Cross

Match

CoGroup

Paralellization Contracts

u.v.m… =>

Graph Ops

compile, analyze, optimize

auf einer atmenden Cloud!

ACID

BASE• Amazon Dynamo• MySQL Replikation

Eventually Consistent

WATER

Consistency Models

© Wilfried Springer NoSQL Rollercoaster

Availability

Partition ToleranceConsistency

CAP Theoreme

ACID / Isolation

Clients see equal data

System is always ‘ on‘

Clients find replicas

Pick 2!

NoSQLKlassiker

„Don‘t throw C away so easy! It‘s complex.“

What you really have is: 1. Application errors2. Repetable DBMS errors3. Unrepeatable DBMS errors4. Operating System errors5. Hardware failure in cluster6. Network partition in local cluster7. A disaster8. WAN failure

• 6 = Network Partition is rare

• 3,4,5,6 is mostly a Single Node

• Algorithms can help!

„give up P rather than sacrificing C. Use VoltDB or NimbusDB”

M:[0,5)

N:[5,10)

O:[10,15)

P:[15,20)

Q:[20,25)

R:[25,30)

HASH KNOTEN REPLIKAT

2 M N,O

8 N O,P

10 O P,Q

17 P Q,R

22 Q R,M

26 R M,N

W = 2*WR = 1*R

Consistent Hashing

• ausfallsicher • leicht erweiterbar • gut verteilt / vnodes

pessimistisches Locking?

MVCCMulti Version Concurrency

Control

laufen

L:1

surfen

P:1L:1

surfen

L:2P:1

laufen

A:1L:1

laufen

A:1L:1P:0

surfen

L:2P:1A:0

surfen

P:2A:1L:2

=>

Anna

Paul

Laura

Vector Clocks