Upload
others
View
10
Download
0
Embed Size (px)
Citation preview
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Confidential – Oracle Internal/Restricted/Highly Restricted 2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 3
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 4
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Enterprises use NoSQL because it
Confidential – Oracle Internal/Restricted/Highly Restricted 5
Enables elastic scale to support dynamic workloads
Supports flexible data models to support any type of data
Delivers extreme read and write capabilities for demanding applications
Simplifies data management for any type of applications
Lowers data management cost.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Developer Focus
– BASE & ACID xtions
– Tables / JSON / Binary
– C, C#, Java, Python & Node.js APIs
– Secondary Indexes
• IT Operations Focus
– Elastic Configuration
– Secure Access
– Data Center Support
– Online management
• Differentiating Features
– ACID transactions
– Rich SQL dialect
– Online rolling upgrades
– Streaming large object support
– Strong Oracle technology Integration
– Engineered Systems and Commodity HW
6
Oracle NoSQL Database - Enterprise Ready
Best NOSQL choice for any company using Oracle products
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Basic Design
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Advanced Key-Value Database
Confidential – Oracle Internal/Restricted/Highly Restricted 8
Key-Value
Large Object Avro Tables
JSON ElasticSearch Streams Hive
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
capacity=x capacity=y
capacity=…
capacity=…
capacity=…
Storage Nodes and Zones
Confidential – Oracle Internal/Restricted/Highly Restricted 9
Zone 1 Zone 2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Replication Nodes and Shards
Confidential – Oracle Internal/Restricted/Highly Restricted 10
Zone 1 Zone 2
rn rn
rn
rn rn
rn
rn rn
rn rn
M R
R
M R
R
R R
R R
Shard 1
Shard 2
ZRF=3 ZRF=2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Writing
Confidential – Oracle Internal/Restricted/Highly Restricted 11
Zone 1 Zone 2
M R
R
M R
R
R R
R R
Shard 1
Shard 2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Reading
Confidential – Oracle Internal/Restricted/Highly Restricted 12
Zone 1 Zone 2
M R
R
M R
R
R R
R R
Shard 1
Shard 2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Scalability
Confidential – Oracle Internal/Restricted/Highly Restricted 13
Shard 1
Shard 2
Oracle NoSQL Client
partition P1 partition P2 ….
….partition Pn+1partition Pn
key
hash(key)
partition #
LUT
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Configurable CAP
• Greater Flexibility
– Configurable • Durability per operation
• Consistency per operation
– ACID by default
– Transaction scope is single API call
– Records share same shard key
– Multiple operations supported
14
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 15
Oracle NoSQL Database – Key value example
// Construct the keyArrayList<String> majorK = new ArrayList<String>();ArrayList<String> minorK = new ArrayList<String>();
majorK.add(“Item1”);minorK.add(“Property1”);
Key myKey = Key.createKey(majorK, minorK);
// The Data portion can be any serializable chunk of information. Here we will use a simple Java string.String myData = “This is some random text that represents the note’s content.”Value myValue = Value.createValue(myData.getBytes());
// Write the Key/Value pair to the store:kvstore.put(myKey, myValue);
What is a Key-Value ?
Key Value
Major key componentMandatory, all records sharing the same major key are allocated to the same shard
Minor key componentOptional; the key is the concatenation of the major key and the minor key components
Key Value
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 16
Tables
UserId PolicyId lname fname email
[Value]Primary Key
Shard Key
CREATE TABLE myUsers (UserId INTEGER,PolicyId INTEGER,lname STRING,fname STRING,email STRING,PRIMARY KEY (SHARD(UserId), PolicyId)
)
Complex types supported: JSON and maps
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL Database – SQL Query
17
Driver
Application
SELECT-FROM-WHERE,ORDER BY, OFFSET, LIMIT,UPDATE+SQL-like DDLCREATE, ALTER and DROP TABLE and INDEX
SQL Commands Now Available in NoSQL
Declarative query language for developers more comfortable with SQL than API level access.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL Database – UPDATE statement for SQL
Confidential – Oracle Internal/Restricted/Highly Restricted 18
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Large Object Support
• Parallel Streaming Interfaces
– Efficient storage and retrieval of large objects
– Client side streaming interface for low memory consumption
– Server side splitting and distribution of object chunks across nodes for better read/write latency
– Automatic partial LOB detection
19
Large
Object
No
SQL
DB
Dri
ver
Ap
plic
atio
n
Shard 2
Shard N
Shard 1
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL Database – Bulk Put API
20
Serial Data Store Bulk Put API
Data 1 API Call
Data 2
Data 3
Data 4
Data 1
API Call
Data 2
Data 3
Data 4
Total = 4 API Calls Total = 1 API Call
API Call
API Call
API Call
Database
Database
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL Database – Full Text Search (FTS)
21
ElasticSearch
3 - Oracle NoSQL DB Bulk-GETof primary keys
4 - Oracle NoSQL DB returnsthe records
ElasticSearch Populated Asynchronously
ElasticSearch
Oracle NoSQL DB – write data
1 – Search ES for content
2 – Returns list of primary keys of matching records
Write to the database Read from the database
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL Database – Kerberos Integration
22
NoSQL Database
Authentication Service
Ticket Granting Service
Kerberos Distribution Center
I am “[email protected]” , Please prove this to NoSQL DB
Here is a Ticket That Proves you are [email protected]
Here is a Kerberos Ticket
1
3
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL Database – Time-To-Live (TTL)• Automatic expiration of persistent records.
• An application user can specify a TTL duration for each record.
• The record would expire logically after the duration. The user can also update the TTL before a record expires.
23
Record 1
Expires in 2 hours
Record 2
Expires in 3 hours
Record 3
Expires in 24 hours
Record 4
Expires in 30 hours
Record 5
Never expire
Record 1
Record 2
Record 3
Record 4
Record 5
Record 2
Record 3
Record 4
Record 5
Record 3
Record 4
Record 5
Record 4
Record 5 Record 5
Time – Not To Scale
Dat
abas
e S
ize
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL Database Streams API
• Changes to the Store can be asynchronously forwarded to stream processing application
• Compliant with http://www.reactive-streams.org
Confidential – Oracle Internal/Restricted/Highly Restricted 24
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Drivers
Confidential – Oracle Internal/Restricted/Highly Restricted 25
Driver
Application JavaPythonNode.jsRESTCC#
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
NoSQL DB integration - Hadoop
HDFS
HDFS Files
REDUCE
MAP
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
NoSQL DB integration – Oracle RDBMS
Externaltable
SELECT name FROM unstruct_people
SQL engine
nosql_streampreprocessor
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 28
Integrating the best of breed
Oracle Big Data SQL
One fast SQL query, on all your data, in parallel
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Integration with Apache Hive and Oracle Big Data SQL
• Oracle Big Data SQL
– Common SQL access layer to data stored in Hadoop HDFS, Hive and Oracle NoSQL Database
– Defines new external table access driver types. ORACLE_HIVE for accessing data stored in or referenced by Hive Tables
– Query Oracle NoSQL Data from Hive or Oracle Database
– Runs Map Reduce Jobs against data stored in Oracle NoSQL Database
• Support for primitive and complex data types
• Common data definition between Oracle NoSQL DB, Hive and Oracle Database
29
BIG DATA SQL
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
NoSQL DB integration – Oracle Coherence
NoSQL DB Driver
TB
Mem
ory
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle Big Data Spatial and Graph (BDSG)
Property Graph Database
Spatial Analysis Features
31
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle Big Data Spatial and Graph
Access Layer
Oracle Big Data Spatial and GraphProperty Graph Architecture
Graph Analytics
Apache Blueprints & Lucene/SolrCloud RDF (RDF/XML, N-Triples, N-Quads,
TriG,N3,JSON)
REST/W
eb
Service
Java, Gro
ovy, P
ytho
n, …
Java APIs
Java APIs/JDBC/SQL/PLSQL
Property graph formats supported
GraphMLGML
Graph-SON Flat Files
CSVRelational Data Sources
32
Oracle NoSQL Database
Apache HBase
Parallel In-Memory Graph Analytics (PGX)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Where is NoSQL used?
33
Simple Data Management
Globally Distributed,
“Always On” data
Competitive Advantages of
“Fast Data”
Lower TCO, commodity
HW scale-out
ERP
EAM
Inventory Control
Accting & Payroll
Process Mgmt
Business Analytics
CRM
…
Driver
ApplicationReal Time Event
Processing
Customer Portals
Online Banking
…
Mobile Data Management
Time Series & Sensor Data Mgmt
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Choose the RIGHT storage option for the job
Hadoop Distributed File System (HDFS)
Oracle NoSQL Database Oracle Database
File System Key-Value Database Relational Database
No inherent structure Simple data structure Complex data structures, rich SQL
High volume writesHigh volume random reads and
writesHigh volume OLTP with 2-PC
Limited functionality,roll-your-own applications
Simple get/put high speed storage, flex configuration
Security, Backup/Restore, Data life cycle mgmt, XML, etc.
Batch OrientedReal-Time, web-scale specialized
applicationsGeneral purpose SQL platform, multiple
applications, ODBC, JDBC
34
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 35
Oracle NoSQL Database - Flavors
Enterprise Edition (EE) Basic Edition (BE) Community Edition (CE)
Connectors to Oracle Products Same as CE Open Source
Enhanced Security Same as CE Full Performance, Same API
Licensed per OCPU License with BDEE List Price - $ 0
Support Available ($) Support available with DB EE support
Support Available ($) / Community
Commercial Oracle License Licensed together with Oracle DB EE
AGPL V3 license
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Common NoSQL applications
• Customer portals (browser, mobile…)
• Real-time event processing
• Mobile Data Management
• Time-series data management
• Online banking
36
• Customer 360
• Customer loyalty programs
• Profile management
• Product catalog
• Content management
• Digital communication (e.g. “click-to-chat”)
• …
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Finance
• Large credit card network provider
• Fraud rule and history management
• 6,500 ops/sec, 85% update on read
Manufacturing
• Large airline manufacturer
• Developing next generation planes
• 750K key inserts/sec, 6TB/hour
37
Telecoms
• Large mobile telecom operator
• Mobile personalized advertizing
• 24,000 ops/sec, 100% update on read
Customer Loyalty
• Operationalize analytics to NoSQL Customer Portal
• Addressed processing conflicts
• 20M+ customers in multiple countries
Oracle NoSQL Database customer success…
……many more
Airbus
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Benefits
Scalable multi-channel customer portal
Predictable performance for all operations
Reduced time to market
Easy application evolution
Coupon redemption, Vendor recommendation
NoSQL for Customer Loyalty
Objectives
Scalable customer loyalty portal
New multi-channel consumer model
Improve operational efficiency
Solution
Personalized multi-channel coupon generation and redemption
Cross-promote affiliated vendors
Scale system with customers and participating retailers
NoSQL DB DriverApplication
Retail Partners
Customer Profiles
End Customers
Available CouponsMarket Segmentation
38
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Financial Services coordinated theft prevention
NoSQL for Fraud Scoring
Objectives
Solution
Benefits
Application Data Ingestion
Tran
sact
ion
Au
tho
riza
tio
nP
roce
sso
r
Combine data sources for complex scoring
Detect, alert analyst with low latency
Handle burst seasonal transaction volumes
Oracle Coherence cluster for real time transaction object management
Oracle NoSQL Database for fraud model and customer profile management
Oracle Database for statistics and fraud modeling-related data
Simple data model, ACID transactions
Scalability, Reliability, Low Latency
Elasticity of sharded data repositories
Easy configuration and administration
NoSQL DB Driver
39
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Objectives
Solution
Benefits
• Increase scalability of data storage•Deliver higher concurrency analytic data access•Scale data loading independently from analysis•Commercial support for mission critical system
•Oracle NoSQL database for high speed storage and range based extraction of time series data.•Oracle NoSQL database for agile schema, replaced
HDF5 storage format, kept analysis client program•Oracle Big Data Appliance for efficient manageability
and lowest TCO•Hadoop post processing and RDBMS connectivity to
Enterprise systems
• Improve scale of storage for flight test sensor data• Increase concurrency of access to data for analysis• Improve system availability for analysts by allowing
simultaneous data ingestion and analysis
Big Data Appliance
NoSQL DB Driver
Event Ingestion and Extraction
NoSQLDB/Oracle RDBMS
Hadoop/Oracle RDBMS
Oracle or Any third parties
SQL/Data Analytics Tools
NoSQL for Sensor Event Storage & ProcessingLarge scale sensor data capture and analysis
40
Airbus
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Product Recommendation Platform
41
Elastic Data Store
RecommendationRule Engine
SourceSystems
Admin
Customers
Corporate Web Page Recommendation Platform
Rules
CustomerInfo
Recommend
User Info, Billing Info,etc
Request RespondLogin
User ID
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Performance Comparison
42
0
10
20
30
40
50
60
70
0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
# o
f se
rve
rs
Transactionsper second
NoSQL HDD
NoSQL SSD
HBASE HDD
1. Oracle NoSQL Database with SSDs achieves 26000 tps with 3 nodes.2. Compared to HBase, Oracle NoSQL Database achieves high performance goals with much smaller number
of nodes.3. Considering high performance goals combined with additional future requirements, the total cost of Oracle
NoSQL Database is estimated to be much lower than HBase.
Over 20,000 tpscan be handled
with just 3 nodes
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL DB competition
43
Oracle NoSQL Database
Linear Scalability 200+
nodes
ACID, Configurable
transaction semantics
Flexible Schema – KV,
JSON, Table
Only NoSQL store with
Enterprise level support.
Easier to setup,
upgrades
Better performance &
predictability.
Tight integration with
Oracle ecosystem –
Database, FMW, Big
Data Appliance
Vs. MongoDB
Well known scalability
problems. (does not
scale well beyond 6
nodes)
Lacks write concurrency
due to collection level
locking. Can lose data.
Update and delete
operations perform
poorly.
Schema Bloat
3-6X slower than Oracle
NoSQL DB.
Vs. Cassandra (Datastax)
Inconsistent reads due
to background
operations.
Columnar, KV, Table
Poor performance when
used with Hadoop.
Difficult to keep data in
sync across the cluster.
Hard to manage large
clusters.
Inefficient I/O and
memory usage.
2-3X slower than Oracle
NoSQL DB.
Vs. HBase
Clusters are complex to
run and hard to manage.
Columnar, KV, Table
Less flexible transaction
model. Can lose updates.
High memory usage.
Inconsistent performance
Difficult to configure.
Less flexible transaction
model.
2-3X slower on inserts
Performance, reliability, scalability, integration
Not Integrated with the Oracle Stack
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Performance comparisonOracle NoSQL DB, Cassandra, Mongo DB
Benchmark specifications
• Yahoo! Cloud Serving
Benchmark (YCSB)
[13 byte key; 1100 byte value]
• 60 Million records in database
(~60 GB of data)
• 3 Node system
(X4170 m2: 2 Xeon(R) X5670
2.93GHz; 72 GB; SSD drives)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Much better READ Performance• Lower READ Latency
• Stable Performance
• Linear Scalability
Setup
• Replication Factor (RF) = 3
• Each BDA node has 12 Disk
NoSQL DB Highlights
HBase vs Oracle NoSQL DatabaseYCSB Benchmark Results
0
10
20
30
40
50
60
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
6 (672M) 12 (1344M)
Ave
rage
Lat
en
cy (
ms)
Thro
ugh
pu
t (o
ps/
sec)
BDA Nodes (record count)
95/5 Read/Update Throughput
NoSQL Throughput (ops/sec) HBase Throughput (ops/sec)
NoSQL Read Latency (ms) Hbase Read Latency (ms)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Predictability, Reliability & Support
• Global, mission-critical application deployment experience
• Decades of enterprise-grade non-relational database technology
• Oracle Support available for both Enterprise and Community Edition
• Designed for Predictability and Manageability
Bulk Insert Test Cluster Expansion Test Rolling Upgrade Test
46
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
YCSB on SSD-backed commodity servers
• What’s the big deal
– Twitter sees ~500M tweets/day• This is 350K a minute
• Capture all tweets with 3 commodity servers
– 1.25M ops/sec
– 2 billion records
– 2 TB of data
– 95% read, 5% update
– Low latency, High Scalability
47
0
1
2
3
4
0
200,000
400,000
600,000
800,000
1,000,000
1,200,000
1,400,000
6 (2x3) 12 (4x3) 24 (8x3) 30 (10x3)
Ave
rage
Lat
en
cy (
ms)
Thro
ugh
pu
t (o
ps/
sec)
Cluster Size ( 3->15 servers )
Mixed Throughput
Throughput (ops/sec) Write Latency (ms)
Read Latency (ms)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Online Rolling Upgrades
• Linear time impact for upgrade– Admin commands available to describe
safe upgrade order– Scripted available hands-free upgrade
experience– Read/Write availability throughout the
upgrade process
200 Node version upgrade in 15 minutes
48
0
2.5
5
7.5
10
12.5
15
17.5
72 (24x3) 144 (48x3) 216 (72x3)
Tim
e t
o U
pgr
ade
(m
in)
Total Nodes (Shards x Rep. Factor)
Online Rolling Upgrade
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Cluster expansion while running YCSB
• What we’re showing
– YCSB running on a 144-node system
– Add 50% more HW
– Gradual increase in throughput, decrease in latency
– Ability to increase workload by 50%
– No interruption to YCSB application
49
0
1
2
3
4
5
6
7
8
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
144(48x3)
20% 40% 60% 80% 216(72x3)
216*(72x3)
Ave
rage
Lat
en
cy (
ms)
Thro
ugh
pu
t (o
ps/
sec)
Nodes (Shards x RF)
95/5 Read/Update Throughput
Throughput (ops/sec) Read Latency (ms)
Update Latency (ms)
* = Clients increased as well