Oracle’s Big Data solutions

<Insert Picture Here>

Oracle’s Big Data solutionsJean-Philippe BreysseOracle Suisse

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remain at the sole discretion of Oracle.

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 134

USE CASE 3: LOGS ANALYSIS OF SERVERS

Short Description :– Daily logs analysis

Issues:– Find correlations on what drives to failures– Log files stored as flat files

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 135

Oracle Technology mapped to Analytics Landscape

Acquire AnalyzeOrganizeData Decide

Stru

ctur

edS

emi-s

truct

ured

Uns

truct

ured

Master & Reference

Transactions

Machine Generated

Text, Image, Video, Audio

Oracle 12g

Files

Oracle NoSQL

Oracle Hadoop HDFS

Oracle Data Integrator

Oracle Hadoop

MapReduce

Oracle 12g

Oracle Essbas

e

Oracle R Enterprise & Oracle Data

Mining

Oracle BI Enterprise

Oracle Real Time Decisions

Oracle Endeca Information Discovery

Oracle Times Ten

Endeca MDEX

Oracle Golden Gate

Agenda

• Big Data• Solution Spectrum• Inside the Big Data Appliance• Big Data Applications Software• Big Data Analytics• Conclusions

<Insert Picture Here>

Big DataWhy Everyone Should Care

Tapping into Diverse Data Sets

Transactions

Information Architectures

Today:Decisions based on database data

Big Data:Decisions based on all your data

Video and Images

Machine-Generated DataSocial Data

Documents

9

A bit of history ...

: Developed initially by Doug Cutting (Nutch - Opensource websearch engine) and Yahoo -> inspired by Google’s papers on MapReduce and GFS (2003-2004) resulted in Apache Hadoop (2006)Amazon Dynamo (2007): distributed systems technologiesCassandra: was developed at Facebook (2008) to power their Inbox Search feature (columnar oriented distributed DB) based initially on Dynamo and Bigtable (built by Google)Voldemort: is a distributed data store that is designed as a key-value store used by LinkedIn for high-scalability storage (NoSql key value)Cloudera: . It contributes to Hadoop and related Apache projects and provides a commercial distribution of Hadoop

10

So What is Big Data Anyway?It’s a matter of perspective. Big Data is both:

• LARGE AND VARIABLE DATASETS that are difficult for traditional database tools to easily manage – including datasets that once seemed not important or too problematic to deal with. Big Data datasets include:

− Extremely large files of unstructured or semi-structured data − Large and highly distributed datasets that are otherwise difficult

to manage as a single unit of information

• NEW SET OF TECHNOLOGIES that can economically capture, store, manage, and extract value from Big Data datasets – thus facilitating better, more informed business decisions

Structured Data vs. Unstructured DataRelational databases work best with structured data – data which has underlying structure (schema) and size that easily fits the specific confines of database columns and rows. Unstructured data is highly variable, lacks fixed structure, and is often too large to easily handle by RDBMS systems.

Source: IDC Digital Universe Study, Extracting Value from Chaos, June 2011 (sponsored by EMC)

<Insert Picture Here>

Drive Value from Big DataBuilding a Big Data Platform

Divided Solution Spectrum

Acquire AnalyzeOrganize

MapReduceSolutions

DBMS (DW)

DBMS (OLTP)

Advanced Analytics

DistributedFile Systems

Transaction (Key-Value)

Stores

ETL

NoSQL Flexible

SpecializedDeveloper

Centric

SQL TrustedSecure

Administered

Schema-less

Unstructured

DataVariety

Schema

Hadoop to Oracle – Bridging the Gap

Acquire AnalyzeOrganize

HadoopMapReduce

HDFS

Cassandra

RDBMS(OLTP)

RDBMS(DW)

Advanced Analytics

ETL

Oracle Loader for Hadoop

Schema-less

Unstructured

DataVariety

Schema

Oracle Integrated Software Solution

Acquire AnalyzeOrganize

Oracle (DW)

Oracle (OLTP)

Schema-less

Unstructured

DataVariety

Schema

HadoopHDFS

Oracle NoSQL DB

OracleAnalytics:

Data Mining

RSpatialGraph

mapreduce

OBI EE

OracleData Integrator

Oracle Loader for Hadoop

<Insert Picture Here>

Inside the Big Data ApplianceOverview

16 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Insert Information Protection Policy Classification from Slide 8

Oracle Engineered Solutions

Acquire AnalyzeOrganize

OracleDatabase

(DW)

OracleDatabase

(OLTP)

In-DBAnalytics

“R”Mining

TextGraphSpatial

OracleBI EE

Oracle NoSQL DB

HDFS Hadoop

OracleData Integrator

Oracle Loader for Hadoop

DataVariety

InformationDensity

Unstructured

Schema

Big Data Appliance• Hadoop• NoSQL Database• Oracle Loader for hadoop• Oracle Data Integrator

Oracle Exadata• OLTP & DW• Data Mining & Oracle R• Semantics• Spatial

Exalytics• Speed of

ThoughtAnalytics

Big Data ApplianceUsage Model

InfiniBand

Oracle Big Data Appliance

Oracle Exadata

Acquire Organize Analyze & VisualizeStream

Oracle Exalytics

InfiniBand

•18 Sun X4270 M2 Servers– 48 GB memory per node = 864 GB memory – 12 Intel cores per node = 216 cores – 24 TB storage per node = 432 TB storage

•40 Gb p/sec InfiniBand•10 Gb p/sec Ethernet

Oracle Big Data Appliance Hardware

Scale Out to Infinity

Scale out by connecting racksto each other using Infiniband

• Expand up to eight racks without additional switches

• Scale beyond eight racks by adding an additional switch

• Oracle Enterprise Linux 5.6• Oracle Hotspot Java VM• Cloudera’s Distribution

including Apache Hadoop• Cloudera Manager• Open Source Distribution of R• Oracle NoSQL Database

Community Edition

Oracle Big Data Appliance Software

<Insert Picture Here>

Big Data Application SoftwareAcquire New Information

Key-Value Store Workloads

• Large dynamic schema based data repositories• Data capture

• Web applications (click-through capture)• Online retail• Sensor/statistics/network capture (factory automation for example)• Backup services for mobile devices

• Data services• Scalable authentication• Real-time communication (MMS, SMS, routing)• Personalization• Social Networks

Oracle NoSQL DB A distributed, scalable key-value database

• Simple Data Model• Key-value pair with major+sub-key paradigm• Read/insert/update/delete operations

• Scalability• Dynamic data partitioning and distribution• Optimized data access via intelligent driver

• High availability• One or more replicas• Disaster recovery through location of replicas• Resilient to partition master failures• No single point of failure

• Transparent load balancing• Reads from master or replicas• Driver is network topology & latency aware

• Elastic (Planned for Release 2)• Online addition/removal of Storage Nodes • Automatic data redistribution

Storage NodesData Center A

Storage NodesData Center B

NoSQLDB Driver

Application

NoSQLDB Driver

Application

<Insert Picture Here>

Big Data Application SoftwareOrganizing Data for Analysis

30 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Loader for Hadoop Features

• Load data into a partitioned or non-partitioned table– Single level, composite or interval partitioned table– Support for scalar datatypes of Oracle Database– Load into Oracle Database 11g Release 2

• Runs as a Hadoop job and supports standard options

• Pre-partitions and sorts data on Hadoop

• Online and offline load modes

31 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Loader for Hadoop

SHUFFLE/SORT

SHUFFLE/SORT

MAP

MAP

MAP

MAPSHUFFLE

/SORT

REDUCE

REDUCE

SHUFFLE/SORT

SHUFFLE/SORT

REDUCE

REDUCE

REDUCE

INPUT2

INPUT1

MAP

MAP

MAP

MAP

MAP

REDUCE

REDUCE

REDUCE

MAP

MAP

MAP

MAP

MAP

MAP

REDUCE

REDUCE

MAP

MAP

MAP

MAP

MAP

REDUCE

REDUCE

REDUCE

ORACLE LOADER FOR HADOOP

32 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Loader for Hadoop: Online Option

SHUFFLE/SORT

SHUFFLE/SORT

REDUCE

REDUCE

REDUCE

MAP

MAP

MAP

MAP

MAP

MAP

REDUCE

REDUCE

ORACLE LOADER FOR HADOOP Connect to the database from reducer nodes, load into database partitions in parallel

Read target table metadata from the database

Perform partitioning, sorting, and data conversion

33 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Loader for Hadoop: Offline Option

SHUFFLE/SORT

SHUFFLE/SORT

REDUCE

REDUCE

REDUCE

MAP

MAP

MAP

MAP

MAP

MAP

REDUCE

REDUCE

ORACLE LOADER FOR HADOOPRead target table metadata from the database

Perform partitioning, sorting, and data conversion

Write from reducer nodes to Oracle Data Pump files

Import into the database in parallel using external table mechanism

35 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Selection Output Option for Use Case

Oracle Loader for Hadoop Output Option Use Case Characteristics

Online load with JDBC The simplest use case for non partitioned tables

Online load with Direct Path Fast online load for partitioned tables

Offline load with datapump files Fastest load method for external tables

On Oracle Big Data ApplianceDirect HDFS

Leave data on HDFSParallel access from databaseImport into database when needed

36 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Automate Usage of Oracle Loader for Hadoop

• ODI has knowledge modules to – Generate data transformation code to run on Hive/Hadoop– Invoke Oracle Loader for Hadoop

• Use the drag-and-drop interface in ODI to– Include invocation of Oracle Loader for Hadoop in any ODI

packaged flow

Oracle Data Integrator (ODI)

37 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

<Insert Picture Here>

Big Data AnalyticsReal Time Analytics Platform

R Statistical Programming Language

Open source language and environment

Used for statistical computing and graphics

Strength in easily producing publication-quality plots

Highly extensible with open source community R packages

<Insert Picture Here>

Drive Value from Big DataConclusions

Big Data ApplianceBig Data for the Enterprise

• Optimized and Complete• Everything you need to store and integrate

your lower information density data• Integrated with Oracle Exadata

• Analyze all your data• Easy to Deploy

• Risk Free, Quick Installation and Setup• Single Vendor Support

• Full Oracle support for the entire system and software set

DECIDE

Oracle Analytic

Applications

Oracle Integrated Solution Stack for Big Data

ACQUIRE

Oracle NoSQL

Database

HDFS

Enterprise Applications

ORGANIZE

Hadoop(MapReduce)

Oracle Loader

for Hadoop

Oracle DataIntegrator

ANALYZE

In-D

atab

ase

Ana

lytic

s

DataWarehouse

Oracle: Big Data for the Enterprise

• The most comprehensive solution• Includes everything needed to acquire, organize and

analyze all your data• Optimized for Extreme Analytics

• Deepest analytics portfolio with access to all data• Engineered to Work Together

• Eliminate deployment risk and support risk• Enterprise Ready

• Deliver extreme performance and scalability

Questions