Oracle Data Integration CON9737 at OpenWorld

Oracle Data IntegrationOVERVIEW | VISION | ROADMAP

Jeff PollockVice President, Product ManagementOctober 2015

Oracle Open World 2015Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Oracle Open World 2015 2


Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Oracle Open World 2015 4

Oracle Data Integration Core Business Use Cases

Analytics & Business Intelligence• Big Data Integration & Governance• Data Warehouse / Mart Integration

IT Data Integration & Governance• Enterprise Shared/Common Data Services• Move, Migrate, Transform, Profile, Cleanse,

Glossary and Metadata Management

Business Continuity• Active-Active for Maximum Availability• Zero Downtime Migrations

24 x 7 x 365

Cloud

On Premise


NoETL Engine100% Native Data

Transformation

DataIntegrator

Big DataPreparation

GoldenGate

DataQuality

Data ServiceIntegrator

MetadataManagement

Non-invasive CDC, Realtime streaming

data delivery

Profile, Cleanse, Match, and

Remediate Data

Prepare, Secure, Enrich and Publish Unstructured Data

Catalog, Trace and View Models across

the Enterprise

Federate Data Across DBs, Services

and Applications


Oracle is Different from Other Data Integration VendorsBETTER WITH OPEN SOURCE

MODERN DATA ARCHITECTURE

MORE GOVERNANCE COVERAGE

HYBRID DEPLOYMENT

90+…tools supported by

Business Glossary and Metadata Management 15more than

different certified open source technologies

Zero …additional hardware for data replicationor transformation Public | Managed | Private

GoldenGate

Data Integrator

Data Quality

Data Preparation


Our Goal: Lead by Innovation with Open Source

Data Preparation:• Product: Oracle Big Data Preparation• The Market: Red-hot subcategory of Big Data• Unique About Oracle: Semantics!

Data Ingestion:• Product: Oracle GoldenGate for Big Data• The Market: Core aspect of Big Data market• Unique About Oracle: Streaming from DBMS!

Data Transformation:• Product: Oracle Data Integrator Advanced Big Data• The Market: ETL Offload in Big Data• Unique About Oracle: 100% Native & Decoupled!

Metadata Governance:• Product: Oracle Enterprise Metadata Management• The Market: Data Governance• Unique About Oracle: 70+ harvesters for 3rd party!

1st

1st

1st

1st

to move beyond Machine Learning and to embrace Semantics (NLP & Graph) for Data Preparation

to deliver DBMS transactions in realtime to Big Data Streaming platforms like Flume and Kafka

to support 100% native Spark based ETL using a graphical, logically de-coupled design studio

to include wide range of Big Data metadata harvesting fully supported as part of 70+ technology bridges


Changing Concepts About Modern Data Management

Polyglot

Fit for Purpose Data

Lambda

Speed Layer

Batch Layer

DataSources

DataServices

Dataflow

DataServices

Data PipelineDataSources


Staging


Modern Platform for Next Generation Solutions

Sqoop

HDFS

Hive

Flume

Data StreamingKafka (MPP Pub/Sub)

Storm and Trident

Spark Streaming

HBase

Unstructured andSemi-Structured Data

Structured Data

Lambda/Kappa Speed Layer

Lambda Batch Layer

Apps

Analytics& DWs

Oracle Data Governance Foundation

Oracle Data Integrator

Oracle Enterprise Data Quality

Oracle Enterprise Metadata Management with Business Glossary

Oracle GoldenGate

Oracle Data Preparation


Oracle Data Governance Foundation

Staging


100% Open and Heterogeneous Solution

Sqoop

HDFS

Hive

Flume

ETL Offload / Transformation

Data StreamingKafka (MPP Pub/Sub)

Storm and Trident

Spark Streaming

HBase

Discovery Sandbox/s

R

Un

stru

ctu

red

Stru

ctu

red



Oracle Enterprise Metadata Management with Business Glossary

Oracle GoldenGate

Oracle Data Preparation


BIG DATA PREPARATION


Oracle Big Data Preparation for Data Without Schema

Big Data Preparation Cloud Service

Data Preparation Designed for data domain

experts, not programmers

Focused on cleansing, enriching and transforming unstructured business data

Operationalize data flows into ETL or Business Intelligence

Key Benefits Easy to get started with

browser based application

Better recommendations engine combines machine learning with semantic technologies

Integrated into Oracle Cloud


Oracle Big Data Preparation Core Capabilities

Publish

• On Demand or Scheduled

• Source / TargetDefinition

• Restful APIs• Event Driven

Govern and Monitor

• Dashboards• Automated Alerts

• Reusable user policies• System Controls

• Security• Stats & Metrics

Ingest

• Import & Ingest• Detect Schema• Cleanse, Normalize

& De-duplicate• Detect & Mask

Sensitive Data

Enrich

• Profile• Annotate• Data Classification• Semantic

Enrichments• Missing Data

Interpolation

• Unified solution to prepareunstructured data

• Simple to use tooling designed for non-programmers

• Unique technology approach combines Machine Learning (ML) with Natural Language Processing (NLP) engine

• Powered by Apache Spark, Hadoop, and UIMA

• Cloud operated from the Oracle Public Cloud


Load Data for Oracle Business Intelligence Cloud Service

Easy Publishing to Oracle BICS Pre-integrated RESTful service

connectivity

Shared single sign on access across Oracle Cloud

Common operational support from OPC services

Key Benefits Self Service access from non

technical users

Cloud based applications can bypass need for extensive IT support for on premise tech

Accelerated Value by enabling business users to quickly ingest data for operational BI

BDP-CS BICS


Highly Differentiated from Other Data Prep Tools

Better Recommendations Engine Only Data Prep/Wrangling tool

to combing Natural Language Processing (Apache NLP) with Machine Learning (Spark ML)

Leverages Linked Open Data graph of domain knowledge

Key Benefits More Efficient Mapping by

leveraging a more effective recommendation service

Higher Quality automation “gets it right” more often

Data Enrichment leveraging domain data for enriching sparse data sets

Hadoop

OracleCloud

Semantics based

Knowledge Graph

SparkMachineLearning

NaturalLanguage

Processing


GOLDENGATE


Announcing: Oracle GoldenGate New Capabilities

Zero Downtime Upgrades & Data Migration

High Availability& Disaster Recovery

Application andReport Offloading

Big Data, Data Warehouseand Data Marts

Message Bus, SOA/JMS, Coherence

Global Data SynchronizationAcross the Enterprise

Real-time Analytics& Data Streaming

New Capabilities GG Studio for easy design

Metadata in Trail File

Delivery to Kafka

Enhanced Monitoring w/Lag

Performance Improvements

MySQL 5.7. SQL Server SE, ALO

Key Benefits Reliability most proven CDC on

the planet – don’t lose data!

Performance when low latency matters most – GG is the best

Breadth of coverage on all your targets, Oracle, Non-Oracle, Big Data and NoSQL

Cap

ture

Trai

l

Ro

ute

Del

iver

Pu

mp

GoldenGate


Streaming Big Data

Batch Big Data

NoSQL

Cap

ture

Trai

l

Ro

ute

Del

iver

Pu

mp


NEW: Oracle GoldenGate for Big Data

GoldenGate

GoldenGate for Big Data Same trusted architecture used

by 1000’s of customers

Data delivery to Big Data

Lambda/Kappa streaming data

Key Benefits Less invasive on Source DBs

when compared to Batch Processing such as Sqoop or ETL processes

Simple ingestion for 1:1 data architecture for populating “raw data” zones

Real-time data delivery for streaming analytics/apps

Reliable, proven at scale


Database as a Service

Hadoop as a Service

3rd Party Cloud

Cap

ture

Trai

l

Ro

ute

Del

iver

Pu

mp


NEW: Oracle GoldenGate Cloud Service

GoldenGate Cloud Service Available in the Public Cloud via

Subscription or Hourly basis

Key Benefits Oracle Database Cloud Service

delivery via GoldenGate

Exadata Cloud Service delivery via GoldenGate

Big Data Cloud Service delivery to Hadoop and NoSQL

3rd Party Cloud More Choices run as BYOL on

other Clouds for delivery to any supported Database

On Premise

DatabaseCloud Service

ExadataCloud Service

Big DataCloud Service

GGCS

GGCS

GoldenGate


DATA INTEGRATOR


Proprietary ETL engines die circa

2015 – folded into big data


Proprietary ETL is Dead. Apache-based ETL is What’s Next

ScriptedSQL

StoredProcs

ODI forColumnar

ODI forIn-Mem

ODI forExadata

ODI forHive

ODI forPig & Oozie

1990’s

Eon of Scripts and PL-SQL Era of SQL E-LT/Pushdown Big Data ETL in Batch Streaming ETL

Period of Proprietary Batch ETL Engines

Informatica

Ascential/IBM

Ab Initio

Acta/SAP

SyncSort

1994


ODI forSpark

ODI forSpark Streaming

WarehouseBuilder


Announcing: Oracle Data Integrator New Capabilities

On Any DB/DWOn Exadata On Big Data

0

50

100

150

200

250

300

350

42XFASTER

ETL

INFA

/ IB

M

Best ETL for any Exadata App

Run ODI on the Exadata Host

ODI Optimized for SmartScan

Proven w/1000’s of customers

Native SQL for 3rd Party DB’s

Faster ETL and lower TCO


First to market with Spark

Reuse maps across languages

Executable runs anywhere

New Capabilities Lifecycle Management updates

to core versioning systems

Simplify setup and deployment of Big Data transformations

Performance improvements across all E-LT platforms

Key Benefits Open Source friendly – run E-LT

on Apache Spark/Hadoop

Best Performance – take processing to the data

No Hardware required – run on the DB host or on Apache big data cluster


Oracle Data Integrator Advanced Big Data Edition

SQOOP

OGG


SQOOP+ NativeLoaders

API/File

Flume/Kafka

OLH/OSCH

Big Data SQL

Data Integrator for Big Data Batch data ingestion with

Sqoop or native loaders

Generate data transformations in Hive, Pig or Spark

Extract data into external DBs, Files or Cloud

Key Benefits NoETL Engine designed from

scratch for E-LT execution

Non-invasive does not require any install on cluster

Decoupled mappings allow for portable runtime execution, saving development time, and simplifying debugging


Oracle Data Integrator for Cloud

Data Integrator for the Cloud Deploy ODI anywhere and

execute ETL on the data

Key Benefits Integrated with OPC

Flexible Deployment architecture that can run on-premise, Java Cloud Service or Big Data Cloud Service

Broad Connectvity for sourcing data from any Apps or DBs

3rd Party Cloud

Flexible run ODI with BYOL on other clouds

On Premise

ODI

SaaS Apps


ODI/EDQ

JavaCloud Service

ODI


DatabaseCloud Service

ExadataCloud Service

StorageCloud Service

Business IntelligenceCloud Service


Oracle Data Integrator Call for Streaming ETL Beta

Contact Us Now! Beta is open to existing ODI

customers, try out Spark Streaming ETL with right now!

Key Benefits1. Shorter batch windows

• Processing before staging (vs after)

2. Multi-use from the engine• In line analytics on the

stream (not possible w/older ETL engines)

• Data pipeline for Log data ingest (not possible at scale w/older ETL engines)


EDWData Transformation

Batch Process

EDW

Micro Batch

Batch Engine

HDFS

vs. Batch ETL

Streaming ETL with Spark Streaming

Events

Analysis

Staging

Staging

Data Transformation

BI

BI


METADATA MANAGEMENT


Announcing: Oracle Metadata Management 12.2.1

1. Business Glossary – simple tools to catalog, link and collaborate on business terms

2. Metadata Management – horizontal and semantic data lineage for all big data sources

Business/Metadata Catalog Report to Source Lineage

Impact Analysis

Audit, Versioning & Diff Reports

Social/Collaboration Features

Annotations and Tagging

Automatic Stitching

Comprehensive Harvesting 3rd Party BI Metadata

3rd Party ETL Metadata

3rd Party DB Metadata

3rd Party Modeling Tools

Hadoop & NoSQL Metadata

Metadata Standards


Data Lineage in Three Crucial Varieties

Vertical Lineage For the Data Stewards Links a business friendly set of

terms to the IT metadata and operational assets

Links Business Glossary, Taxonomy, Ontology, Conceptual Models to Technical Metadata (Columns, Files, Objects etc)

Horizontal Column Level For the Architects Links the data fields from Business

Intelligence Dashboards or Reports back to the Source DB/App Columns

Schemas, BI View Layers, ETL Transformations, Calculations, etc.

Business Lineage For the Business Users Links a business friendly set of

Glossary terms to the BI reporting models (eg; Semantic Models, Universe, etc.)

Links Business Glossary, Taxonomy, Ontology, Conceptual Models to Reporting Fields

BIZTERM

BIZTERM

REPORTFIELD

DBCOLS REPORT

FIELDSOURCE

COLS


Open Source, Standards and Third Party Integrations

Adaptive Altova Apache HCatalog Apache Hive/HQL Borland CA ERwin Cloudera Impala COBOL Copybook DataStax Embarcadero EMC ProActivity GentleWare Google BigQuery Grandite Hadapt Hive Hortonworks Hive IBM Cognos IBM DB2 IBM DataStage IBM Discovery IBM Federation Server IBM Lotus Notes IBM Netezza IBM Rational Rose IBM Rational Architect Informatica Metadata Manager

Informatica PowerCenter CoSORT ISO SQL Standard (DDL) MapR Hadoop Hive MicroFocus Microsoft Access Microsoft Office Excel Microsoft Visio Microsoft SQL Server Microsoft SSIS Microsoft Visual Studio Microstrategy Magic Draw OMG CWM Standard OMG UML Standard Oracle BI Answers Oracle BI Enterprise Edition Oracle BI Server Oracle DAC Oracle Data Integrator Oracle Data Modeler Oracle Database Oracle Designer Oracle Hyperion Applications Oracle Hyperion Essbase Oracle Warehouse Builder

Pivotal Greenplum PostgreSQL QlikView SAP BO Crystal Reports SAP BO Designer SAP BO Desktop Intelligence SAP BO Repository SAP BO Data Integrator SAP BO Data Steward SAP Master Data Management SAP Sybase PowerDesigner SAP Sybase ASE Database SAS Data Integration Studio SAS BI Server SAS Information Map SAS Metadata Management SAS OLAP Server Select Sparx Architect Syncsort Tableau Talend Teradata Tigris Visible W3C DTD & XSD Schema

Metadata Harvesting (Glossary, Lineage & Impact Analysis)Key Standards


DATA QUALITY


Announcing: Oracle Enterprise Data Quality 12.2.1

New Capabilities Enhanced High Availabilty

WebLogic Clustering

DB Failover Support

Improved Matching Flexibility

Scoring for Matching

Easier Definitions

Hadoop/Hive Certification

Audit Framework Support

REST Web Services

Key Benefits Simpler tools to very efficiently

create match rules

Accuracy of scoring models and easier access to parameters


Fusion DQ (EDQ)

EDQ 12c

Profile Audit Transform Match Parse

WebLogic 12c

DaaS DQ (EDQ)

EDQ


Pragmatic Operational Governance Focus

Profile and Cleanse Data Flows

Quality KPIs Case Management

Governance Cockpit for Data Stewards & Stakeholders

Exception Review

Metadata Management

Business Glossary

Prepare & Enrich

Data Integration

http://hadoop.apache.org/

http://hadoop.apache.org/

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 33

Data Quality Core Strategy is Two-Pronged

2. BEST FOR ORACLE RED-STACK1. BEST OF BREED DATA GOVERNANCE

1. EDQ for Oracle MDM –Customer Hubs

2. EDQ for Oracle MDM –Product Hubs

3. EDQ for Oracle Fusion Applications

4. EDQ for Oracle SaaS Applications

5. EDQ for Oracle DaaSServices

6. EDQ for Oracle Database

7. EDQ for Oracle Data Integration & Governance

Oracle Open World 2015


DATA FEDERATION


Industry Broadest Data Federation Capabilities


Oracle Big DataSpatial and Graph

B B B

Oracle Data Service Integrator Oracle Big Data SQL

DBAsApp Dev

TransactionalData Federation

AnalyticData Federation

From a single Oracle SELECT statement…

getObject()Web Services

/ XQuery


VISION & VALUE


DataIntegrator

Big DataPreparation

GoldenGate

DataQuality

Data ServiceIntegrator

MetadataManagement


Our Vision: Polyglot | Lambda | Dataflow | Open| Cloud

Data Anywhere Cloud Infrastructure

Data in Parallel Open Source Platforms

Any Data Action Dataflow | Pipes

Any Data Speed Lambda

Any Data Shape Polyglot

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 39

DataflowML – A Data Integration Fabric with Smarts

Come see me at Oracle OpenWorld for more

information about this one!!


Our Vision: ALL Capabilities on the Cloud

Data Integration

SaaS Data Quality

DaaS Data Quality

ODI on Java Cloud Service

EDQ on Java Cloud Service

MM on Java Cloud Service*

Big Data Preparation

GoldenGateCloud Service


GG on Big Data Cloud Service*

ODI on Big DataCloud Service

* Roadmap


Safe Harbor Statement

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.



Presen-tationson:

Oracle Confidential 42

Data Integration Solutions Program - tinyurl.com/DISOOW15

DemoStations:

Hands-on labs:

OracleEnterprise Metadata

Management

OracleEnterprise

Data Quality

Oracle GoldenGate

OracleData

Integrator

OracleBig Data

PreparationCloud Service

OracleEnterprise

Data QualityHOL10464

Oracle GoldenGateDeep DiveHOL10460

ODI and OGGFor Big DataHOL10463

Big DataPreparation

Cloud ServiceHOL10467

Oracle DataIntegratorDeep DiveHOL10462

MiddlewareDemoground

- Moscone South

Oracle Cloud Platform andInfrastructure Showcase

- Moscone West

Big Data Showcase

- Moscone South

DatabaseDemoground

- Moscone South

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 43

Data Integration Solutions Program - tinyurl.com/DISOOW15

Monday, Oct 26• Oracle Data Integration Solutions—Platform Overview and Roadmap

[CON9737]• Oracle GoldenGate Product Update and Strategy [CON9720]

• How Oracle GoldenGate Helped Increase Data Availability for

Financial Close by 600% at Humana [CON4398]• Into the Cloud and Back with Oracle Data Integrator 12c [CON3506]

Tuesday, Oct 27• Oracle Data Integrator Product Update and Strategy [CON9601]

• Oracle GoldenGate 12.2 New Features Deep Dive [CON8737]

• The Role of Data Governance in Data Integration [CON9736]

• Oracle Fusion Middleware: Meet This Year’s Most Impressive

Innovators [CON10374]

• The Oracle Enterprise Data Quality Platform—Product Overview and

Roadmap [CON9587]

• Enabling Real-Time Data Integration with Big Data [CON9724]

• Solving the “Dirty Secret” of Big Data with Oracle Big Data

Preparation Cloud Service [CON9615]

Wednesday, Oct 28• Oracle Data Integration Solutions: the Foundation for Cloud Integration

[CON9614]

• Oracle Data Integration Product Family: a Cornerstone for Big Data

[CON9609]

• The Essential Core of Data Governance with Oracle Enterprise Data

Quality [CON9593]

• Accelerate Cloud Onboarding Using Oracle GoldenGate Cloud Service

[CON9717]

• Introducing Oracle GoldenGate Studio for Faster Design and

Deployment [CON9713]

Thursday, Oct 29• Integrating Big Data: Balancing the Data Warehouse and the Data

Reservoir [CON8768]

• Zero-Downtime Upgrading Using Oracle GoldenGate [CON7892]

• Best Practices for Migrating to Oracle Data Integrator [CON9604]

• Oracle GoldenGate Performance Tuning and Monitoring [CON8736]

• Cloud Data Quality: Lessons Learned from Oracle’s Journey to the

Sales Cloud [CON9595]

• Oracle Enterprise Metadata Management and the Cloud [CON9612]

• Developer Best Practices for Oracle Data Integrator Lifecycle

Management [CON9611]

• Oracle Management Pack for Oracle GoldenGate Update [CON9716]


Connect with Oracle Data Integration

@OracleDI

Blogs.oracle.com/DataIntegration/

Oracle Data Integration

Oracle Data Integration



Software

Oracle Data Integration CON9737 at OpenWorld