01 coherence & golden gate - technical overview

Coherence GoldenGate

Oracle Coherence & GoldenGateTechnical Overview

Insert Picture Here

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remain at the sole discretion of Oracle.

Agenda• Coherence Overview• GoldenGate Overview• Integration


Copyright © 2013 Oracle and/or its affiliates. All rights reserved.5

The Need for an In-Memory Data GridScalability, Performance and Reliability at Internet Scale

ScalabilityPerformance

Less # hitsBetter cost

DataSupply

ApplicationServers

Web Servers

Universe of Users


Modern Caching Architecture Pioneered by Coherence

First Generation Cache Solutions(still commonly seen today)

• Difficult to scale – must scale vertically only• Complex to keep cache synchronized• Often performance problems due to large

memory usage, large data sizes

Application

Collocated

Local Cache

Application

Collocated

Local Cache

• Easy to scale – scales horizontally & vertically• Clustered caching avoids synchronization issues• Can dramatically improve performance with

parallel query capabilities and compute

Coherence Cache Solutions

Application

Collocated

Local Cache

Application

Collocated

Local Cache

Data Fabric


The Data Grid at a Glance

• Lower cost / Backend systems protection- Less #hits to backend systems- Match data supply with data demand by without scaling-up backends- Extreme scale-out using commodity hardware

• Enhanced performance / Better user-experience- In-memory data access – information close in memory, not in backend

systems• Enhanced availability / Better user-experience

- Reliable in-memory data grid providing high-availability to applications- Reliable transaction processing at any load

Major Benefits


Oracle Coherence Through 12cPioneer & Market Leader Since 2001

2001 ….. 2008 20092000 2007 2010

Coherence

1.0

2011

Tangosol founded to

pioneer distributed

caching

Cool Vendor Award

Oracle acquires Tangosol

Coherence

3.7.1.1

Coherence

3.6

Coherence

3.5

Coherence

3.7

Increasing market share and feature capabilities for In Memory Data Grid. Market widened All Categories – Financial Services, E-Commerce, Travel, Insurance,

Retail, Manufacturing, Telco …

Deep adoption in Fusion Middleware, Oracle Horizontal and Vertical and Business Applications and Exalogic

2012 2013

Coherence

12.1.2

Alignment with Fusion Middleware

Releases


Oracle Coherence DefinedDistributed in-Memory Object-Oriented Data Grid

No cost per object assembly

Object – Oriented

Distributed Grid / Clustered Caching

• Identified by a key (key-value pair)• Any serializable object • No proprietary classes to extend• Not forced to use relational models, ORMs, SQL • Fully native Java, .NET , C++ interoperability

KEY

• Data distribution > massive scale-out• Backup replicas > High availability• No single points of failure (SPoF)• No single points of bottleneck (SPoB)


Coherence Clustered Caching ExplainedPartitioned, Fault Tolerant, Self-Healing Cache

• All nodes verify health of each other• In the event a node is unhealthy, other nodes diagnose

state?

• Unhealthy node isolated from cluster• Remaining nodes redistribute primary and back-up

responsibilities to healthy nodes

X

• Cluster of nodes holding % of primary data locally

• Back-up of primary data is distributed across all other nodes

• Logical view of all data from any node


Data Distribution Options

• Data is load balanced across the data grid• Data and processing capacity scales linearly • Ownership responsibilities also partitioned• Access and update latency are constant

Partitioned Caching > Scalability



• Automatically manages the fault tolerance of your data• Backups guaranteed to be on separate node / machine /

rack / site from primary• Backup for data owned by one node is shared across the

others

Partitioned Caching w/ Backup Replicas > High Availability

Backup Storage

Primary Storage



• Storage enabled grid members > Cache Servers• Storage disabled grid members > Cache Clients

Cache Server vs. Cache Client > Application Protection

Application Application Application Cache Clients

Cache Servers



• 2-level cache• Most recently/frequently used data is stored locally

Near Caching > Enhanced Performance

Cache Clients

Cache Servers

Application Application Application

Front Local Cache

Back Distributed Cache


Querying The Data grid

• Programmatic & SQL-like query mechanism• Queries performed in parallel across the grid• Index support for better performance• Aggregation support

Parallel Queries & Aggregation

Application


Querying The Data grid

• Snapshot automatically updated as back entries change• Event – based

Continuous Query Cache

Application


Event Notifications

• JavaBean Model (addListener), key and filter based events• Real-time notifications• Full-state & Lite events

Real Time Event Driven Architectures

Application

!

Application

!

Application

!


Processing on the grid

• Sends the processing to where the data is in the grid- Once and only once guarantees / automatically fault-tolerant

• Key(s) and filter based processing

In-place Data Processing and Map/Reduce Aggregation

Application

ProcessingUnit


Managing PersistenceCaching Patterns

• Cache Aside – Architecture manages cache‒ Check the cache before reading from data source

‒ Put data into cache after reading from data source

‒ Evict or update cache when updating data source Cache

DAO

CacheDAO

CacheDAO

• Cache On Top – All data reads/writes occur through cache‒ Read Through/Write Through

• Cache miss causes load from data source automatically

• Updates to cache written synchronously to the data source

‒ Write Behind

• Updates to cache written asynchronously to the data source


Use casesCachingReduced latency / Backend (DB, Mainframe) downsizing

Reliability / ScalabilityData Grid acts as a transactional system of record, hosting data and business logic

Parallel Computing & EventsParallel queries and processing / Automated processing based on event

InteroperabiltiyData exchange between heterogeneous apps (Java, .Net, C++)

Shared contextsHTTP Session externalization / Data consolidation (profiles)


Join the Coherence Communityhttp://coherence.oracle.com

@OracleCoherence

facebook.com/OracleCoherence

blogs.oracle.com/OracleCoherence

Oracle Coherence Users

youtube.com/OracleCoherence

coherence.oracle.com/display/CSIGCoherence Special Interest Group



Oracle GoldenGate Defined

• Oracle GoldenGate is a real-time, log-based change data capture and replication software platform

• The software provides:- Capture- Routing- Transformation- Delivery

• of transactional data across heterogeneous data sources in real time

No-Impact, Real-Time Data Integration & Transactional Replication


The Need for a Change Data Capture SystemCapturing Changes and Applying them Throughout the Enterprise

New DB/HW/OS/APP

Fully Active Distributed DB

Reporting Database

Data Warehouse

Global Data Centers

Zero Downtime Upgrade & Migration

Query OffloadingMainframe Offloading

Data Synch Across the Enterprise

Real-time BI, Operational Reporting, MDM

Event Driven Architecture, SOA

Highly Available / Disaster Recovery

Log-based,

changed data

Database

Legacy

Message Bus

HW Migration

Mainframe modernization

Mainframe downsizing

Web specialization

Active - Active

Disaster Recovery

Exadata


SourceOracle & Non-Oracle DBs

TargetOracle & Non-Oracle DBs

Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs.

How Oracle GoldenGate Works



Trail: stages and queues data for routing.







Pump: distributes data for routing to target(s).








Route: data is compressed, encrypted for routing to target(s).









Delivery: applies data with transaction integrity, transforming the data as required.









Delivery: applies data with transaction integrity, transforming the data as required.

Bi-directional





UnidirectionalQuery Offloading

Zero-Downtime Migration

Bi-DirectionalHot Standby or

Active-Active for HA

Peer-to-PeerLoad Balancing

Multi-Master

BroadcastData Distribution

Integration/ConsolidationData Warehouse

BPM

BAM

OEP

Data Distributionvia Messaging

Differentiator: FlexibilityOne-to-One, One-to-Many, Many-to-One, Many-to-Many

Java


Oracle GoldenGateDifferentiators

• Zero-impact capture, routing, transformation, and delivery of transactional data

• Sub-second latency for data movement

• Open, modular architecture• Supports heterogeneous sources and targets• One-to-one, one-to-many, many-to-many, many-to-one

• Maintains transactional integrity• Resilient against interruptions and failures• Fast Recovery after process interruptions.

• Conflict resolution in active-active configurations• Filtering, mapping between different data models• Simple transformations• Encryption, compression,…

Performance

Flexible & Extendable

Reliable

Logical replication


Oracle GoldenGate

• Filtering & “Compression”- Unnecessary change data is discarded immediately- Updates and deletes are “compressed” by default- Before images are discarded by default

• Transformations and mappings - Within either the Capture or Delivery modules - no middle tier server is

needed - Built-in functions, PL/SQL, custom functions (user-exits)

• Flexible topology support- Variety of topologies, one-to-one, one-to-many, many-to-one, many-to-

many, and cascading and bidirectional configurations.

Differentiators


Oracle GoldenGate

• Bidirectional configuration support- Oracle GoldenGate enables both active-passive and active-active solutions

for maximum system availability. • Conflict detection and resolution

- Implemented globally, object by object, based on data values and complex filters, or through event-driven criteria.

• Routing and compression - Oracle GoldenGate uses TCP/IP for sending data, so no geographical

distance constraints are imposed.• Security

- Data encryption (trail files, data sent)

Differentiators


Oracle Golden Gate

• Transaction integrity- Transaction boundaries recorded in GoldenGate trail file- Checkpointing and recovery are based on transaction boundaries- Original commit sequence is maintained by the delivery process

• Resiliency & fast recovery- Individual processes can be restarted automatically- Tolerance to network outages (configurable)- Recovery ensures that no operations are skipped or duplicated after failure

of any kind

Differentiators

Copyright © 2013 Oracle and/or its affiliates. All rights reserved.36 36

Oracle GoldenGate Supported PlatformsDatabases O/S and Platforms

Oracle GoldenGate Capture and Delivery: Oracle DB2 Microsoft SQL Server Sybase ASE Teradata Enscribe SQL/MP SQL/MX MySQL v 5.5 JMS message queues

Oracle GoldenGate Delivery: All listed above, plus:TimesTen, Postgres Netezza, Greenplum, HP Neoview ETL products

Linux

Sun Solaris v11

Windows 2000, 2003, XP

HP NonStop

HP-UX

IBM AIX v7.1

IBM z Series

zLinux

IBM i Series


Use Cases

New DB/HW/OS/APP

Fully Active Distributed DB

Reporting Database

Data Warehouse

Global Data Centers

Zero Downtime Upgrade & Migration

Query OffloadingMainframe Offloading

Data Synch Across the Enterprise

Real-time BI, Operational Reporting, MDM

Event Driven Architecture, SOA

Highly Available / Disaster Recovery

Log-based,

changed data

Database

Legacy

Message Bus

HW Migration

Mainframe modernization

Mainframe downsizing

Web specialization

Active - Active

Disaster Recovery

Exadata


Join the GoldenGate Communityhttp://www.oracle.com/technetwork/middleware/goldengate/overview/index.html

twitter.com/ORCLGoldenGate

facebook.com/OracleDataIntegration

blogs.oracle.com/dataintegration

Oracle Data Integration

youtube.com/oraclegoldengate



Coherence HotCacheSolve the Stale Cache Problem

Coherence Application

Coherence Application

3rd Party Application Read / Write

Read / Write

HotCache


Coherence HotCache

• Propagate database changes to cache in real time• Leverage existing technologies

- TopLink Grid- GoldenGate

Solve the Stale Cache Problem

• Event-driven• Low latency• No application

changes


GoldenGate as a Coherence FeedTurn the Database Into an Event Source

Read / Write

HotCache

3rd PartyApplications

Batch Process

Technology

01 coherence & golden gate - technical overview