Oracle GoldenGate – Real-time Access to Real-time Information

Oracle GoldenGate

Real-time Access to Real-time Information

Gavin SoormaSenior Principal Consultant, OnCall DBA

Oracle GoldenGate – Real-time Access to Real-time Information

Introduction

Oracle buys GoldenGate -transaction closed September 3, 2009

A private company, founded in 1995 and based in San Francisco, CA

Recognized by industry analysts as a leader in real-time data integration

Fortune 500 customers across several key industries including Financial Services,Communications, Healthcare, Public Sector, Retail and Utilities

Extends Oracle’s data integration capabilities with the fastest, most scalable realtime Heterogeneous data integration solution

Another Oracle High Availability solution

Oracle Streams not widely accepted by the community – a strategic replacement?

Oracle GoldenGate – Real-time Access to Real-time Information

GoldenGate Oracle Product Positioning

©2009 Oracle Corporation 3

Oracle GoldenGate

Real-Time Data Integration

High Availability Continuous

Operations

Continuous Low Latency Data Movement

Oracle GoldenGate – Real-time Access to Real-time Information

GoldenGate is Strategic to Oracle

Best-in-class leader in real-time data solutions Complements existing Oracle products

Continuous Availability for heterogeneous systems

Real-time data access for Reporting & BI/EPM Zero-downtime migrations / upgrades to Oracle

Database and Applications

• Over 500 customers with 4,000+ implementations across Fortune 500 companies: Financial Services, Communications, Healthcare, Public Sector, Retail & Utilities industries

Top 3 of 5 largest commercial banks Top 3 of 3 busiest ATM networks Top 7 of 10 financial data services companies Top 4 of 5 telecommunications providers Top 3 of 5 largest food & drug stores

Oracle GoldenGate – Real-time Access to Real-time Information

Some Success Stories

• 5 way Active-Active configuration provides same data across worldwide plants

• Zero downtime DB migration to new version using phased approach

• Real-time data integration from Siebel CRM on Oracle to central Teradata Warehouse

• GoldenGate moves 150-200 million records per day with 1.5 second latency.

• Supports 1.6TB of data movement per day to read-only servers

• Source– Oracle 10g | Target – MySQL

Oracle GoldenGate – Real-time Access to Real-time Information6

Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of transactional data across heterogeneous environments in real-time

High Performance

Extensibility &Flexibility

Reliability

Why Oracle GoldenGate?

Non-intrusive, low-impact, sub-second latency

Open, modular architecture - Supports heterogeneous sources and targets

Maintains transactional integrity - Resilient against interruptions and failures

What is Oracle GoldenGate?

Oracle GoldenGate – Real-time Access to Real-time Information

Oracle GoldenGate Heterogeneity

Databases O/S and Platforms

Capture: Oracle DB2 Microsoft SQL Server Sybase ASE Teradata

Delivery: All listed above, plus: MySQL, TimesTen, Netezza, Greenplum, HP Neoview and any ODBC compatible databases ETL products JMS message queues

Windows 2000, 2003, XP

Linux

Sun Solaris

HP NonStop

HP-UX

HP TRU64

HP OpenVMS

IBM AIX

IBM z/OS

Supports Applications running on ….

Oracle GoldenGate – Real-time Access to Real-time Information

Oracle GoldenGate Products

Oracle GoldenGate GoldenGate Management Pack Golden Gate Veridata Available for download on Oracle E-Delivery web site. Available on all major platforms for Oracle versions 10g and 11g

Oracle GoldenGate – Real-time Access to Real-time Information

Oracle GoldenGate Solutions

Enterprise-wide Solution for Real Time Data Needs

Log Based, Real-Time Change Data Capture

Heterogeneous Source Systems

EDWODS

EDW

Standby(Open & Active)

ReportingDatabase

GoldenGate

ETL

ETL

Operational Reporting

Real-time BI

Query Offloading

Data Distribution

Zero Downtime Migration and Upgrades

Disaster Recovery, Data

Protection

Oracle

Microsoft SQL Server

Sybase

Oracle GoldenGate – Real-time Access to Real-time Information

Oracle GoldenGate TopologiesUnidirectional

Query OffloadingBi-Directional

Live Standby orActive-Active for HA

Peer-to-PeerLoad Balancing,

Multi-Master

Broadcast Data Distribution

Integration/ConsolidationData Warehouse

CascadingData Marts

Oracle GoldenGate – Real-time Access to Real-time Information

Integration with Data Warehousing

• Integration via:– Staging tables– Flat files– Messaging

ETL

JMS / MoMQueue or

Topic

datafilesdatafilesdatafiles

Integration via Staging Tables

Integration via Files

Integration via Messaging

Capture Trail File

Delivery

Delivery

Delivery

OLTP

ETL

ETL

Oracle GoldenGate – Real-time Access to Real-time Information

Architecture

Manager process Capture – Extract process Delivery – Replicat process Data Pump Trail files Checkpoints Data definition files Log files, Discard files

Oracle GoldenGate – Real-time Access to Real-time Information

GoldenGate Architecture

Extract

Manager

Data Pump

Collector

Manager

Replicat

Remote TrailLocal Trail

Source Database

Transaction Logs

Target Database

Network (TCP/IP)

Oracle GoldenGate – Real-time Access to Real-time Information

Manager

Manager process runs on source and target One manager per GoldenGate installation Supports multiple Extract and Replicat processes Starts other GoldenGate processes Performs trail file management – purge old extracts Responsible for Event and Error reporting

Oracle GoldenGate – Real-time Access to Real-time Information

Extract

Extract process runs on the source system Capture mechanism of GoldenGate Configure extract for

Initial Data Load Online Change Synchronization

Multiple Extract processes can operate on multiple database objects

Oracle GoldenGate – Real-time Access to Real-time Information

Data Pump

Secondary extract group within source GoldenGate configuration Optional When not used, Extract sends data over network to remote trail on

destination When used, Extract process writes to local trail. Data Pump reads this trail

and writes to the remote trail Data pump isolates the primary extract process from TCP/IP activity Performs data filtering, mapping and conversion Protects against network and target failures

Oracle GoldenGate – Real-time Access to Real-time Information

Replicat

Runs on target system Reads and extracts data changes (and DDL changes when configured) Configured for initial load or online change synchronization. Multiple replicat processes to increase throughput Configured with a apply delay if required

Oracle GoldenGate – Real-time Access to Real-time Information

Trail Files

Series of files stored on disk to enable continuous extraction and propagation of changes

Can exist on the source (local trail), target (remote trail) or on intermediary systems

By default 10 MB in size Files are named at configuration time with a two letter identifier and are

appended with an unique 6 digit serial number GoldenGate writes to trail files in Universal Data Format – a proprietary

format This proprietary format enables the heterogeneous data integration Checkpoints provide known position in trail from where to start in case of

unexpected failure Records of checkpoints maintained in file on disk or database checkpoint

table

Oracle GoldenGate – Real-time Access to Real-time Information

How Oracle GoldenGate Works

LAN / WAN / Internet

Source Oracle Database

TargetDatabaseBi-directional

Trail files: Universal data format enables heterogeneity.

Route: No distance constraints via TCP/IP. Compression & encryption.

Capture: Committed changes are captured (and can be filtered) as they occur by reading the transaction logs.

Delivery: Applies transactional data with guaranteed integrity.

CaptureSource Trail Target Trail

Source TrailTarget Trail

Deliver

Deliver Capture

Redo/archived log

Oracle GoldenGate – Real-time Access to Real-time Information

GoldenGate and Oracle Data Source Support for Oracle RAC Support for ASM Support for index-organized tables

(IOTs) with overflow Support for clustered tables Support for object tables Support for object types (UDTs) Support for DDL operations Archived log only (ALO) mode Off-platform capture (LOGSOURCE) Multi-threaded capture

Capture

Capture Commit Ordered Trail

Archived Log Online Log

OCI/Oracle Net

Oracle GoldenGate – Real-time Access to Real-time Information

Capture

Capture Commit Ordered Trail

Backups Transaction Log

ODBC

GoldenGate and Microsoft SQL Server

Microsoft SQL Server Capture

Ability to capture from native

backups Coexistence with SQL Server

Replication SQL Server Replication components

are not required Support tables with no unique key

(not supported by SQL Server Replication)

Support for computed columns Support for identity columns

Oracle GoldenGate – Real-time Access to Real-time Information

Zero-Downtime Upgrades, Migrations and Maintenance

Eliminate “planned downtime” during:

– Application Upgrades (validated for Siebel CRM)

– Database Upgrades: Oracle 8i or 9i 11g

– Database Migrations (e.g. DB2 to Oracle)

– Database Maintenance (e.g. de-fragmentation, table re-orgs, de-chaining rows, etc.)

– Database Server Upgrades

Oracle GoldenGate – Real-time Access to Real-time Information

Oracle 8i or 9i to 10/11g Upgrades/Migrations

• What GoldenGate TDM offers:

– Minimal impact and continuous availability during the project

– Technology solution for eliminating database downtime

– Improved capabilities for data integrity, accuracy

– Fail-back solution in the event of unexpected issues/downtime

– Proven – with customers in production

Oracle GoldenGate – Real-time Access to Real-time Information

Cross Platform Migration

9i Solaris

1. Start GoldenGate TDM’s Capture process

2. Set up Clone database, then Upgrade to 10g

3. Cross platform transportable tablespaces metadata export

4. Use a full database NOROWS export (Views, Packages, etc)

1

9i Clone 10g

Solaris23,4

Oracle GoldenGate – Real-time Access to Real-time Information

Cross Platform Migration

9i Solaris

9i Clone 10g

Solaris

10g

Linux

1. Start GoldenGate TDM’s Capture process

2. Set up Clone database, then Upgrade to 10g

3. Cross platform transportable tablespaces metadata export

4. Use a full database NOROWS export (Views, Packages, etc)

5. Set up a new 10g vanilla target

1

23,4

5

6,7

6. Cross platform transportable tablespaces metadata import

7. Full import with IGNORE option

Oracle GoldenGate – Real-time Access to Real-time Information

Cross Platform Migration

9i Solaris

9i Clone 10g

Solaris

10g

Linux

1. Start GoldenGate TDM’s Capture process

2. Set up Clone database, then Upgrade to 10g

3. Cross platform transportable tablespaces metadata export

4. Use a full database NOROWS export (Views, Packages, etc)

5. Set up a new 10g vanilla target

1

2

5

8

6. Cross platform transportable tablespaces metadata import

7. Full import with IGNORE option8. Start GoldenGate TDM Apply process

at target 9. **SWITCHOVER** (not depicted)

3,4

6,7

Oracle GoldenGate – Real-time Access to Real-time Information

Cross Platform Migration

9i Solaris

10g

Linux

1. Start GoldenGate TDM’s Capture process

2. Set up Clone database, then Upgrade to 10g

3. Cross platform transportable tablespaces metadata export

4. Use a full database NOROWS export (Views, Packages, etc)

5. Set up a new 10g vanilla target

1

5

8

6. Cross platform transportable tablespaces metadata import

7. Full import with IGNORE option8. Start GoldenGate TDM Apply process

at target9. Verify data (optional)10. **SWITCHOVER** (not depicted)11. Start TDM Capture at target (new

source)12. Start TDM Apply at target (old source)

10

11

9i Clone 10g

Solaris23,4

6,7

Failback contingency

Oracle GoldenGate – Real-time Access to Real-time Information

GoldenGate Management Pack

Graphical interface for designing, configuring, managing,

monitoring, and reporting on the Oracle GoldenGate components.

Golden Gate Director Server GoldenGate Director Client

GoldenGate Web Director

Oracle GoldenGate – Real-time Access to Real-time Information

GoldenGate Veridata™

“GoldenGate Veridata is a high-speed, low impact data comparison solution that identifies and reports data discrepancies between two databases, without interrupting those systems or the business processes they support.”

High-speed data comparison solution Identifies and reports on data discrepancies between

heterogeneous databases with minimal impact on infrastructure Companies can audit and verify large volumes of data across

different business applications with certainty It enables users to research root causes of the discrepancies and

determine how to remedy the problem and avoid future risk.

Oracle GoldenGate – Real-time Access to Real-time Information

GoldenGate Veridata™

Oracle GoldenGate – Real-time Access to Real-time Information

GoldenGate Veridata: How it Works

• The user chooses tables or files on the source and target databases• The comparison is initiated from the GUI, command line or batch• As the databases continue to change, GoldenGate Veridata reports:

– Persistent discrepancies – In-flight data discrepancies (user configurable)

Oracle GoldenGate – Real-time Access to Real-time Information

GoldenGate Director

Oracle GoldenGate – Real-time Access to Real-time Information

GoldenGate Director

Oracle GoldenGate – Real-time Access to Real-time Information

GoldenGate Director Web

Oracle GoldenGate – Real-time Access to Real-time Information

GoldenGate Demo

Oracle GoldenGate – Real-time Access to Real-time Information

Create the Manager, Extract and Replicat Process

./ggsci

GGSCI (linux01.oncalldba.com) 1> edit params mgr

PORT 7809USERID ggs_owner, PASSWORD ggs_owner

Source – Extract Process

GGSCI (linux01.oncalldba.com) 1> ADD EXTRACT ext1, TRANLOG, BEGIN NOWEXTRACT added.

GGSCI (linux01.oncalldba.com) 2> ADD RMTTRAIL /home/oracle/goldengate/dirdat/rt, EXTRACT ext1

RMTTRAIL added.

Target – Replicat Process

GGSCI (linux03.oncalldba.com) 1> ADD REPLICAT rep1, EXTTRAIL /home/oracle/goldengate/dirdat/rt

REPLICAT added.

Oracle GoldenGate – Real-time Access to Real-time Information

Start Manager (Source)

[oracle@linux01 goldengate]$ ./ggsci

Oracle GoldenGate Command Interpreter for OracleVersion 10.4.0.19 Build 002Linux, x86, 32bit (optimized), Oracle 10 on Sep 17 2009 23:49:42

Copyright (C) 1995, 2009, Oracle and/or its affiliates. All rights reserved.

GGSCI (linux01.oncalldba.com) 1> start manager

Manager started.

GGSCI (linux01.oncalldba.com) 2> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING

Oracle GoldenGate – Real-time Access to Real-time Information

Start Manager (Target)

[oracle@linux03 goldengate]$ ./ggsci

Oracle GoldenGate Command Interpreter for OracleVersion 10.4.0.19 Build 002Linux, x86, 32bit (optimized), Oracle 11 on Sep 29 2009 08:50:50

Copyright (C) 1995, 2009, Oracle and/or its affiliates. All rights reserved.

GGSCI (linux03.oncalldba.com) 1> start manager

Manager started.

GGSCI (linux03.oncalldba.com) 2> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING

Oracle GoldenGate – Real-time Access to Real-time Information

Source – Start Extract Process

GGSCI (linux01.oncalldba.com) 4> view params ext1

EXTRACT ext1USERID ggs_owner@oncalldb.world,password ggs_ownerRMTHOST linux03, MGRPORT 7809RMTTRAIL /home/oracle/goldengate/dirdat/rtDDL INCLUDE ALLTABLE ggs_owner.myobjects;;

GGSCI (linux01.oncalldba.com) 5> start extract ext1

Sending START request to MANAGER ...EXTRACT EXT1 starting

GGSCI (linux01.oncalldba.com) 6> info extract ext1

EXTRACT EXT1 Last Started 2010-08-13 13:12 Status RUNNINGCheckpoint Lag 03:00:32 (updated 00:00:00 ago)Log Read Checkpoint Oracle Redo Logs 2010-08-13 10:12:23 Seqno 4, RBA 19823616

Oracle GoldenGate – Real-time Access to Real-time Information

Target – Start Replicat Process

GGSCI (linux03.oncalldba.com) 3> view params rep1

REPLICAT rep1ASSUMETARGETDEFSUSERID ggs_owner, PASSWORD ggs_ownerdiscardfile /home/oracle/goldengate/dirout/discard.txt,appendDDL INCLUDE MAPPEDMAP ggs_owner.myobjects, TARGET ggs_owner.myobjects;

GGSCI (linux03.oncalldba.com) 4> start replicat rep1

Sending START request to MANAGER ...REPLICAT REP1 starting

GGSCI (linux03.oncalldba.com) 5> info replicat rep1

REPLICAT REP1 Last Started 2010-08-13 13:15 Status RUNNINGCheckpoint Lag 00:00:00 (updated 00:00:04 ago)Log Read Checkpoint File /home/oracle/goldengate/dirdat/rt000010 2010-08-13 13:12:55.105247 RBA 1030

Oracle GoldenGate – Real-time Access to Real-time Information

Test DDL Replication

[oracle@linux01 ~]$ sqlplus ggs_owner/ggs_owner

SQL*Plus: Release 10.2.0.4.0 - Production on Fri Aug 13 13:09:38 2010

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Connected to:Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> create table myobjects as select * from dba_objects where 1=2;

Table created.

SQL> alter table myobjects add constraint pk_myobjects 2 primary key (object_id);

Table altered.

Create a table and index on source database

Oracle GoldenGate – Real-time Access to Real-time Information

Test DDL Replication

SQL> desc myobjects Name Null? Type ----------------------------------------- -------- ---------------------------- OWNER VARCHAR2(30) OBJECT_NAME VARCHAR2(128) SUBOBJECT_NAME VARCHAR2(30) OBJECT_ID NOT NULL NUMBER DATA_OBJECT_ID NUMBER OBJECT_TYPE VARCHAR2(19) CREATED DATE LAST_DDL_TIME DATE TIMESTAMP VARCHAR2(19) STATUS VARCHAR2(7) TEMPORARY VARCHAR2(1) SQL> select index_name from user_indexes where table_name='MYOBJECTS';

INDEX_NAME------------------------------PK_MYOBJECTS

On Target database MYOBJECTS table and index created

Oracle GoldenGate – Real-time Access to Real-time Information

Test Online Change Synchronization

SQL> !dateFri Aug 13 13:24:14 WST 2010

SQL> insert into myobjects select * from dba_objects where owner='SYSMAN';

1341 rows created.

SQL> commit;

Commit complete.SQL> !dateFri Aug 13 13:27:01 WST 2010

SQL> insert into myobjects select * from dba_objects where owner='SYS';

22937 rows created.

SQL> commit;

Commit complete.

INSERT rows on source tables

Oracle GoldenGate – Real-time Access to Real-time Information

Test Online Change Synchronization

SQL> !dateFri Aug 13 13:24:44 WST 2010

SQL> select count(*) from myobjects;

COUNT(*)---------- 1341

SQL> !dateFri Aug 13 13:27:14 WST 2010

SQL> select count(*) from myobjects;

COUNT(*)---------- 24278

Check target tables to see that all the rows have been inserted

Oracle GoldenGate – Real-time Access to Real-time Information

Monitor Replicat Progress

GGSCI (linux03.oncalldba.com) 8> stats replicat rep1

Start of Statistics at 2010-08-13 13:35:39.

DDL replication statistics:

*** Total statistics since replicat started *** Operations 2.00 Mapped operations 2.00 Unmapped operations 0.00 Other operations 0.00 Excluded operations 0.00 Errors 0.00 Retried errors 0.00 Discarded errors 0.00 Ignored errors 0.00

Replicating from GGS_OWNER.MYOBJECTS to GGS_OWNER.MYOBJECTS:

*** Total statistics since 2010-08-13 13:24:33 *** Total inserts 24278.00 Total updates 0.00 Total deletes 0.00 Total discards 0.00 Total operations 24278.00

Oracle GoldenGate – Real-time Access to Real-time Information

Cleanup – some more DDL ReplicationSOURCE

SQL> truncate table myobjects;

Table truncated.

TARGET

SQL> select count(*) from myobjects;

COUNT(*)---------- 0

SOURCE

SQL> drop table myobjects;

Table dropped.

TARGET

SQL> select count(*) from myobjects;select count(*) from myobjects *ERROR at line 1:ORA-00942: table or view does not exist

Oracle GoldenGate – Real-time Access to Real-time Information

QA&gavin.soorma@oncalldba.com.au

http://www.gavinsoorma.comhttp://www.oncalldba.com.au