Oracle GoldenGate Performance Tuning and …...• Note 1448324.1 OGG Integrated Healthcheck Script • Note 1488668.1 GoldenGate Performance Data Gathering • Note 1596493.1 GoldenGate

Oracle GoldenGate Performance Tuning and Monitoring Ease of Manageability

Volker Kuhr Senior Principal Product Manager Jiahui Shao Product Manager

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |


Oracle GoldenGate and Data Integration

Energy/Industrial Insurance / Health Retail Other

Communications Finance / Banking Media Services Over 10K Customers Worldwide

http://www.telecomitalia.it/cgi-bin/tiportal/TIPortal/ep/home.do?BV_UseBVCookie=Yes&tabId=0�

http://www.ibasis.net/index.htm�

http://www.diebold.com/default.htm�

http://www.fitchratings.com/index.cfm�

http://www.abp.nl/abp/abp/�

http://www.inttra.com/home/Home.aspx�

http://www.astrazeneca.com/ncm.aspx�

http://www.dhl-usa.com/home/Home.asp�

http://www.genzyme.com/default.asp�

http://www.alticor.com/�

http://www.snapon.com/�

http://www.jcrew.com/home.jhtml�

http://www.schneider.com/index.htm�

http://images.google.com/imgres?imgurl=http://hrff.jalmc.org/_Images/News/adp.jpg&imgrefurl=http://www.ortop.org/sd.htm&h=671&w=1500&sz=74&hl=en&start=1&um=1&tbnid=lSxuKOYqXblBlM:&tbnh=67&tbnw=150&prev=/images?q=adp&imgsz=xxlarge&svnum=10&um=1&hl=en&rlz=1T4GZHY_enUS230US230&sa=N�

http://www.google.com/url?sa=i&source=images&cd=&cad=rja&docid=bvvK6u1i7NekpM&tbnid=1KzWCQTNm7Kc2M:&ved=0CAgQjRwwAA&url=http://androidspin.com/2013/05/13/bt-returning-to-mobile-after-10-years/&ei=vSHLUcDoF4eTiALL1IDYCQ&psig=AFQjCNHK9hoF_BuecfbDKFvcNUQikuE_rw&ust=1372353341432974�

http://www.google.com/imgres?imgurl=http://4.bp.blogspot.com/_c4CPvm_1jP8/TIUKT6eFFAI/AAAAAAAAAB8/QMaZ4iSbYIk/s1600/Griffith+Uni+Logo.gif&imgrefurl=http://www.shababaustralia.org/griffith-university-griffith/&h=344&w=1277&sz=19&tbnid=CdgpCOwwYPCtEM:&tbnh=31&tbnw=114&prev=/search?q=griffith+university+logo+australia&tbm=isch&tbo=u&zoom=1&q=griffith+university+logo+australia&usg=__DgEj_tO1xyZs8IO_283BycchPzo=&hl=en&sa=X&ei=vzlXUIuxGIaS2AXFrYGYCQ&ved=0CCwQ9QEwCA�

http://www.google.com/url?sa=i&source=images&cd=&cad=rja&docid=I2NcgUJPHhciuM&tbnid=RZkeyq7ImB5shM:&ved=0CAgQjRwwAA&url=http://www.americancasinoguide.com/missouri/ameristar-casino-hotel-kansas-city.html&ei=jSLLUfjaB6OpiQLar4GoAQ&psig=AFQjCNHtcK24kM_BUrBK6e9EAkWG6TnSSg&ust=1372353549165343�

http://www.suddenlink.com/�

http://www.suddenlink.com/�

http://www.google.com/imgres?imgurl=http://deniseleeyohn.com/bites/wp-content/uploads/2008/09/saks_logo2.jpg&imgrefurl=http://deniseleeyohn.com/bites/2008/09/08/saks-gets-it-bloomies-not-so-much/&docid=vvVSXoR6m27yvM&tbnid=goBM_0uxsQY1yM&w=200&h=200&ei=eCPLUbz8C8eRiALMkoC4Dw&ved=0CAQQxiAwAg&iact=c�

http://www.google.com/imgres?imgurl=http://upload.wikimedia.org/wikipedia/en/f/ff/Columbus_State_University_logo.jpg&imgrefurl=http://en.wikipedia.org/wiki/File:Columbus_State_University_logo.jpg&docid=-3j4PNj6gPRaSM&tbnid=wCMuZv1S7439mM:&w=286&h=286&ei=zSPLUaa5CaWdiQL084CYCQ&ved=0CAIQxiAwAA&iact=c�


Oracle GoldenGate Architecture

Bi-Directional

Capture Trail Files Network Firewall Cloud

Trail Files Delivery Database

Database

3


Program Agenda

Automatic Heartbeat Functionality

Oracle GoldenGate Diagnosis

1

2

4


Program Agenda

Automatic Heartbeat Functionality


1

2

5


New 12.2 Feature - Automatic Heartbeats

6

Built-in Functionality to Monitor End-to-End Replication Lag

Simple Automatic • Enable via a single “ADD

HEARTTBEATTABLE” command at each site

• Automatically discovers replication flows for all topologies

–Unidirectional –Bidirectional –N-way

Real-time • Database views to query

end-to-end real-time latency

• Lag information on incoming flow to and outgoing flow from a database


Simple Bidirectional Replication Example • Enable Heartbeat functionality by executing GGSCI command ‘ADD

HEARTBEATTABLE’ at each database – Creates required heartbeat tables, views and jobs – Updates heartbeat every 60s by default

7

ExtA

ExtB

PmpAB RepAB

RepBA PmpBA

In Database A: GG_LAG View contents

LOCAL_ DATABASE

REMOTE_ DATABASE INCOMING_PATH INCOMING_

LAG

INCOMING_ HEARTBEAT_

AGE OUTGOING_PATH OUTGOING_

LAG

OUTGOING_ HEARTBEAT_

AGE

A B ExtB==>PmpBA==>RepBA 1.066182 14.150614 ExtA==>PmpAB==>RepAB 1.391718 25.387458

Database A Database B


Monitoring Lag using GG_LAG View

8

Column Name Data type Description

LOCAL_DATABASE VARCHAR2(30) Local database name

CURRENT_LOCAL_TS TIMESTAMP(6) Current timestamp in UTC time zone

REMOTE_DATABASE VARCHAR2(30) Remote database name

INCOMING_PATH VARCHAR2(4000) Group names on the incoming flow

INCOMING_LAG NUMBER Period of time between remote database generating heartbeat and local database receiving heartbeat

INCOMING_HEARTBEAT_AGE NUMBER CURRENT_LOCAL_TS minus most recent heartbeat timestamp of remote database

OUTGOING_PATH VARCHAR2(4000) Group names on the outgoing flow

OUTGOING_LAG NUMBER Period of time between local database generating heartbeat and remote database receiving heartbeat

OUTGOING_HEARTBEAT_AGE NUMBER CURRENT_LOCAL_TS minus recent heartbeat timestamp of local database


• GG_LAG_HISTORY View – Historical heartbeat lag

• GG_HEARTBEAT and GG_HEARTBEAT_HISTORY Tables – Underlying tables for the views – Get lag for each process on the path

9

Additional Tables and Views

Column Name

LOCAL_DATABASE

CURRENT_LOCAL_TS

REMOTE_DATABASE

INCOMING_PATH

INCOMING_LAG

INCOMING_HEARTBEAT_AGE

OUTGOING_PATH

OUTGOING_LAG

OUTGOING_HEARTBEAT_AGE

Column Name

INCOMING_HEARTBEAT_TS

INCOMING_EXTRACT_TS

INCOMING_ROUTING_TS

INCOMING_REPLICAT_TS

OUTGOING_HEARTBEAT_TS

OUTGOING_EXTRACT_TS

OUTGOING_ROUTING_TS

OUTGOING_REPLICAT_TS

…

} Extract lag } Pump lag } Replicat lag


Discovers New Replication Flows Automatically

• Simply execute ADD HEARTBEATTABLE for new site

10

Example: Add new site Database C

In Database A: GG_LAG View contents LOCAL_

DATABASE REMOTE_ DATABASE INCOMING_PATH OUTGOING_PATH

A B ExtB==>PmpBA==>RepBA ExtA==>PmpAB==>RepAB

A C ExtC==>PmpCA==>RepCA ExtA==>PmpAC==>RepAC

Database A Database B

Database C (Newly added site)

ExtA==>PmpAB==>RepAB

RepBA<==PmpBA<==ExtB


Diagnosing Replication Issues using Heartbeat Table

11

Diagnosis Cause

INCOMING_LAG is high

OUTGOING_HEARTBEAT_AGE is high

INCOMING_HEARTBEAT_AGE is high

OUTGOING_LAG is high

Action • Replication processes on the incoming path have stopped • Network issues • Heartbeat scheduler job has stopped

• Check processes on incoming path and the remote heartbeat scheduler job

• Replication processing is slow on the incoming path

• Use GG_HEARTBEAT table to determine slow process on the incoming path

• Replication processes on the outgoing path have stopped • Network issues • Heartbeat scheduler job has stopped

•Check processes on outgoing path and the local heartbeat scheduler job

• Replication processing is slow on the outgoing path

•Use GG_HEARTBEAT table to determine slow process on the outgoing path


Program Agenda

Heartbeat Functionality


• GoldenGate Extended Metrics

• Integrated Healthcheck

1

2

12


Potential Sources of Lag • System Resources

– Memory – CPU – IO and Network

• Configuration – Database – GoldenGate

13


Tools for Performance Analysis

LogMining Server

Extract Pump

Replicat Inbound Server

• HealthCheck • UTL_SPADV • AWR & ASH

• HealthCheck • UTL_SPADV • AWR & ASH

GoldenGate Extended Metrics

OS Utilities (netstat)

OS Utilities (top, vmstat, iostat)

OS Utilities (top, vmstat, iostat)

14


New GoldenGate Extended Metrics

• Access to Monitoring Point through Restful Web Services http://<hostname>:<mgr_port>/mpointsx

• Real-time insight into GoldenGate client programs

• Requires “ENABLEMONITORING” in GLOBALS

15

Fine-grained Performance Monitoring


New Extended Metrics • Status and Configuration Information • Process Level Metrics for Extract, Pump and Replicat • Thread Level Metrics for Extract and Replicat • Database Statistics for Extract and Replicat

– In-flight-transactions and Cache- & Queue statistics for Extract – Table statistics for Replicat

• Network Statistics

16


17

https://java.net/projects/oracledi/downloads/download/GoldenGate/OGGPTK.jar Graphical real-time instance monitoring

Replication Components

Process Status Change

Monitoring Metrics

Recording Function

https://java.net/projects/oracledi/downloads/download/GoldenGate/OGGPTK.jar�


Integrated Healthcheck Report

• Comprehensive snapshot of the database components of Integrated Extract and Apply.

• Best Practices: Run periodically for differential analysis • Used for diagnostic purposes

– Configuration Issues – Performance

• Analyze at component and sub-component level

Snapshot of all GoldenGate related Information within the Database

18


Healthcheck Script

1. Database configuration

3. Extract/Replicat Configuration

4. Extract/Replicat Statistics

2. Summary and Notification

19


Notification Section

• Pre-Analysis with INFO, WARNING & ERROR Messages

• Heartbeat Table

20


Record changes

Capture changes

Redo

Integrated Extract

Oracle Database

LCR1 LCR2 LCR3 …. …. ….

Logical Change Records

Extract Trail Files

Lag within the Integrated Extract

21


GoldenGate Logmining Server Architecture

Extract

• Requests LCRs from logmining server

• Performs Mapping and Transformations

• Writes Trail File

Logmining Server (Database Capture Process)

• Reader: Reads logfile and splits into regions

• Preparer: Scans regions of logfiles and prefilters based on extract parameters

• Builder: Merges prepared records in SCN order

• Capture: Formats Logical Change Records(LCRs) and passes to GoldenGate Extract

Reader Capture Builder

Logmining Server

Preparer N

Preparer 2

Preparer 1

Oracle Database

Integrated Extract

Trail Files

22


Monitoring Integrated Extract

PATH 2 RUN_ID 15 RUN_TIME 2014-FEB-21 22:32:17 CCA N <C> OGG$CAP_EXT_1A 305260 305245 790813 LMR 0.0% 53.3% 33.3% "CPU + Wait for CPU" proc %idle %flowctrl %TopWait “Event” LMP(2) 0.0% 0.0% 146.7% "CPU + Wait for CPU" LMB 73.3% 0.0% 6.7% " " CAP 20.0% 0.0% 73.3% "CPU + Wait for CPU" <Q> "OGGADMIN"."OGG$Q_EXT_1A" 0.01 0.01 0 <E> EXT_1A 304027 80076708 790813 20% 6.7% 73.3% "CPU + Wait for CPU" <B> NO BOTTLENECK IDENTIFIED

• % of Idle, Flow Control and major Wait Event for each sub component • Check output for multiple occurrences over time • Increase Parallelism of Capture – if CPU is available

(LMR in flow control, and LMP is 0% idle with high CPU)

Streams Performance Advisor (UTL_SPADV)

23


• State and "Ready to send Chunks" as one Indicator for an Extract program issue

• If "Ready to send Chunks" >> " Delivered Chunks" and Capture State indicates " Waiting for Client Requests", Extract program needs to checked.

• Check Extended Metrics and TRANLOGOPTIONS • BUFFER_SIZE , _READAHEADCOUNT, ASYNCTRANSPROCESSING …

Lag between Database and Extract client

24


Integrated Replicat Replicat: Apply section

Trail Files

Replicat Lightweight Streaming

API Receiver Preparer Coordinator

Inbound Server

Applier N

Applier 2

Applier 1

Oracle Database

• Waiting for memory might indicate Memory issues Replicat IntegratedParams MAX_SGA_SIZE

25


• Total LCRs with Dependencies indicates transaction dependencies • One reason for Total LCRs with WM Dependency might be a large transaction

Integrated Replicat Transaction Dependencies

… … … … … … … … … …

20000

• Increasing EAGER_SIZE from Default to a moderately higher number

20000

26


Apply Server Statistics – Wait Event Analysis

• Wait Event Analysis of Apply Server Processes – “Classical Waits”: Enqueues, IO Waits, … – Replication related Wait Events

• Check IO Subsystem, Classic Database Tuning (e.g.: INITRANS, Index Rebuild, …)

Apply Server Processes as common Database sessions

(wait on CPU /on CPU)

(wait on CPU /on CPU)

REPL Apply: dependency

27


Additional Database Resources AWR & UTL_SPADV • . • UTL_SPADV Package to monitor database background

processes for Extract and Replicat – Two ways of displaying information: single line or HTML – Start and Stop monitoring – Shows bottleneck based on database wait events for the interval

28


Key Performance Tuning Parameters • Database

– STREAMS_POOL_SIZE

• Integrated Extract – TRANLOGOPTIONS

• BUFSIZE • ASYNCTRANSPROCESSING • _READAHEADCOUNT

– TRANLOGOPTIONS INTEGRATEDPARAMS (…)

• MAX_SGA_SIZE • PARALLELISM

29 •

• Integrated REPLICAT – DBOPTIONS

• BATCHSQL • GROUPTRANSOPS (if PARALLELISM=1)

– DBOPTIONS INTEGRATEDPARAMS (…)

• MAX_SGA_SIZE • PARALLELISM • MAX_PARALLELISM • COMMIT_SERIALIZATION • EAGER_SIZE


Summary • Monitoring:

– Use Automatic Heartbeat Table – Run periodically Healthcheck Reports

• Analysis – Use Heartbeat Table to drill down to the component – Monitor OGG extended Metrics – Healthcheck Reports – UTL_SPADV – Check Database Performance (AWR & ASH) and OS Performance

30 •


References

• MAA white paper Oracle GoldenGate Performance Best Practices http://www.oracle.com/technetwork/database/availability/maa-gg-performance-

1969630.pdf

• Note 1557031.1 Recommended patches for Integrated Capture/Replicat • Note 1448324.1 OGG Integrated Healthcheck Script • Note 1488668.1 GoldenGate Performance Data Gathering • Note 1596493.1 GoldenGate Integration with AWR • Note 1485620.1 Best Practices Downstream Capture

MOS Notes

31

http://www.oracle.com/technetwork/database/availability/maa-gg-performance-1969630.pdf�

http://www.oracle.com/technetwork/database/availability/maa-gg-performance-1969630.pdf�

Documents

Oracle GoldenGate Performance Tuning and …...• Note 1448324.1 OGG Integrated Healthcheck Script • Note 1488668.1 GoldenGate Performance Data Gathering • Note 1596493.1 GoldenGate