Oracle GoldenGate Performance Tuning and Monitoring€¦ · · 2016-06-03Title: How to Use the PowerPoint Template Author: The Presentation Company Created Date: 12/17/2015 12:14:00

Oracle GoldenGate Performance Tuning and Monitoring Ease of Manageability

Volker Kuhr Senior Principal Product Manager Jiahui Shao Product Manager

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

2


Oracle GoldenGate and Data Integration

Energy/Industrial Insurance / Health Retail Other

Communications Finance / Banking Media Services

Over 10K Customers Worldwide

http://www.telecomitalia.it/cgi-bin/tiportal/TIPortal/ep/home.do?BV_UseBVCookie=Yes&tabId=0

http://www.ibasis.net/index.htm

http://www.diebold.com/default.htm

http://www.fitchratings.com/index.cfm

http://www.abp.nl/abp/abp/

http://www.inttra.com/home/Home.aspx

http://www.astrazeneca.com/ncm.aspx

http://www.dhl-usa.com/home/Home.asp

http://www.genzyme.com/default.asp

http://www.alticor.com/

http://www.snapon.com/

http://www.jcrew.com/home.jhtml

http://www.schneider.com/index.htm

http://images.google.com/imgres?imgurl=http://hrff.jalmc.org/_Images/News/adp.jpg&imgrefurl=http://www.ortop.org/sd.htm&h=671&w=1500&sz=74&hl=en&start=1&um=1&tbnid=lSxuKOYqXblBlM:&tbnh=67&tbnw=150&prev=/images?q=adp&imgsz=xxlarge&svnum=10&um=1&hl=en&rlz=1T4GZHY_enUS230US230&sa=N

http://www.google.com/url?sa=i&source=images&cd=&cad=rja&docid=bvvK6u1i7NekpM&tbnid=1KzWCQTNm7Kc2M:&ved=0CAgQjRwwAA&url=http://androidspin.com/2013/05/13/bt-returning-to-mobile-after-10-years/&ei=vSHLUcDoF4eTiALL1IDYCQ&psig=AFQjCNHK9hoF_BuecfbDKFvcNUQikuE_rw&ust=1372353341432974

http://www.google.com/imgres?imgurl=http://4.bp.blogspot.com/_c4CPvm_1jP8/TIUKT6eFFAI/AAAAAAAAAB8/QMaZ4iSbYIk/s1600/Griffith+Uni+Logo.gif&imgrefurl=http://www.shababaustralia.org/griffith-university-griffith/&h=344&w=1277&sz=19&tbnid=CdgpCOwwYPCtEM:&tbnh=31&tbnw=114&prev=/search?q=griffith+university+logo+australia&tbm=isch&tbo=u&zoom=1&q=griffith+university+logo+australia&usg=__DgEj_tO1xyZs8IO_283BycchPzo=&hl=en&sa=X&ei=vzlXUIuxGIaS2AXFrYGYCQ&ved=0CCwQ9QEwCA

http://www.google.com/url?sa=i&source=images&cd=&cad=rja&docid=I2NcgUJPHhciuM&tbnid=RZkeyq7ImB5shM:&ved=0CAgQjRwwAA&url=http://www.americancasinoguide.com/missouri/ameristar-casino-hotel-kansas-city.html&ei=jSLLUfjaB6OpiQLar4GoAQ&psig=AFQjCNHtcK24kM_BUrBK6e9EAkWG6TnSSg&ust=1372353549165343

http://www.suddenlink.com/

http://www.suddenlink.com/

http://www.google.com/imgres?imgurl=http://deniseleeyohn.com/bites/wp-content/uploads/2008/09/saks_logo2.jpg&imgrefurl=http://deniseleeyohn.com/bites/2008/09/08/saks-gets-it-bloomies-not-so-much/&docid=vvVSXoR6m27yvM&tbnid=goBM_0uxsQY1yM&w=200&h=200&ei=eCPLUbz8C8eRiALMkoC4Dw&ved=0CAQQxiAwAg&iact=c

http://www.google.com/imgres?imgurl=http://upload.wikimedia.org/wikipedia/en/f/ff/Columbus_State_University_logo.jpg&imgrefurl=http://en.wikipedia.org/wiki/File:Columbus_State_University_logo.jpg&docid=-3j4PNj6gPRaSM&tbnid=wCMuZv1S7439mM:&w=286&h=286&ei=zSPLUaa5CaWdiQL084CYCQ&ved=0CAIQxiAwAA&iact=c


Oracle GoldenGate Architecture

4

Network

Firewall

Cloud

Capture Trail Files Trail Files Delivery

Source

Oracle / Non-Oracle

Database

Bi-directional

Target

Oracle / Non-Oracle

Database


Program Agenda

Automatic Heartbeat Functionality

Oracle GoldenGate Diagnosis

1

2

5


New 12.2 Feature - Automatic Heartbeats

6

Built-in Functionality to Monitor End-to-End Replication Lag

Simple Automatic

• Enable via a single “ADD HEARTTBEATTABLE” command at each site

•Automatically discovers replication flows for all topologies

–Unidirectional

–Bidirectional

–N-way

Real-time

•Database views to query end-to-end real-time latency

• Lag information on incoming flow to and outgoing flow from a database


Simple Bidirectional Replication Example

• Enable Heartbeat functionality by executing GGSCI command ‘ADD HEARTBEATTABLE’ at each database

– Creates required heartbeat tables, views and jobs

– Updates heartbeat every 60s by default

7

ExtA

ExtB

PmpAB RepAB

RepBA PmpBA

In Database A: GG_LAG View contents

LOCAL_ DATABASE

REMOTE_ DATABASE

INCOMING_PATH INCOMING_

LAG

INCOMING_ HEARTBEAT_

AGE OUTGOING_PATH

OUTGOING_LAG

OUTGOING_ HEARTBEAT_

AGE

A B ExtB==>PmpBA==>RepBA 1.066182 14.150614 ExtA==>PmpAB==>RepAB 1.391718 25.387458

Database A Database B


Monitoring Lag using GG_LAG View

8

Column Name Data type Description

LOCAL_DATABASE VARCHAR2(30) Local database name

CURRENT_LOCAL_TS TIMESTAMP(6) Current timestamp in UTC time zone

REMOTE_DATABASE VARCHAR2(30) Remote database name

INCOMING_PATH VARCHAR2(4000) Group names on the incoming flow

INCOMING_LAG NUMBER Period of time between remote database generating heartbeat and local database receiving heartbeat

INCOMING_HEARTBEAT_AGE NUMBER CURRENT_LOCAL_TS minus most recent heartbeat timestamp of remote database

OUTGOING_PATH VARCHAR2(4000) Group names on the outgoing flow

OUTGOING_LAG NUMBER Period of time between local database generating heartbeat and remote database receiving heartbeat

OUTGOING_HEARTBEAT_AGE NUMBER CURRENT_LOCAL_TS minus recent heartbeat timestamp of local database


• GG_LAG_HISTORY View

– Historical heartbeat lag

• GG_HEARTBEAT and GG_HEARTBEAT_HISTORY Tables

– Underlying tables for the views

– Get lag for each process on the path

9

Additional Tables and Views

Column Name

LOCAL_DATABASE

CURRENT_LOCAL_TS

REMOTE_DATABASE

INCOMING_PATH

INCOMING_LAG

INCOMING_HEARTBEAT_AGE

OUTGOING_PATH

OUTGOING_LAG

OUTGOING_HEARTBEAT_AGE

Column Name

INCOMING_HEARTBEAT_TS

INCOMING_EXTRACT_TS

INCOMING_ROUTING_TS

INCOMING_REPLICAT_TS

OUTGOING_HEARTBEAT_TS

OUTGOING_EXTRACT_TS

OUTGOING_ROUTING_TS

OUTGOING_REPLICAT_TS

…

} Extract lag

} Pump lag

} Replicat lag


Discovers New Replication Flows Automatically

• Simply execute ADD HEARTBEATTABLE for new site

10

Example: Add new site Database C

In Database A: GG_LAG View contents

LOCAL_ DATABASE

REMOTE_ DATABASE

INCOMING_PATH OUTGOING_PATH

A B ExtB==>PmpBA==>RepBA ExtA==>PmpAB==>RepAB

A C ExtC==>PmpCA==>RepCA ExtA==>PmpAC==>RepAC

Database A Database B

Database C (Newly added site)

ExtA==>PmpAB==>RepAB

RepBA<==PmpBA<==ExtB


Diagnosing Replication Issues using Heartbeat Table

11

Diagnosis Cause

INCOMING_LAG is high

OUTGOING_HEARTBEAT_AGE is high

INCOMING_HEARTBEAT_AGE is high

OUTGOING_LAG is high

Action • Replication processes on the incoming path have stopped • Network issues • Heartbeat scheduler job has stopped

• Check processes on incoming path and the remote heartbeat scheduler job

• Replication processing is slow on the incoming path

• Use GG_HEARTBEAT table to determine slow process on the incoming path

• Replication processes on the outgoing path have stopped • Network issues • Heartbeat scheduler job has stopped

•Check processes on outgoing path and the local heartbeat scheduler job

• Replication processing is slow on the outgoing path

•Use GG_HEARTBEAT table to determine slow process on the outgoing path


Program Agenda

Heartbeat Functionality

Oracle GoldenGate Diagnosis

• GoldenGate Extended Metrics

• Healthcheck

1

2

12


Potential Sources of Lag

• System Resources

– Memory

– CPU

– IO and Network

• Configuration

– Database

– GoldenGate

13


Tools for Performance Analysis

LogMining Server

Extract Capture client

Extract Pump

Replicat client

Inbound Server

• HealthCheck • UTL_SPADV • AWR & ASH

• HealthCheck • UTL_SPADV • AWR & ASH

GoldenGate Extended Metrics

OS Utilities (netstat)

OS Utilities (top, vmstat, iostat)

OS Utilities (top, vmstat, iostat)

14


New GoldenGate Extended Metrics

• Access to Monitoring Point through Restful Web Services

http://<hostname>:<mgr_port>/mpointsx

• Real-time insight into GoldenGate client programs

• Requires “ENABLEMONITORING” in GLOBALS

15

Fine-grained Performance Monitoring


New Extended Metrics

• Status and Configuration Information

• Process Level Metrics for Extract, Pump and Replicat

• Thread Level Metrics for Extract and Replicat

• Database Statistics for Extract and Replicat

– In-flight-transactions and Cache- & Queue statistics for Extract

– Table statistics for Replicat

• Network Statistics

16

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 17

Utility available on https://java.net/projects/oracledi

Graphical real-time instance monitoring

Replication Components

Process Status Change

Monitoring Metrics

Recording Function


Integrated Healthcheck Report

• The Healthcheck reports a snapshot of the database information for Integrated Extract and Apply.

• Recommendation: Run periodically for differential analysis

• Used for diagnostic purposes

– Configuration Issues

– Performance • Analyze at component and sub-component level

Snapshot of all GoldenGate related Information within the Database

18


Healthcheck Script

1. Database configuration

3. Extract/Replicat Configuration

4. Extract/Replicat Statistics

2. Summary and Notification

19


Notification Section

• Pre-Analysis with INFO, WARNING & ERROR Messages

• Heartbeat Table

20


Record changes

Capture changes

Redo

Integrated Extract

Oracle Database

LCR1 LCR2 LCR3 …. ….

….

Logical Change Records

Extract Trail

Files

Lag within the Integrated Extract

21


GoldenGate Logmining Server Architecture

Extract

• Requests LCRs from logmining

server

• Performs Mapping and

Transformations

• Writes Trail File

Logmining Server (Database Capture Process)

• Reader: Reads logfile and splits into regions

• Preparer: Scans regions of logfiles and prefilters based on extract parameters

• Builder: Merges prepared records in SCN order

• Capture: Formats Logical Change Records(LCRs) and passes to GoldenGate Extract

Reader Capture Builder

Logmining Server

Preparer N

Preparer 2

Preparer 1

Oracle Database

Integrated

Extract Trail

Files

22


Monitoring Integrated Extract

PATH 2 RUN_ID 15 RUN_TIME 2014-FEB-21 22:32:17 CCA N

<C> OGG$CAP_EXT_1A 305260 305245 790813

LMR 0.0% 53.3% 33.3% "CPU + Wait for CPU" proc %idle %flowctrl %TopWait “Event”

LMP(2) 0.0% 0.0% 146.7% "CPU + Wait for CPU"

LMB 73.3% 0.0% 6.7% " "

CAP 20% 0% 73.3% "CPU + Wait for CPU"

<Q> "OGGADMIN"."OGG$Q_EXT_1A" 0.01 0.01 0

<E> EXT_1A 304027 80076708 790813 20% 6.7% 73.3% "CPU + Wait for CPU"

<B> NO BOTTLENECK IDENTIFIED

• % of Idle, Flow Control and major Wait Event for each sub component

• Check output for multiple occurrences over time

• Increase Parallelism of Capture – if CPU is available (LMR in flow control, and LMP is 0% idle with high CPU)

Streams Performance Advisor (UTL_SPADV)

23


• State and "Ready to send Chunks" as one Indicator for an Extract program issue

• If "Ready to send Chunks" >> " Delivered Chunks" and Capture State indicates " Waiting for Client Requests", Extract program needs to checked.

• Check TRANLOGOPTIONS for BUFFER_SIZE , _READAHEADCOUNT & ASYNCTRANSPROCESSING.

Lag between Database and Extract client

24


Integrated Replicat Replicat: Apply section

Trail

Files Replicat

Lightweight Streaming

API

Receiver Preparer Coordinator

Inbound Server

Applier N

Applier 2

Applier 1

Oracle Database

• Waiting for memory might indicate Memory issues Replicat IntegratedParams MAX_SGA_SIZE

25


• Total LCRs with Dependencies indicates transaction dependencies

• One reason for Total LCRs with WM Dependency might be a large transaction

Integrated Replicat Transaction Dependencies

… … … … … … … … … …

20000

• Increasing EAGER_SIZE from Default to a moderately higher number

•

20000

26


Apply Server Statistics – Wait Event Analysis

• Wait Event Analysis of Apply Server Processes

– “Classical Waits”: Enqueues, IO Waits, …

– Replication related Wait Events: Waiting for Dependency

• Check IO Subsystem, Classic Database Tuning (e.g.: INITRANS, Index Rebuild, …)

Apply Server Processes as common Database sessions

REPL Apply: dependency

(CPU or WAIT on CPU)

27


Additional Database Resources AWR & UTL_SPADV

• .

• UTL_SPADV Package to monitor database background processes for Extract and Replicat

– Two ways of displaying information: single line or HTML

– Start and Stop monitoring

– Shows bottleneck based on database wait events for the interval

28


Performance Parameter • Database

– STREAMS_POOL_SIZE,

• Integrated Extract

– TRANLOGOPTIONS

• ASYNCTRANSPROCESSING

• _READAHEADCOUNT

– TRANLOGOPTIONS (INTEGRATEDPARAMS …)

• MAX_SIZE_SIZE

• PARALLELISM

• BUFSIZE

29

•

• Integrated REPLICAT

– DBOPTIONS

• BATCHSQL

• GROUPTRANSOPS (if PARALLELISM=1)

– DBOPTIONS (INTEGRATEDPARAMS …)

• MAX_SGA_SIZE

• PARALLELISM

• MAX_PARALLELISM

• COMMIT_SERIALIZATION

• EAGER_SIZE


Summary

• Monitoring:

– Use Automatic Heartbeat Table

– Run periodically Healthcheck Reports

• Analysis – Use Heartbeat Table to drill down to the component

– Monitor OGG extended Metrics

– Healthcheck Reports

– UTL_SPADV

– Check Database Performance (AWR & ASH) and OS Performance

30

•


References

• Note 1557031.1 Recommended patches for Integrated Capture/Replicat

• Note 1448324.1 OGG Integrated Healthcheck Script

• Note 1488668.1 GoldenGate Performance Data Gathering

• Note 1596493.1 GoldenGate Integration with AWR

• Note 1485620.1 Best Practices Downstream Capture

• MAA white paper Oracle GoldenGate Performance Best Practices

http://www.oracle.com/technetwork/database/availability/maa-gg-performance-1969630.pdf

MOS Notes

31








Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 32

Classroom Training

Learning Subscription

Live Virtual Class

Training On Demand

Keep Learning with Oracle University

education.oracle.com

Cloud

Technology

Applications

Industries


Session Surveys

Help us help you!! • The organizing committee would like to invite you to take a moment to give

us your session feedback. Your feedback will help us to improve your conference.

• Please be sure to add your feedback for your attended sessions by using the Mobile Survey or in Schedule Builder.

33

Documents

Oracle GoldenGate Performance Tuning and Monitoring€¦ · · 2016-06-03Title: How to Use the PowerPoint Template Author: The Presentation Company Created Date: 12/17/2015 12:14:00