Oracle GoldenGate Performance Tuning and Monitoring Ease of Manageability
Volker Kuhr Senior Principal Product Manager Jiahui Shao Product Manager
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate and Data Integration
Energy/Industrial Insurance / Health Retail Other
Communications Finance / Banking Media Services Over 10K Customers Worldwide
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate Architecture
Bi-Directional
Capture Trail Files Network Firewall Cloud
Trail Files Delivery Database
Database
3
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Automatic Heartbeat Functionality
Oracle GoldenGate Diagnosis
1
2
4
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Automatic Heartbeat Functionality
Oracle GoldenGate Diagnosis
1
2
5
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
New 12.2 Feature - Automatic Heartbeats
6
Built-in Functionality to Monitor End-to-End Replication Lag
Simple Automatic • Enable via a single “ADD
HEARTTBEATTABLE” command at each site
• Automatically discovers replication flows for all topologies
–Unidirectional –Bidirectional –N-way
Real-time • Database views to query
end-to-end real-time latency
• Lag information on incoming flow to and outgoing flow from a database
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Simple Bidirectional Replication Example • Enable Heartbeat functionality by executing GGSCI command ‘ADD
HEARTBEATTABLE’ at each database – Creates required heartbeat tables, views and jobs – Updates heartbeat every 60s by default
7
ExtA
ExtB
PmpAB RepAB
RepBA PmpBA
In Database A: GG_LAG View contents
LOCAL_ DATABASE
REMOTE_ DATABASE INCOMING_PATH INCOMING_
LAG
INCOMING_ HEARTBEAT_
AGE OUTGOING_PATH OUTGOING_
LAG
OUTGOING_ HEARTBEAT_
AGE
A B ExtB==>PmpBA==>RepBA 1.066182 14.150614 ExtA==>PmpAB==>RepAB 1.391718 25.387458
Database A Database B
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Monitoring Lag using GG_LAG View
8
Column Name Data type Description
LOCAL_DATABASE VARCHAR2(30) Local database name
CURRENT_LOCAL_TS TIMESTAMP(6) Current timestamp in UTC time zone
REMOTE_DATABASE VARCHAR2(30) Remote database name
INCOMING_PATH VARCHAR2(4000) Group names on the incoming flow
INCOMING_LAG NUMBER Period of time between remote database generating heartbeat and local database receiving heartbeat
INCOMING_HEARTBEAT_AGE NUMBER CURRENT_LOCAL_TS minus most recent heartbeat timestamp of remote database
OUTGOING_PATH VARCHAR2(4000) Group names on the outgoing flow
OUTGOING_LAG NUMBER Period of time between local database generating heartbeat and remote database receiving heartbeat
OUTGOING_HEARTBEAT_AGE NUMBER CURRENT_LOCAL_TS minus recent heartbeat timestamp of local database
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
• GG_LAG_HISTORY View – Historical heartbeat lag
• GG_HEARTBEAT and GG_HEARTBEAT_HISTORY Tables – Underlying tables for the views – Get lag for each process on the path
9
Additional Tables and Views
Column Name
LOCAL_DATABASE
CURRENT_LOCAL_TS
REMOTE_DATABASE
INCOMING_PATH
INCOMING_LAG
INCOMING_HEARTBEAT_AGE
OUTGOING_PATH
OUTGOING_LAG
OUTGOING_HEARTBEAT_AGE
Column Name
INCOMING_HEARTBEAT_TS
INCOMING_EXTRACT_TS
INCOMING_ROUTING_TS
INCOMING_REPLICAT_TS
OUTGOING_HEARTBEAT_TS
OUTGOING_EXTRACT_TS
OUTGOING_ROUTING_TS
OUTGOING_REPLICAT_TS
…
} Extract lag } Pump lag } Replicat lag
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Discovers New Replication Flows Automatically
• Simply execute ADD HEARTBEATTABLE for new site
10
Example: Add new site Database C
In Database A: GG_LAG View contents LOCAL_
DATABASE REMOTE_ DATABASE INCOMING_PATH OUTGOING_PATH
A B ExtB==>PmpBA==>RepBA ExtA==>PmpAB==>RepAB
A C ExtC==>PmpCA==>RepCA ExtA==>PmpAC==>RepAC
Database A Database B
Database C (Newly added site)
ExtA==>PmpAB==>RepAB
RepBA<==PmpBA<==ExtB
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Diagnosing Replication Issues using Heartbeat Table
11
Diagnosis Cause
INCOMING_LAG is high
OUTGOING_HEARTBEAT_AGE is high
INCOMING_HEARTBEAT_AGE is high
OUTGOING_LAG is high
Action • Replication processes on the incoming path have stopped • Network issues • Heartbeat scheduler job has stopped
• Check processes on incoming path and the remote heartbeat scheduler job
• Replication processing is slow on the incoming path
• Use GG_HEARTBEAT table to determine slow process on the incoming path
• Replication processes on the outgoing path have stopped • Network issues • Heartbeat scheduler job has stopped
•Check processes on outgoing path and the local heartbeat scheduler job
• Replication processing is slow on the outgoing path
•Use GG_HEARTBEAT table to determine slow process on the outgoing path
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Heartbeat Functionality
Oracle GoldenGate Diagnosis
• GoldenGate Extended Metrics
• Integrated Healthcheck
1
2
12
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Potential Sources of Lag • System Resources
– Memory – CPU – IO and Network
• Configuration – Database – GoldenGate
13
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Tools for Performance Analysis
LogMining Server
Extract Pump
Replicat Inbound Server
• HealthCheck • UTL_SPADV • AWR & ASH
• HealthCheck • UTL_SPADV • AWR & ASH
GoldenGate Extended Metrics
OS Utilities (netstat)
OS Utilities (top, vmstat, iostat)
OS Utilities (top, vmstat, iostat)
14
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
New GoldenGate Extended Metrics
• Access to Monitoring Point through Restful Web Services http://<hostname>:<mgr_port>/mpointsx
• Real-time insight into GoldenGate client programs
• Requires “ENABLEMONITORING” in GLOBALS
15
Fine-grained Performance Monitoring
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
New Extended Metrics • Status and Configuration Information • Process Level Metrics for Extract, Pump and Replicat • Thread Level Metrics for Extract and Replicat • Database Statistics for Extract and Replicat
– In-flight-transactions and Cache- & Queue statistics for Extract – Table statistics for Replicat
• Network Statistics
16
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
17
https://java.net/projects/oracledi/downloads/download/GoldenGate/OGGPTK.jar Graphical real-time instance monitoring
Replication Components
Process Status Change
Monitoring Metrics
Recording Function
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Integrated Healthcheck Report
• Comprehensive snapshot of the database components of Integrated Extract and Apply.
• Best Practices: Run periodically for differential analysis • Used for diagnostic purposes
– Configuration Issues – Performance
• Analyze at component and sub-component level
Snapshot of all GoldenGate related Information within the Database
18
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Healthcheck Script
1. Database configuration
3. Extract/Replicat Configuration
4. Extract/Replicat Statistics
2. Summary and Notification
19
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Notification Section
• Pre-Analysis with INFO, WARNING & ERROR Messages
• Heartbeat Table
20
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Record changes
Capture changes
Redo
Integrated Extract
Oracle Database
LCR1 LCR2 LCR3 …. …. ….
Logical Change Records
Extract Trail Files
Lag within the Integrated Extract
21
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
GoldenGate Logmining Server Architecture
Extract
• Requests LCRs from logmining server
• Performs Mapping and Transformations
• Writes Trail File
Logmining Server (Database Capture Process)
• Reader: Reads logfile and splits into regions
• Preparer: Scans regions of logfiles and prefilters based on extract parameters
• Builder: Merges prepared records in SCN order
• Capture: Formats Logical Change Records(LCRs) and passes to GoldenGate Extract
Reader Capture Builder
Logmining Server
Preparer N
Preparer 2
Preparer 1
Oracle Database
Integrated Extract
Trail Files
22
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Monitoring Integrated Extract
PATH 2 RUN_ID 15 RUN_TIME 2014-FEB-21 22:32:17 CCA N <C> OGG$CAP_EXT_1A 305260 305245 790813 LMR 0.0% 53.3% 33.3% "CPU + Wait for CPU" proc %idle %flowctrl %TopWait “Event” LMP(2) 0.0% 0.0% 146.7% "CPU + Wait for CPU" LMB 73.3% 0.0% 6.7% " " CAP 20.0% 0.0% 73.3% "CPU + Wait for CPU" <Q> "OGGADMIN"."OGG$Q_EXT_1A" 0.01 0.01 0 <E> EXT_1A 304027 80076708 790813 20% 6.7% 73.3% "CPU + Wait for CPU" <B> NO BOTTLENECK IDENTIFIED
• % of Idle, Flow Control and major Wait Event for each sub component • Check output for multiple occurrences over time • Increase Parallelism of Capture – if CPU is available
(LMR in flow control, and LMP is 0% idle with high CPU)
Streams Performance Advisor (UTL_SPADV)
23
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
• State and "Ready to send Chunks" as one Indicator for an Extract program issue
• If "Ready to send Chunks" >> " Delivered Chunks" and Capture State indicates " Waiting for Client Requests", Extract program needs to checked.
• Check Extended Metrics and TRANLOGOPTIONS • BUFFER_SIZE , _READAHEADCOUNT, ASYNCTRANSPROCESSING …
Lag between Database and Extract client
24
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Integrated Replicat Replicat: Apply section
Trail Files
Replicat Lightweight Streaming
API Receiver Preparer Coordinator
Inbound Server
Applier N
Applier 2
Applier 1
Oracle Database
• Waiting for memory might indicate Memory issues Replicat IntegratedParams MAX_SGA_SIZE
25
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
• Total LCRs with Dependencies indicates transaction dependencies • One reason for Total LCRs with WM Dependency might be a large transaction
Integrated Replicat Transaction Dependencies
… … … … … … … … … …
20000
• Increasing EAGER_SIZE from Default to a moderately higher number
20000
26
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Apply Server Statistics – Wait Event Analysis
• Wait Event Analysis of Apply Server Processes – “Classical Waits”: Enqueues, IO Waits, … – Replication related Wait Events
• Check IO Subsystem, Classic Database Tuning (e.g.: INITRANS, Index Rebuild, …)
Apply Server Processes as common Database sessions
(wait on CPU /on CPU)
(wait on CPU /on CPU)
REPL Apply: dependency
27
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Additional Database Resources AWR & UTL_SPADV • . • UTL_SPADV Package to monitor database background
processes for Extract and Replicat – Two ways of displaying information: single line or HTML – Start and Stop monitoring – Shows bottleneck based on database wait events for the interval
28
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Key Performance Tuning Parameters • Database
– STREAMS_POOL_SIZE
• Integrated Extract – TRANLOGOPTIONS
• BUFSIZE • ASYNCTRANSPROCESSING • _READAHEADCOUNT
– TRANLOGOPTIONS INTEGRATEDPARAMS (…)
• MAX_SGA_SIZE • PARALLELISM
29 •
• Integrated REPLICAT – DBOPTIONS
• BATCHSQL • GROUPTRANSOPS (if PARALLELISM=1)
– DBOPTIONS INTEGRATEDPARAMS (…)
• MAX_SGA_SIZE • PARALLELISM • MAX_PARALLELISM • COMMIT_SERIALIZATION • EAGER_SIZE
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Summary • Monitoring:
– Use Automatic Heartbeat Table – Run periodically Healthcheck Reports
• Analysis – Use Heartbeat Table to drill down to the component – Monitor OGG extended Metrics – Healthcheck Reports – UTL_SPADV – Check Database Performance (AWR & ASH) and OS Performance
30 •
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
References
• MAA white paper Oracle GoldenGate Performance Best Practices http://www.oracle.com/technetwork/database/availability/maa-gg-performance-
1969630.pdf
• Note 1557031.1 Recommended patches for Integrated Capture/Replicat • Note 1448324.1 OGG Integrated Healthcheck Script • Note 1488668.1 GoldenGate Performance Data Gathering • Note 1596493.1 GoldenGate Integration with AWR • Note 1485620.1 Best Practices Downstream Capture
MOS Notes
31