Upload
bryan-roundtree
View
217
Download
0
Tags:
Embed Size (px)
Citation preview
2
Background GoldenGate• Overview• Setup• Operations• Troubleshooting• Monitoring• Challenges
Q&A
www.LamkoDB.com www.MonitorMyDatabase.com
Agenda
3
Using Oracle GoldenGate since 2010 At clients• Oracle 9i, 10g near-zero downtime migrations to 11g RAC• Oracle 11g RAC setup for DR and EDW load• Upgrade OGG 10.4 to 11.2 for Oracle 11g clusters (no app
downtime)• Oracle 9i, 10g, 11g EDW load• SQL Server 2008 to Oracle 11g EDW load
At LamkoDB• Oracle 10g, 11g, 12c, SQL Server 2008, 2012, MySQL tests with
OGG 11g and 12c
www.LamkoDB.com www.MonitorMyDatabase.com
Background
4
What GoldenGate does• Continuous data replication from one database to another• Source and target databases can be of different
vendors/releases• Non-intrusive, low-impact, sub-second latency • Maintains transactional integrity - Resilient against interruptions
and failures • Data transformation on the fly• Few or many tables in a schema• DDL replication if desired• Used for EDW feeds, DR, DB version upgrades, DB migrations
What GoldenGate does not• Is not a replacement for Data Guard
www.LamkoDB.com www.MonitorMyDatabase.com
Overview
5www.LamkoDB.com www.MonitorMyDatabase.com
Overview
6
Use of replication
www.LamkoDB.com www.MonitorMyDatabase.com
Overview
7
Source Databases/Platforms supported
www.LamkoDB.com www.MonitorMyDatabase.com
Overview
8
Target Databases/Platforms supported
www.LamkoDB.com www.MonitorMyDatabase.com
Overview
9
Processes/Files involved
Classic Capture or Integrated Capture mode Nonintegrated Replicat or Integrated Replicat
www.LamkoDB.com www.MonitorMyDatabase.com
Overview
10
Observations• Not all data types are supported• Need data type translation in heterogeneous environments• Encryption of data in files and across network• Optional compression of data copied by pump process• Cryptic names for processes: 8 characters• Cryptic names of trail files e1xxxxxx (2 characters followed by
incrementing number)• Administrator must like command line tool (similar to sqlplus)
www.LamkoDB.com www.MonitorMyDatabase.com
Overview
11
Installation GG software (source & target server)• Download software from Oracle site. Software to use is Oracle
DBMS release specific.• On Linux/Unix create account that owns the software & add to
dba group• Create profile to set $ORACLE_HOME, $PATH, etc. for the
account• Unzip & untar software or use Oracle Universal Installer• Configure GLOBALS and mgr.prm files
Installation in DB• DB in archive log mode (at least source instance)• Turn off recyclebin (in source instance for DDL replication)• Put DB in supplemental logging mode (source instance)• Run OGG supplied installation scripts (source & target)
www.LamkoDB.com www.MonitorMyDatabase.com
Setup
12
How many schemas/tables to replicate & how much data will be replicated?
Have tables the same structure in source and target database or do target tables have additional columns?
Do tables have PKs? Are there referential constraints to deal with if not all
tables are being replicated? Delete Cascade constraints When bi-directional replication• Sequences• Triggers firing on either database
www.LamkoDB.com www.MonitorMyDatabase.com
Setup items to keep in mind
13
Setup replication flow & initial load of data• Create/start Extract on source database• Create/start Pump process on source database• Do initial data load from source to target database• Create/start Replicat on target database
www.LamkoDB.com www.MonitorMyDatabase.com
Replication Setup Example
14
Extract• Extract process E1ORA1• E1ORA1 extracts data from MMDBORA1 (Oracle) to local trail
files in directory/gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1
www.LamkoDB.com www.MonitorMyDatabase.com
Setup
15
EXTRACT
E01ORA1.obey
ADD EXTRACT E01ORA1, TRANLOG, BEGIN NOW, params /gg/product/11.2.1_11g/ggs/dirprm/MMDBORA1/E01ORA1.prm
ADD EXTTRAIL /gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1, EXTRACT E01ORA1
E01ORA1.prm
EXTRACT E01ORA1
. . .
USERID ggate, PASSWORD abc
EXTTRAIL /gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1 megabytes 100
. . .
TABLE CHINOOK.*;
start E01ORA1
www.LamkoDB.com www.MonitorMyDatabase.com
Setup
16
Pump• Pump process P1ORA1 copies local trail file contents to remote
trail files on another server in directory C:\gg\dirdat\MMDBORA1\p1
www.LamkoDB.com www.MonitorMyDatabase.com
Setup
17
PUMP
P01ORA1.obey
ADD EXTRACT P01ORA1, EXTTRAILSOURCE /gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1, params /gg/product/11.2.1_11g/ggs/dirprm/MMDBORA1/P01ORA1.prm
ADD RMTTRAIL C:\gg\dirdat\MMDBORA1\p1, EXTRACT P01ORA1, MEGABYTES 100
P01ORA1.prm
EXTRACT P01ORA1
RMTHOST 192.168.21.94, MGRPORT 7809
RMTTRAIL C:\gg\dirdat\MMDBORA1\p1, megabytes 100
PASSTHRU
TABLE CHINOOK.*;
start P01ORA1
www.LamkoDB.com www.MonitorMyDatabase.com
Setup
18
Initial load of data• Export/Import• Data pump• RMAN• Load data over a DB link• bcp• OGG utilities• many more …
www.LamkoDB.com www.MonitorMyDatabase.com
Setup
19
Replicat• Replicat process R01ORA1 (SQL Server 2012) reads the remote
trail file contents and applies rows to the database
www.LamkoDB.com www.MonitorMyDatabase.com
Setup
20
REPLICAT
R01ORA1.obey
DBLOGIN SOURCEDB GGATE, USERID ggate, PASSWORD abcadd replicat R01ORA1, exttrail C:\gg\dirdat\MMDBORA1\p1 params C:\gg\dirprm\CHINOOK\R01ORA1.prm
R01ORA1.prm
REPLICAT R01ORA1
SOURCEDB GGATE, USERID ggate, PASSWORD abc
. . .
MAP CHINOOK.ALBUM, TARGET dbo.ALBUM;
MAP CHINOOK.ARTIST, TARGET dbo.ARTIST;
MAP CHINOOK.CUSTOMER, TARGET dbo.CUSTOMER;
MAP CHINOOK.TRACK, TARGET dbo.TRACK;
start R01ORA1
www.LamkoDB.com www.MonitorMyDatabase.com
Setup
21
Replication flow complete
www.LamkoDB.com www.MonitorMyDatabase.com
Setup
22
ggsci command line interface• info all• info * or info E*, etc• info E01ORA1• info E01ORA1 detail• start E01ORA1• stop E01ORA1• stats E01ORA1• send E01ORA1 status• many more …
www.LamkoDB.com www.MonitorMyDatabase.com
Operations
23
ggsci commands
GGSCI (MMDBLNXDB01) 1> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
EXTRACT RUNNING E01ORA1 00:00:00 00:00:04
EXTRACT RUNNING E02ORA1 00:00:00 00:00:04
EXTRACT RUNNING P01ORA1 00:00:00 00:00:09
EXTRACT RUNNING P02ORA1 00:00:00 00:00:00
REPLICAT RUNNING R01SQL1 00:00:00 00:00:04
GGSCI (MMDBLNXDB01) 2> info E01ORA1
EXTRACT E01ORA1 Last Started 2014-02-20 15:49 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:02 ago)
Log Read Checkpoint Oracle Redo Logs
2014-03-13 21:52:56 Seqno 3726, RBA 25596928
SCN 0.127734271 (127734271)
www.LamkoDB.com www.MonitorMyDatabase.com
Operations
24
Network issues causing pump to crash• Usually restart pump• Use AUTORESTART in mgr.prm
www.LamkoDB.com www.MonitorMyDatabase.com
Operational Issues
25
Network issues causing pump to crash• Usually restart pump• Use AUTORESTART in mgr.prm
Server reboot and processes not starting• Use AUTOSTART in mgr.prm
www.LamkoDB.com www.MonitorMyDatabase.com
Operational Issues
26
Network issues causing pump to crash• Usually restart pump• Use AUTORESTART in mgr.prm
Server reboot and processes not starting• Use AUTOSTART in mgr.prm
Extract performance bad• Use dedicated volumes where redo/archived logs are stored• Have separate volumes for local trail files• Use multiple extracts
www.LamkoDB.com www.MonitorMyDatabase.com
Operational Issues
27
Network issues causing pump to crash• Usually restart pump• Use AUTORESTART in mgr.prm
Server reboot and processes not starting• Use AUTOSTART in mgr.prm
Extract performance bad• Use dedicated volumes where redo/archived logs are stored• Have separate volumes for local trail files• Use multiple extracts
Pump performance bad• Compress data when sending data across• Have separate volumes for local/remote trail files• Use multiple extracts/pumps
www.LamkoDB.com www.MonitorMyDatabase.com
Operational Issues
28
Replicat performance bad • Check for tables with missing PKs• Use batchsql in prm file of replicat• Have separate volumes for remote trail files• Use multiple replicats
www.LamkoDB.com www.MonitorMyDatabase.com
Operational Issues
29
Replicat performance bad • Check for tables with missing PKs• Use batchsql in prm file of replicat• Have separate volumes for remote trail files• Use multiple replicats
Disk full for local and/or remote trail files• Occasionally need to setup replication again, with or without
initial load• Monitor disk space usage by storage group or sysadmins• Have separate volumes for local and remote trail files• Use PURGEOLDEXTRACTS in mgr.prm
www.LamkoDB.com www.MonitorMyDatabase.com
Operational Issues
30
Crashing replicats when tables added/changed• Add/modify tables on target database, possibly update data
definition file, and restart replicat• Make extract process stop when it notices DDL
www.LamkoDB.com www.MonitorMyDatabase.com
Operational Issues
31
Crashing replicats when tables added/changed• Add/modify tables on target database, possibly update data
definition file, and restart replicat• Make extract process stop when it notices DDL
Extract process does not stop• Commit open transactions in DB• Kill session having open transaction• Force stop of Extract process if not many archived logs to re-
read at next Extract startup
www.LamkoDB.com www.MonitorMyDatabase.com
Operational Issues
32
Crashing replicats when tables added/changed• Add/modify tables on target database, possibly update data
definition file, and restart replicat• Make extract process stop when it notices DDL
Extract process does not stop• Commit open transactions in DB• Kill session having open transaction• Force stop of Extract process if not many archived logs to re-
read at next Extract startup Crashing replicats because of manual data changes in
target database• Do more manual changes to fix the data• Possibly reload affected tables• Educate users
www.LamkoDB.com www.MonitorMyDatabase.com
Operational Issues
33
Where to look for information• Report file of process (view report <process>)• Discard file• OGG Error log• Oracle alert log / SQL Server ERRORLOG
Compare source and target data after fixing issue• Veridata• select data from source table MINUS select data from target
table• Oracle DBMS_COMPARISON• Other tools (TOAD, …)
www.LamkoDB.com www.MonitorMyDatabase.com
Troubleshooting
34
Setup Heartbeat in replication flow• Document ID: 1299679.1 OGG_Best_Practice_-_heartbeat_table_using_DBMS_SCHEDULER-_V11_0_ID1299679.1.pdf
• Update heartbeat record in source database (every minute)• Extract picks up record in source database, adds information
and write to local trail file• Pump picks up record, adds information and write to remote
trail file• Replicat picks up record, adds information and writes to target
database (as an insert)
www.LamkoDB.com www.MonitorMyDatabase.com
Monitoring
35
Monitor using Oracle tool (Director)• Need to install Java agent in every GG instance & maintain
bunch of servers to keep tool running (Weblogic, …) (Remote) Monitor using own developed tools• Monitor status of processes• Monitor number of rows replicated• Report on Heartbeat data• Queries against heartbeat table in target database• Graphs generated based on data in heartbeat table (lag
times)• ggsci-like GG Dashboard for all GG instances monitored
www.LamkoDB.com www.MonitorMyDatabase.com
Monitoring
36
OGG Management Pack
www.LamkoDB.com www.MonitorMyDatabase.com
Monitoring
37
LamkoDB Monitoring
www.LamkoDB.com www.MonitorMyDatabase.com
Monitoring
38
Historical stats for Extract process
www.LamkoDB.com www.MonitorMyDatabase.com
Monitoring
39
Lag times for Extract/Pump/Replicat
www.LamkoDB.com www.MonitorMyDatabase.com
Monitoring
40
GG Dashboard
www.LamkoDB.com www.MonitorMyDatabase.com
Monitoring
41
GG process detail information
www.LamkoDB.com www.MonitorMyDatabase.com
Monitoring
42
GG process relationships
www.LamkoDB.com www.MonitorMyDatabase.com
Monitoring
43
Who supports GoldenGate?• DBA• Middleware group
www.LamkoDB.com www.MonitorMyDatabase.com
Challenges
44
Who supports GoldenGate?• DBA• Middleware group
What to do about table structure changes• software releases need to be evaluated/tested/coordinated
www.LamkoDB.com www.MonitorMyDatabase.com
Challenges
45
Who supports GoldenGate?• DBA• Middleware group
What to do about table structure changes• software releases need to be evaluated/tested/coordinated
What do developers, support, clients know about OGG?• Educate personnel
www.LamkoDB.com www.MonitorMyDatabase.com
Challenges
46
Who supports GoldenGate?• DBA• Middleware group
What to do about table structure changes• software releases need to be evaluated/tested/coordinated
What do developers, support, clients know about OGG?• Educate personnel
Monitor what is running and what is broken• Have proper monitoring in place with alerting
www.LamkoDB.com www.MonitorMyDatabase.com
Challenges
47
Who supports GoldenGate?• DBA• Middleware group
What to do about table structure changes• software releases need to be evaluated/tested/coordinated
What do developers, support, clients know about OGG?• Educate personnel
Monitor what is running and what is broken• Have proper monitoring in place with alerting
Keeping track of new, existing and changing replication flows. Number of processes quickly becomes a spaghetti bowl.• Have automated tool to show relationships
www.LamkoDB.com www.MonitorMyDatabase.com
Challenges