Configuring High Availability for the PMG DB...Configuring High Availability for the PMG DB ThischapterdescribestheprocessofconfiguringhighavailabilityforthePMGDB.Itprovidesthe

Configuring High Availability for the PMG DB

This chapter describes the process of configuring high availability for the PMG DB. It provides theprerequisites and procedures required to configure and test the hot standby and cold standby for Cisco RMS.It also includes procedures required to back up and restore standby and primary configurations, troubleshootData Guard on PMG DB, and upgrade Cisco RMS.

The Oracle Data Guard tool is used to ensure high availability and disaster recovery for PMG DB in CiscoRMS. This tool is used to configure one primary database and one or more standby databases. These databasesin a Data Guard configuration are connected by Oracle Net andmay be dispersed geographically. The primaryand standby databases can be managed using the SQL CLIs or the Data Guard Broker interfaces, includinga CLI and a GUI that is integrated in the Oracle Enterprise Manager.

The Oracle Enterprise Manager GUI or the Data Guard (DGMGRL) CLI is used to enable fast-start failoverto fail over automatically when the primary database becomes unavailable. When a fast-start failover is

High Availability for Cisco RAN Management Systems 1

enabled, the Data Guard Broker determines if a failover is necessary and initiates the failover to the specifiedtarget standby database automatically, without a need for DBA intervention.

A failover occurs when the primary database is unavailable. Failover is performed only in the event of afailure of the primary database, and the failover results in a transition of a standby database to the primaryrole.

A switchover is a role reversal between the primary database and one of its standby databases. This is typicallydone for planned maintenance of the primary system. During a switchover, the primary database transitionsto a standby role, and the standby database transitions to the primary role.

The following sections describes the prerequisites, process for configuring high availability for the PMGDB, and Cisco RMS upgrade information.

The output of some of the commands may slightly differ based on the installation setup and time of theexecution.

Note

• Prerequisites, page 2

• Configuration Workflow, page 3

• Configuring the Standby Server Setup, page 4

• Setting Up the Oracle Data Guard Broker, page 16

• Enabling Flashback, page 20

• Configuring Hot Standby for PMG DB, page 21

• Configuring Cold Standby , page 34

• Testing Hot Standby, page 35

• Testing Cold Standby, page 45

• Rolling Back and Cleaning Up Standby and Primary Configurations, page 55

• Troubleshooting Data Guard on PMG DB, page 65

Prerequisites• Primary server should have a running instance.

• Standby server should have a software only installation.

• Operator or user of this procedure should have the following experience:

◦Knowledge of basic database/DBA know-how

◦Knowledge of basic Linux/shell commands

◦Ability to edit files with vi or vim

◦Ability to view files (cat, tail, more, less)

High Availability for Cisco RAN Management Systems2

Configuring High Availability for the PMG DBPrerequisites

• ORACLE_HOME directory path should be /u01/app/oracle/product/11.2.0/dbhome_1. If this path isdifferent, respective file paths (for example, data files) would differ.

• To enable the failover feature for Cisco RMS applications connecting to PMGDB, the Cisco RMSCentralnode should be configured for PMG DB as per the install guide (see Cisco RAN Management SystemInstallation Guide).

The installation can be performed on either UCS 5108 Blade Server or UCS 240 Server.There is no specific dependency based type of hardware.

Note

Configuration WorkflowThe following table provides the general flow which can be followed to perform tasks for different types ofredundancy setup for PMGDB using the Oracle Data Guard tool.

ProcedureTaskSl. No.

Configuring the Standby Server Setup, onpage 4

Configure the Standby Server1

Configuring the Hot Standby Server, on page10

Add Only Hot Standby to Primary Server2

Configuring Primary With Only ColdStandby, on page 34

Add Only Cold Standby to Primary Server3

Configuring Primary With Hot and ColdStandby, on page 34

Add Both, Hot and Cold Standby toPrimary Server

4

Testing Hot Standby, on page 35Test Hot Standby Setup5

• Testing Site Failure, on page 46

• Recovering Original Primary After SiteFailure, on page 48

Test Cold Standby Setup6

Troubleshooting Data Guard on PMG DB,on page 65

Troubleshooting PMG DB7

Rolling Back and Cleaning Up Standby andPrimary Configurations, on page 55

Rollback and Clean-up8


Configuring High Availability for the PMG DBConfiguration Workflow

Configuring the Standby Server SetupThe following procedure describes how to configure a standby server for the primary server. If the PMG DBredundancy setup involves primary with only cold standby, this standby server can be configured as coldstandby server. If the redundancy setup involves primary with only hot standby or primary with both hot andcold standby, this standby can be configured as a hot standby.

• Configuring the Primary Server, on page 4

• Configuring the Hot Standby Server, on page 10

Configuring the Primary Server

Step 1 Log in to the primary server with the username, “oracle”.Step 2 Log in to the sql prompt.

Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 17:51:51 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL>

Step 3 Create the pfile as a backup vSphere client. This pfile can be used later when rollback is needed.The specific path for initPMGDB_pre_dgsetup.ora file can be specified if the file is to be stored in some backup directory(for example, /backup/initPMGDB_pre_dgsetup.ora). If the path is not specified the file gets created under the defaultdirectory, that is, $ORACLE_HOME/dbs.

Example:SQL> CREATE PFILE='initPMGDB_pre_dgsetup.ora' FROM SPFILE;File created.

Logging

Step 1 Ensure that the primary database is in archivelog mode.Step 2

Example:SQL> SELECT log_mode FROM v$database;LOG_MODE------------NOARCHIVELOG


Configuring High Availability for the PMG DBConfiguring the Standby Server Setup

If it is noarchivelog mode, switch to archivelog mode:SQL> SHUTDOWN IMMEDIATE;Database closed.Database dismounted.ORACLE instance shut down.

SQL> STARTUP MOUNT;ORACLE instance started.

Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.

SQL> ALTER DATABASE ARCHIVELOG;Database altered.

SQL> ALTER DATABASE OPEN;Database altered.

SQL> SELECT log_mode FROM v$database;LOG_MODE------------ARCHIVELOG

Step 3 Enable forced logging by using the following command.

Example:SQL> ALTER DATABASE FORCE LOGGING;Database altered.

Initializing Parameters

Step 1 Check the setting for the DB_NAME and DB_UNIQUE_NAME parameters. In this case they are both set to "PMGDB"on the primary database.

Example:SQL> show parameter db_nameNAME TYPE VALUE------------------------------------ ----------- -----------------db_name string PMGDB

SQL> show parameter db_unique_nameNAME TYPE VALUE------------------------------------ ----------- -----------------db_unique_name string PMGDB

Step 2 Enter the DB_NAME of the standby database. It should be the same as that of the primary, but it must have a differentDB_UNIQUE_NAME value. The DB_UNIQUE_NAME values of the primary and standby database should be used inthe DG_CONFIG setting of the LOG_ARCHIVE_CONFIG parameter. For this example, the standby database has thevalue "PMGDB_STBY".


Configuring High Availability for the PMG DBConfiguring the Primary Server

Example:SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(PMGDB,PMGDB_STBY)';System altered.

Step 3 Set the suitable remote archive log destinations. In this case, flash recovery area for the local location is used; however,other location can be specified. Note the SERVICE and the DB_UNIQUE_NAME for the remote location reference thestandby location.

Example:SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=pmgdb_stby NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PMGDB_STBY';LOG_ARCHIVE_CONFIG='DG_CONFIG=(PMGDB,PMGDB_STBY)';System altered.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;System altered.

Step 4 Set the LOG_ARCHIVE_FORMAT and LOG_ARCHIVE_MAX_PROCESSES parameters to appropriate values andthe REMOTE_LOGIN_PASSWORDFILE must be set to exclusive.

Example:SQL> ALTER SYSTEM SET LOG_ARCHIVE_FORMAT='%t_%s_%r.arc' SCOPE=SPFILE;System altered.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_MAX_PROCESSES=30;System altered.

SQL> ALTER SYSTEM SET REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE SCOPE=SPFILE;System altered.

Step 5 Set the following parameters (in addition to the previous setting) to ensure that the primary server is ready to switch tostandby mode. (This setting is recommended.)Adjust the *_CONVERT parameters to account for filename and path differences between the servers.

Example:SQL> ALTER SYSTEM SET FAL_SERVER=PMGDB_STBY;System altered.

SQL> ALTER SYSTEM SET DB_FILE_NAME_CONVERT='PMGDB_STBY','PMGDB' SCOPE=SPFILE;System altered.

SQL> ALTER SYSTEM SET LOG_FILE_NAME_CONVERT='PMGDB_STBY','PMGDB' SCOPE=SPFILE;System altered.

SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;System altered.

Step 6 Restart the database to implement the modifications made to the parameters.

Example:SQL> shutdown immediateDatabase closed.Database dismounted.ORACLE instance shut down.

SQL> startupORACLE instance started.Total System Global Area 3340451840 bytes



Fixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.Database opened.

Step 7 Exit from sql prompt.

Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

Setting Up the Service

Step 1 Provide the values for the primary and standby databases in the "$ORACLE_HOME/network/admin/tnsnames.ora" fileson both the servers. The following values are required. The values of database server hostname or IP address, ports, andoracle home should be specified as per the installation setup.

Example:PMGDB =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <primary-server-host-address>)(PORT = <port number>)(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)

))

PMGDB_STBY =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <standby-server-host-address>)(PORT = <port number>))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)

))

Step 2 Edit "$ORACLE_HOME/network/admin/listener.ora" to add the entry that the Data Guard Broker will refer to. IfSID_LIST is already present, add SID_DESC to the list.

Example:SID_LIST_LISTENER =(SID_LIST =(SID_DESC =(GLOBAL_NAME = PMGDB_DGMGRL)(ORACLE_HOME = <oracle-home-directory-path>)(SID_NAME = PMGDB)

))

Starting the ListenerEnsure that the listener is started on the primary server.



All of the output is not displayed.Note

$ lsnrctl statusIf not running:LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 02-JUL-2014 09:35:37Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<host address>)(PORT=<port number>))TNS-12541: TNS:no listenerTNS-12560: TNS:protocol adapter errorTNS-00511: No listenerLinux Error: 111: Connection refused

If running:LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 02-JUL-2014 09:43:23Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<host address>)(PORT=<port number>)))STATUS of the LISTENER------------------------…Services Summary...……The command completed successfully

1 If listener is not running (that is, TNS-12541: TNS:no listener), start it.$ lsnrctl startLSNRCTL for Linux: Version 11.2.0.1.0 - Production on 17-JUN-2014 19:10:31Copyright (c) 1991, 2009, Oracle. All rights reserved.Starting /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr:please wait...……The command completed successfully

2 If listener is running, reload it.$ lsnrctl reloadLSNRCTL for Linux: Version 11.2.0.1.0 - Production on 17-JUN-2014 19:12:23Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to…The command completed successfully

Backing Up the Primary Database

Step 1 Back up the primary database, which will be restored on the standby database. To do this, open another console for theprimary database server and log in as an oracle user.

Step 2 Set the ORACLE_SID to PMGDB and take a backup.The complete output is notdisplayed.

Note

Example:$ export ORACLE_SID=PMGDB$ rman target=/

Recovery Manager: Release 11.2.0.1.0 - Production on Thu Jul 3 18:53:35 2014

Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.



connected to target database: PMGDB (DBID=122120137)

RMAN> BACKUP DATABASE PLUS ARCHIVELOG;Starting backup at 03-JUL-14current log archivedusing target database control file instead of recovery catalogallocated channel: ORA_DISK_1channel ORA_DISK_1: SID=97 device type=DISKchannel ORA_DISK_1: starting archived log backup setchannel ORA_DISK_1: specifying archived log(s) in backup setinput archived log thread=1 sequence=15 RECID=93 STAMP=851365692input archived log thread=1 sequence=16 RECID=95 STAMP=851367216………………channel ORA_DISK_1: starting full datafile backup setchannel ORA_DISK_1: specifying datafile(s) in backup setinput datafile file number=00005 name=/u01/app/oracle/oradata/PMGDB/mapinfo_ts.dbfinput datafile file number=00006 name=/u01/app/oracle/oradata/PMGDB/pmgdb_ts.dbfinput datafile file number=00001 name=/u01/app/oracle/oradata/PMGDB/system01.dbfinput datafile file number=00002 name=/u01/app/oracle/oradata/PMGDB/sysaux01.dbfinput datafile file number=00003 name=/u01/app/oracle/oradata/PMGDB/undotbs01.dbfinput datafile file number=00004 name=/u01/app/oracle/oradata/PMGDB/users01.dbfpiecehandle=/u01/app/oracle/flash_recovery_area/PMGDB/backupset/2014_07_03/o1_mf_annnn_TAG20140703T185830_9vbpmykl_.bkptag=TAG20140703T185830 comment=NONEchannel ORA_DISK_1: backup set complete, elapsed time: 00:00:01Finished backup at 03-JUL-14

RMAN> exitRecovery Manager complete.

To check if the backup is completed successfully, check the last statement in the output. For example, “Finishedbackup at 03-JUL-14”.

Note

Creating Standby Control File and PFILE

Step 1 Login to the sql prompt.

Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 17:51:51 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL>

Step 2 Create a control file for the standby database using the following command on the primary database.



Example:SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/tmp/pmgdb_stby.ctl';Database altered.

Step 3 Create a parameter file for the standby database.

Example:SQL> CREATE PFILE='/tmp/initPMGDB_stby.ora' FROM SPFILE;File created.

Step 4 Exit from the sql prompt.

Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production…

Step 5 Edit the PFILE generated, that is, /tmp/initPMGDB_stby.ora, making the entries relevant for the standby database.Because this is a replica of the original server, only the following parameters should be modified. Modify the parameterif it exists, otherwise add the parameter with the specified value.

Example:*.db_unique_name='PMGDB_STBY'*.fal_server='PMGDB'*.log_archive_dest_2='SERVICE=pmgdb ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)DB_UNIQUE_NAME=PMGDB'

Configuring the Hot Standby Server

Step 1 Log in to the secondary or standby server with the username “oracle”.Step 2 Add the tnsnames entries to "$ORACLE_HOME/network/admin/tnsnames.ora".

Example:PMGDB =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <primary-server-host-address>)(PORT = <port number>))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)

))

PMGDB_STBY =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <standby-server-host-address>)(PORT = <port number>))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)

))

Step 3 Edit "$ORACLE_HOME/network/admin/listener.ora" to add the entry that the Data Guard Broker will refer to.


Configuring High Availability for the PMG DBConfiguring the Hot Standby Server

Example:SID_LIST_LISTENER =(SID_LIST =(SID_DESC =(GLOBAL_NAME = PMGDB_STBY_DGMGRL)(ORACLE_HOME = <oracle-home-directory-path>)(SID_NAME = PMGDB)

))

Copying Files

Step 1 Create the necessary directories on the standby server.

Example:$ mkdir -p /u01/app/oracle/oradata/PMGDBThe system responds with a command prompt.

$ mkdir -p /u01/app/oracle/flash_recovery_area/PMGDBThe system responds with a command prompt.

$ mkdir -p /u01/app/oracle/admin/PMGDB/adumpThe system responds with a command prompt.

Step 2 Copy the files from the primary to the standby server. Copy the standby control file to all locations.

Example:$ scp oracle@<primary-server-host-address>:/tmp/pmgdb_stby.ctl/u01/app/oracle/oradata/PMGDB/control01.ctl<scp-output>

$ cp /u01/app/oracle/oradata/PMGDB/control01.ctl/u01/app/oracle/flash_recovery_area/PMGDB/control02.ctlThe system responds with a command prompt.

Step 3 Copy archivelogs and backups.

Example:$ scp -r oracle@<primary-server-host-address>:/u01/app/oracle/flash_recovery_area/PMGDB/archivelog /u01/app/oracle/flash_recovery_area/PMGDB<scp-output>

$ scp -r oracle@<primary-server-host-address>:/u01/app/oracle/flash_recovery_area/PMGDB/backupset/u01/app/oracle/flash_recovery_area/PMGDB<scp-output>

Step 4 Copy the parameter file.

Example:$ scp oracle@<primary-server-host-address>:/tmp/initPMGDB_stby.ora /tmp/initPMGDB_stby.ora<scp-output>

Step 5 Copy the remote login password file.



Example:$ scp oracle@<primary-server-host-address>:$ORACLE_HOME/dbs/orapwPMGDB $ORACLE_HOME/dbs<scp-output>

The backups are copied across to the standby server as part of the flash recovery area (FRA) copy. Ifbackups are not held within the FRA, ensure that those are copied to the standby server and make themavailable from the same path as used on the primary server.

Note

Starting the Listener on Standby ServerEnsure that the listener is started on the standby server.


$ lsnrctl status

If not running:

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 02-JUL-2014 09:35:37Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<primary-server-host-address>)(PORT=<port number>))TNS-12541: TNS:no listenerTNS-12560: TNS:protocol adapter errorTNS-00511: No listenerLinux Error: 111: Connection refused

If running:

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 02-JUL-2014 09:43:23Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<primary-server-host-address>)(PORT=<port number>)))STATUS of the LISTENER------------------------…Services Summary...……The command completed successfully





Restoring Backup

Step 1 Create the SPFILE from the modified PFILE.

Example:$ export ORACLE_SID=PMGDBThe system responds with a command prompt.

$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 12:24:01 2014

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to an idle instance.

SQL> CREATE SPFILE FROM PFILE='/tmp/initPMGDB_stby.ora';File created.


Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release11.2.0.1.0 - 64bit Production

Step 3 Restore the backup files. Depending on database size, restore time will vary.All of the output is notdisplayed.

Note

Example:$ export ORACLE_SID=PMGDB$ rman target=/Recovery Manager: Release 11.2.0.1.0 - Production on Thu Jul 3 19:23:50 2014


connected to target database (not started)

RMAN> STARTUP MOUNT;Oracle instance starteddatabase mounted

Total System Global Area 3340451840 bytes

Fixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytes

RMAN> RESTORE DATABASE;Starting restore at 03-JUL-14



Starting implicit crosscheck backup at 03-JUL-14using target database control file instead of recovery catalogallocated channel: ORA_DISK_1………channel ORA_DISK_1: piece handle=/u01/app/oracle/flash_recovery_area/PMGDB/backupset/2014_07_03/o1_mf_nnndf_TAG20140703T185812_9vbpmf2x_.bkp tag=TAG20140703T185812channel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 00:00:26Finished restore at 03-JUL-14


Creating Redo Logs

Step 1 Exit from the sqlplus prompt if already logged in and re-login.

Example:SQL> exitDisconnected




Step 2 Create online redo logs for the standby.

Example:SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=MANUAL;System altered.

SQL> ALTER DATABASE ADD LOGFILE ('/u01/app/oracle/oradata/PMGDB/online_redo01.log') SIZE 50M;Database altered.




Create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logs shouldbe at least as big as the largest online redo log and there should be one extra group per thread compared the online redo logs.In this case, the following standby redo logs must be created on both servers.SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo01.log') SIZE 50M;Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo02.log') SIZE 50M;



Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo03.log') SIZE 50M;Database altered.


Step 3 Create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logsshould be at least as big as the largest online redo log and there should be one extra group per thread compared the onlineredo logs. In this case, the following standby redo logs must be created on both servers.

Example:SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo01.log') SIZE 50M;Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo02.log') SIZE 50M;Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo03.log') SIZE50M;Database altered.


Step 4 Repeat steps 1 to 3 to create standby redo logs on the primary server. After this is complete, the apply process can bestarted.

Starting the Apply ProcessStart the apply process on the standby server.SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;Database altered.

Checking Status

Step 1 Check status on primary database.The status LOG SWITCH GAP or RESOLVABLE GAP will change after the Data Guard Broker is setup.

Note

Example:SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE,SWITCHOVER_STATUS from v$database

NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS--------- -------------------- ------------------------------ ---------------- --------------------PMGDB READ WRITE PMGDB PRIMARYLOG SWITCH GAP

Step 2 Check status on secondary database.The switchover status could be TO PRIMARYor SESSIONSACTIVATE.Note




NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS--------- -------------------- ---------------------------- ---------------

--------------------PMGDB MOUNTED PMGDB_STBYPHYSICAL STANDBY TO PRIMARY

Setting Up the Oracle Data Guard BrokerThe Oracle Data Guard Broker is used to create a broker configuration that allows the broker to manage andmonitor primary and standby databases together as an integrated unit.

Step 1 Log in to the primary server with the username “oracle”.Step 2 Check the Data Guard Broker process on the primary database.

Example:SQL> sho parameter dg_broker

NAME TYPE VALUE------------------------------------ ----------- ------------------------------dg_broker_config_file1 string /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB.dat

dg_broker_config_file2 string /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB.dat

dg_broker_start boolean FALSE

Step 3 Start the Data Guard Broker process on the primary database.

Example:SQL> alter system set dg_broker_start=true scope=both;System altered.

Step 4 Check the DG_BROKER on the standby database and start it.

Example:SQL> sho parameter dg_brokerNAME TYPE VALUE----------------- ---------- ----------dg_broker_start boolean FALSE

SQL> alter system set dg_broker_start=true scope=both ;System altered.

Step 5 Verify the "$ORACLE_HOME/network/admin/listener.ora" file.If required, edit the file that includes the db_unique_name_DGMGRL.db_domain values for the GLOBAL_DBNAMEin both primary and standby database. To set the value, check the db_domain value on the primary and standby database.


Configuring High Availability for the PMG DBSetting Up the Oracle Data Guard Broker

Example:SQL> show parameter db_domainNAME TYPE VALUE-------------- ----------- --------------db_domain stringBecause the value of the db_domain is null so is the value of GLOBAL_DBNAME = PMGDB_DGMGRL for primarydatabase and for standby GLOBAL_DBNAME = PMGDB_STBY_DGMGRL. The primary"$ORACLE_HOME/network/admin/listener.ora" file is as.

Example:SID_LIST_LISTENER =

(SID_DESC =(GLOBAL_NAME = PMGDB_DGMGRL)(ORACLE_HOME = <oracle-home-directory-path>)(SID_NAME = PMGDB)

))

Step 6 Verify the "$ORACLE_HOME/network/admin/listener.ora" file on the standby database.Step 7 Configure the Data Guard configuration. Open another console on primary or standby server to open the DGMGRLCLI.

DGMGRLCLI can be accessed from any server where Oracle Client Administrator is installed.Note

Example:$ dgmgrlDGMGRL for Linux: Version 11.2.0.1.0 - 64bit Production


Welcome to DGMGRL, type "help" for information.

DGMGRL> connect sys@PMGDBPassword:.Connected.

DGMGRL> create configuration 'dgPMGDB' as primary database is 'PMGDB' connect identifier is PMGDB ;Configuration "dgPMGDB" created with primary database " PMGDB" .

Step 8 Check the status of the configuration after the configuration is created.

Example:DGMGRL> show configurationConfiguration - dgPMGDBProtection Mode : MaxPerformanceDatabases : PMGDB - Primary databaseFast-Start Failover : DISABLEDConfiguration Status : DISABLED

Step 9 Add the standby database to the Data Guard Broker configuration.

Example:DGMGRL> add database 'PMGDB_STBY' as connect identifier is PMGDB_STBY maintained as physical ;Database " PMGDB_STBY " added

DGMGRL> show configurationConfiguration - dgPMGDBProtection Mode : MaxPerformanceDatabases : PMGDB - Primary database

: PMGDB_STBY - Physical standby database



Fast-Start Failover : DISABLEDConfiguration Status : DISABLED

Step 10 Enable the configuration.

Example:DGMGRL> enable configurationEnabled.

DGMGRL> show configurationConfiguration - dgPMGDBProtection Mode : MaxPerformanceDatabases : PMGDB - Primary database

: PMGDB_STBY - Physical standby databaseFast-Start Failover : DISABLEDConfiguration Status : SUCCESS

Step 11 View the primary and standby database properties.While copying the command from the document, the single quotes (for example, as in 'PMGDB') may not becopied correctly, the command may have to be entered at the prompt.

Note

Example:DGMGRL> show database verbose 'PMGDB'Database - PMGDBRole: PRIMARYIntended State: TRANSPORT-ONInstance(s):PMGDB

Properties:DGConnectIdentifier = 'pmgdb'ObserverConnectIdentifier = ''LogXptMode = 'ASYNC'DelayMins = '0'Binding = 'optional'MaxFailure = '0'MaxConnections = '1'ReopenSecs = '300'NetTimeout = '30'RedoCompression = 'DISABLE'LogShipping = 'ON'PreferredApplyInstance = ''ApplyInstanceTimeout = '0'ApplyParallel = 'AUTO'StandbyFileManagement = 'AUTO'ArchiveLagTarget = '0'LogArchiveMaxProcesses = '30'LogArchiveMinSucceedDest = '1'DbFileNameConvert = 'PMGDB_STBY, PMGDB'LogFileNameConvert = 'PMGDB_STBY, PMGDB'FastStartFailoverTarget = ''StatusReport = '(monitor)'InconsistentProperties = '(monitor)'InconsistentLogXptProps = '(monitor)'SendQEntries = '(monitor)'LogXptStatus = '(monitor)'RecvQEntries = '(monitor)'HostName = 'oracle-vm-primary'SidName = 'PMGDB'

StaticConnectIdentifier ='(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oracle-vm-primary)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=PMGDB_DGMGRL)(INSTANCE_NAME=PMGDB)(SERVER=DEDICATED)))'

StandbyArchiveLocation = 'USE_DB_RECOVERY_FILE_DEST'AlternateLocation = ''LogArchiveTrace = '0'LogArchiveFormat = '%t_%s_%r.arc'TopWaitEvents = '(monitor)'



Database Status:SUCCESS

DGMGRL> show database verbose 'PMGDB_STBY'Database - PMGDB_STBYRole: PHYSICAL STANDBYIntended State: APPLY-ONTransport Lag: 0 secondsApply Lag: 0 secondsReal Time Query: OFFInstance(s):PMGDB

Properties:DGConnectIdentifier = 'pmgdb_stby'ObserverConnectIdentifier = ''LogXptMode = 'ASYNC'DelayMins = '0'Binding = 'OPTIONAL'MaxFailure = '0'MaxConnections = '1'ReopenSecs = '300'NetTimeout = '30'RedoCompression = 'DISABLE'LogShipping = 'ON'PreferredApplyInstance = ''ApplyInstanceTimeout = '0'ApplyParallel = 'AUTO'StandbyFileManagement = 'AUTO'ArchiveLagTarget = '0'LogArchiveMaxProcesses = '30'LogArchiveMinSucceedDest = '1'DbFileNameConvert = 'PMGDB_STBY, PMGDB'LogFileNameConvert = 'PMGDB_STBY, PMGDB'FastStartFailoverTarget = ''StatusReport = '(monitor)'InconsistentProperties = '(monitor)'InconsistentLogXptProps = '(monitor)'SendQEntries = '(monitor)'LogXptStatus = '(monitor)'RecvQEntries = '(monitor)'HostName = 'blr-oracle2-standby'SidName = 'PMGDB'

StaticConnectIdentifier ='(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=blr-oracle2-standby)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=PMGDB_STBY_DGMGRL)(INSTANCE_NAME=PMGDB)(SERVER=DEDICATED)))'

StandbyArchiveLocation = 'USE_DB_RECOVERY_FILE_DEST'AlternateLocation = ''LogArchiveTrace = '0'LogArchiveFormat = '%t_%s_%r.arc'TopWaitEvents = '(monitor)'

Database Status:SUCCESS

Verifying Log Synchronization on Standby Server

Step 1 Identify the existing archived redo log files on the standby server. Note the latest sequence number.Sequence number and timestamp in the output will vary depending on the installationsetup.

Note


Configuring High Availability for the PMG DBVerifying Log Synchronization on Standby Server

Example:SQL> ALTER SESSION SET nls_date_format='DD-MON-YYYY HH24:MI:SS';Session altered.

SQL> SELECT sequence#, first_time, next_time, appliedFROM v$archived_logORDER BY sequence#;SEQUENCE# FIRST_TIME NEXT_TIME APPLIED---------- -------------------- -------------------- ---------

41 10-JUL-2014 09:06:27 10-JUL-2014 09:06:53 YES42 10-JUL-2014 09:06:53 10-JUL-2014 09:36:09 YES43 10-JUL-2014 09:36:09 10-JUL-2014 09:51:38 IN-MEMORY

Step 2 Force a log switch to archive the current online redo log file on the primary server.

Example:SQL> ALTER SYSTEM SWITCH LOGFILE;System altered.

Step 3 Verify on the standby, the new redo data that was archived on the standby database. That is, new sequence number withlatest timestamp of redo apply is displayed.

The value of the APPLIED column for the most recently received log file will either be IN-MEMORY or YESif that log file has been applied.

Note

Example:SQL> SELECT sequence#, first_time, next_time, appliedFROM v$archived_logORDER BY sequence#;SEQUENCE# FIRST_TIME NEXT_TIME APPLIED---------- -------------------- -------------------- ---------

41 10-JUL-2014 09:06:27 10-JUL-2014 09:06:53 YES42 10-JUL-2014 09:06:53 10-JUL-2014 09:36:09 YES43 10-JUL-2014 09:36:09 10-JUL-2014 09:51:38 YES44 10-JUL-2014 09:51:38 10-JUL-2014 10:03:48 IN-MEMORY

Enabling FlashbackFlashback database is needed to enable fast-start failover for hot standby. In case of cold standby also it isadvisable to enable Flashback database because it is helpful in recovering the database.

Step 1 Check if flashback is on from the primary database sql prompt.

Example:SQL> select flashback_on from v$database;FLASHBACK_ON------------------NO

Step 2 Set flashback "on" if it is not already on.


Configuring High Availability for the PMG DBEnabling Flashback

Example:Database altered.

Step 3 Check if the flashback is on from the standby database sql prompt.

Example:SQL> select flashback_on from v$database;FLASHBACK_ON------------------NO

Step 4 Cancel standby apply process if flashback mode has to be made on.

Example:SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;Database altered.

Step 5 Set flashback “on” if it is not already on.

Example:SQL> alter database flashback on;Database altered.

Step 6 Start back apply process.

Example:SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;Database altered.

Configuring Hot Standby for PMG DBHot standby refers to a standby server that becomes active and acts as primary when the original primaryserver fails. The failover is automatic without manual intervention and with a little or no downtime. Typicallyhot standby servers are placed on different hosts in the same site as that of the primary server. The followingprocedure describes how to add a hot standby. Skip this section if there is no need to configure a hot standby.

Setting Up the Hot Standby

Before You Begin

Ensure that you have completed the Configuring the Standby Server Setup, on page 4 to create a standbyserver.


Configuring High Availability for the PMG DBConfiguring Hot Standby for PMG DB

Enabling Failover

DETAILED STEPS

PurposeCommand or Action

To enable failover, return to the console where the DGMGRL tool is running andchange the LogXptMode property for the PMGDB.

Step 1

Example:DGMGRL> edit database "PMGDB" set property LogXptMode='SYNC';Property "logxptmode" updated

Change the LogXptMode PMGDB_STBY.Step 2

Example:DGMGRL> edit database "PMGDB_STBY" set property LogXptMode='SYNC';Property "logxptmode" updated

Check the LogXptStatus.Step 3

Example:DGMGRL> show database 'PMGDB' 'LogXptStatus';LOG TRANSPORT STATUSPRIMARY_INSTANCE_NAME STANDBY_DATABASE_NAME STATUS

PMGDB PMGDB_STBY

Set the protection mode to MaxAvailabilityStep 4

Example:DGMGRL> edit configuration set protection mode as MaxAvailability;Succeeded.

Return to the DGMGRL console and enable failover. If not connected, connect assys@PMGDB.

Step 5

Example:DGMGRL> enable fast_start failover;Enabled

It is recommendedrunning the Observer ona server host that isseparate from the primaryand standby servers. Thishost needs Oracle ClientAdministrator installed torun DGMGRL tool andthe Observer.

Note

Otherwise, run the Observer onthe standby server. In case of

Open another console for the standby server and login as an oracle user. Start theObserver as a background process. The Observer monitors both primary and standbydatabases and detects failures, if any.

Example:$ nohup dgmgrl -silent sys/<password>@PMGDB "start observer" &$ cat nohup.outObserver started

Step 6

failure, if primary and standbyroles are switched, ensure that theobserver is running on the newstandby server.


Configuring High Availability for the PMG DBSetting Up the Hot Standby


Return to the previous DGMGRL console and verify the configuration.Step 7

Example:DGMGRL> show configuration verboseConfiguration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databaseWarning: ORA-16826: apply service state is inconsistent with the

DelayMins property

(*) Fast-Start Failover target

Fast-Start Failover: ENABLEDThreshold: 30 secondsTarget: PMGDB_STBYObserver: oracle-vm-primaryLag Limit: 30 seconds (not in use)Shutdown Primary: TRUEAuto-reinstate: TRUE

Configuration Status:WARNING

Start recovery from the current log file on the standby server if the above error(mentioned in the previous step) is observed.

Step 8

Example:SQL> alter database recover managed standby database cancel;Database altered.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENTLOGFILE DISCONNECT FROM SESSION;Database altered.

Return to the DGMGRL console to check the configuration again.Step 9

Example:DGMGRL> show configuration verboseConfiguration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby database

(*) Fast-Start Failover target

Fast-Start Failover: ENABLED

Threshold: 30 secondsTarget: PMGDB_STBYObserver: oracle-vm-primaryLag Limit: 30 seconds (not in use)Shutdown Primary: TRUEAuto-reinstate: TRUE

Configuration Status:SUCCESS



Checking Status

Step 1 Check status on primary database.The status LOG SWITCH GAP or RESOLVABLE GAP will change after the Data Guard Broker is setup.

Note


NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS--------- -------------------- ------------------------------ ---------------- --------------------PMGDB READ WRITE PMGDB PRIMARYLOG SWITCH GAP

Step 2 Check status on secondary database.The switchover status could be TO PRIMARYor SESSIONSACTIVATE.Note


NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS--------- -------------------- ---------------------------- ---------------

--------------------PMGDB MOUNTED PMGDB_STBYPHYSICAL STANDBY TO PRIMARY

Initializing Parameters for Standby Server

Step 1 Add db_unique_name to log_archive_config

Example:SQL> ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(PMGDB,PMGDB_STBY,PMGDB_STBY2)';System altered.

Step 2 Set and enable LOG_ARCHIVE_DEST_3.

Example:SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_3='SERVICE=PMGDB_stby2 NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PMGDB_STBY2';System altered.

Step 3 Set the parameters to ensure that the primary is ready to switch roles to become a standby.

Example:SQL> ALTER SYSTEM SET FAL_SERVER=PMGDB_STBY,PMGDB_STBY2;System altered.

SQL> ALTER SYSTEM SET DB_FILE_NAME_CONVERT='PMGDB','PMGDB_STBY', 'PMGDB', 'PMGDB_STBY2' SCOPE=SPFILE;System altered.



SQL> ALTER SYSTEM SET LOG_FILE_NAME_CONVERT ='PMGDB','PMGDB_STBY', 'PMGDB', 'PMGDB_STBY2' SCOPE=SPFILE;System altered.

Step 4 Restart the database to implement the modifications made to the parameters.


SQL> startupORACLE instance started.Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.Database opened.

Setting Up the Service for Standby ServerProvide the values for the new standby databases in the "$ORACLE_HOME/network/admin/tnsnames.ora"files on both the servers. The following values are used during the setup.PMGDB_STBY2 =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <standby2-server-host-address>)(PORT = <port number>))

(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)

))

Backing Up the Primary Database

Step 1 Back up the primary database, which will be restored on the standby database. To do this, open another console for theprimary database server and log in as an oracle user.

Step 2 Set the ORACLE_SID to PMGDB and take a backup.The complete output is notdisplayed.

Note

Example:$ export ORACLE_SID=PMGDB$ rman target=/

Recovery Manager: Release 11.2.0.1.0 - Production on Thu Jul 3 18:53:35 2014


connected to target database: PMGDB (DBID=122120137)



RMAN> BACKUP DATABASE PLUS ARCHIVELOG;Starting backup at 03-JUL-14current log archivedusing target database control file instead of recovery catalogallocated channel: ORA_DISK_1channel ORA_DISK_1: SID=97 device type=DISKchannel ORA_DISK_1: starting archived log backup setchannel ORA_DISK_1: specifying archived log(s) in backup setinput archived log thread=1 sequence=15 RECID=93 STAMP=851365692input archived log thread=1 sequence=16 RECID=95 STAMP=851367216………………channel ORA_DISK_1: starting full datafile backup setchannel ORA_DISK_1: specifying datafile(s) in backup setinput datafile file number=00005 name=/u01/app/oracle/oradata/PMGDB/mapinfo_ts.dbfinput datafile file number=00006 name=/u01/app/oracle/oradata/PMGDB/pmgdb_ts.dbfinput datafile file number=00001 name=/u01/app/oracle/oradata/PMGDB/system01.dbfinput datafile file number=00002 name=/u01/app/oracle/oradata/PMGDB/sysaux01.dbfinput datafile file number=00003 name=/u01/app/oracle/oradata/PMGDB/undotbs01.dbfinput datafile file number=00004 name=/u01/app/oracle/oradata/PMGDB/users01.dbfpiecehandle=/u01/app/oracle/flash_recovery_area/PMGDB/backupset/2014_07_03/o1_mf_annnn_TAG20140703T185830_9vbpmykl_.bkptag=TAG20140703T185830 comment=NONEchannel ORA_DISK_1: backup set complete, elapsed time: 00:00:01Finished backup at 03-JUL-14


To check if the backup is completed successfully, check the last statement in the output. For example, “Finishedbackup at 03-JUL-14”.

Note

Creating the Standby Control File and PFILE for Standby Server

Step 1 Return to the sqlplus prompt of the primary database and create a control file for the standby database using the followingcommand on the primary database.

Example:SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/tmp/pmgdb_stby2.ctl';Database altered.

Step 2 Create a parameter file for the standby database.

Example:SQL> CREATE PFILE='/tmp/initPMGDB_stby2.ora' FROM SPFILE;File created.

Step 3 Modify the PFILE “/tmp/initPMGDB_stby2.ora” making the entries relevant for the standby database. Because this is areplica of the original server, only the following parameters need to be modified, added, or updated. Modify the parameterif it exists otherwise add the parameter with specified value.



Example:*.db_unique_name='PMGDB_STBY2'*.fal_server='PMGDB'*.log_archive_dest_2='SERVICE=PMGDB ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)DB_UNIQUE_NAME=PMGDB'*.log_archive_config='DG_CONFIG=(PMGDB,PMGDB_STBY2)'*.db_file_name_convert='PMGDB_STBY2','PMGDB'*.log_file_name_convert='PMGDB_STBY2','PMGDB'Also, remove the following parameters:

Example:*.log_archive_dest_3='ENABLE'*.log_archive_dest_state_3='ENABLE'

Configuring the Standby Server

Step 1 Log in to the standby 2 server (to be used as cold standby) with the username “oracle”.Step 2 Add tnsnames entries to "$ORACLE_HOME/network/admin/tnsnames.ora".

Example:PMGDB =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <primary-server-host-address>)(PORT = <port number>))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)

))

PMGDB_STBY2 =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = <standby2-server-host-address>)(PORT = <port number>))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = PMGDB)

))

Step 3 Edit "$ORACLE_HOME/network/admin/listener.ora” to add a entry that the Data Guard Broker will refer to.

Example:SID_LIST_LISTENER =

(SID_DESC =(GLOBAL_NAME = PMGDB_STBY2_DGMGRL)(ORACLE_HOME = <oracle-home-directory-path>)(SID_NAME = PMGDB)

))



Copying Files to the Standby Server

Step 1 Create the necessary directories on the standby server.

Example:$ mkdir -p /u01/app/oracle/oradata/PMGDBThe system responds with a command prompt.

$ mkdir -p /u01/app/oracle/flash_recovery_area/PMGDBThe system responds with a command prompt.

$ mkdir -p /u01/app/oracle/admin/PMGDB/adumpThe system responds with a command prompt.

Step 2 Copy the files from the primary to the standby server. Copy the standby control file to all locations.

Example:$ scp oracle@<primary-server-host-address>:/tmp/pmgdb_stby.ctl/u01/app/oracle/oradata/PMGDB/control01.ctl<scp-output>

$ cp /u01/app/oracle/oradata/PMGDB/control01.ctl/u01/app/oracle/flash_recovery_area/PMGDB/control02.ctlThe system responds with a command prompt.

Step 3 Copy archivelogs and backups.

Example:$ scp -r oracle@<primary-server-host-address>:/u01/app/oracle/flash_recovery_area/PMGDB/archivelog /u01/app/oracle/flash_recovery_area/PMGDB<scp-output>

$ scp -r oracle@<primary-server-host-address>:/u01/app/oracle/flash_recovery_area/PMGDB/backupset/u01/app/oracle/flash_recovery_area/PMGDB<scp-output>

Step 4 Copy the parameter file.

Example:$ scp oracle@<primary-server-host-address>:/tmp/initPMGDB_stby.ora /tmp/initPMGDB_stby.ora<scp-output>

Step 5 Copy the remote login password file.

Example:$ scp oracle@<primary-server-host-address>:$ORACLE_HOME/dbs/orapwPMGDB $ORACLE_HOME/dbs<scp-output>

Starting the Listener on Standby ServerEnsure that the listener is started on the standby server.




$ lsnrctl status

If not running:

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 02-JUL-2014 09:35:37Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<primary-server-host-address>)(PORT=<port number>))TNS-12541: TNS:no listenerTNS-12560: TNS:protocol adapter errorTNS-00511: No listenerLinux Error: 111: Connection refused

If running:

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 02-JUL-2014 09:43:23Copyright (c) 1991, 2009, Oracle. All rights reserved.Connecting to(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=<primary-server-host-address>)(PORT=<port number>)))STATUS of the LISTENER------------------------…Services Summary...……The command completed successfully



Restoring Backup on the Standby Server

Step 1 Create the SPFILE from the modified PFILE.

Example:$ export ORACLE_SID=PMGDBThe system responds with a command prompt.






SQL> CREATE SPFILE FROM PFILE='/tmp/initPMGDB_stby.ora';File created.


Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release11.2.0.1.0 - 64bit Production

Step 3 Restore the backup files. Depending on database size, restore time will vary.All of the output is notdisplayed.

Note

Example:$ export ORACLE_SID=PMGDB$ rman target=/Recovery Manager: Release 11.2.0.1.0 - Production on Thu Jul 3 19:23:50 2014



RMAN> STARTUP MOUNT;Oracle instance starteddatabase mounted

Total System Global Area 3340451840 bytes

Fixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytes

RMAN> RESTORE DATABASE;Starting restore at 03-JUL-14Starting implicit crosscheck backup at 03-JUL-14using target database control file instead of recovery catalogallocated channel: ORA_DISK_1………channel ORA_DISK_1: piece handle=/u01/app/oracle/flash_recovery_area/PMGDB/backupset/2014_07_03/o1_mf_nnndf_TAG20140703T185812_9vbpmf2x_.bkp tag=TAG20140703T185812channel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 00:00:26Finished restore at 03-JUL-14




Creating Redo Logs for Standby Server

Step 1 Exit from the sqlplus prompt if already logged in and re-login.

Example:SQL> exitDisconnected




Step 2 Create online redo logs for the standby.

Example:SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=MANUAL;System altered.





Create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logs shouldbe at least as big as the largest online redo log and there should be one extra group per thread compared the online redo logs.In this case, the following standby redo logs must be created on both servers.SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo01.log') SIZE 50M;Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo02.log') SIZE 50M;Database altered.



Step 3 Create standby redo logs on both the standby and the primary database (in case of switchovers). The standby redo logsshould be at least as big as the largest online redo log and there should be one extra group per thread compared the onlineredo logs. In this case, the following standby redo logs must be created on both servers.

Example:SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo01.log') SIZE 50M;Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE('/u01/app/oracle/oradata/PMGDB/standby_redo02.log') SIZE 50M;



Database altered.



Step 4 Repeat steps 1 to 3 to create standby redo logs on the primary server. After this is complete, the apply process can bestarted.

Starting the Apply Process on Standby ServerStart the apply process on the standby server.SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;Database altered.

Configuring the Data Guard Broker on Standby Server

Step 1 Start the Data Guard Broker Process on the standby server.

Example:SQL> alter system set dg_broker_start=true scope=both;System altered.

Step 2 Configure the Data Guard Broker by opening another console on the primary or standby server to invoke the DGMGRLCLI.

DGMGRLCLI can be accessed from any server where the Oracle Client Administrator is installed.Note

Example:$ dgmgrlDGMGRL for Linux: Version 11.2.0.1.0 - 64bit ProductionCopyright (c) 2000, 2009, Oracle. All rights reserved.Welcome to DGMGRL, type "help" for information.


Step 3 Add the standby database to the Data Guard Broker configuration.DGMGRL> add database 'PMGDB_STBY2' as connect identifier is PMGDB_STBY2 maintainedas physical ;Database " PMGDB_STBY2 " added

Step 4 Update the primary database property and enable the database.

Example:DGMGRL> edit database 'PMGDB' set property DbFileNameConvert='PMGDB,PMGDB_STBY,PMGDB,PMGDB_STBY2';Property "dbfilenameconvert" updated



DGMGRL> edit database 'PMGDB' set property LogFileNameConvert='PMGDB,PMGDB_STBY,PMGDB,PMGDB_STBY2';Property "logfilenameconvert" updated

DGMGRL> enable database 'PMGDB';Enabled.

Step 5 Enable the standby database.

Example:DGMGRL> enable database 'PMGDB_STBY2';Enabled.

Step 6 Verify the configuration.

Example:DGMGRL> show configurationConfiguration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databasePMGDB_STBY2 - Physical standby database

Fast-Start Failover: ENABLEDConfiguration Status:SUCCESS

Verifying Log Synchronization on Standby Server

Step 1 Identify the existing archived redo log files on the standby server. Note the latest sequence number.Sequence number and timestamp in the output will vary depending on the installationsetup.

Note

Example:SQL> ALTER SESSION SET nls_date_format='DD-MON-YYYY HH24:MI:SS';Session altered.

SQL> SELECT sequence#, first_time, next_time, appliedFROM v$archived_logORDER BY sequence#;SEQUENCE# FIRST_TIME NEXT_TIME APPLIED---------- -------------------- -------------------- ---------

41 10-JUL-2014 09:06:27 10-JUL-2014 09:06:53 YES42 10-JUL-2014 09:06:53 10-JUL-2014 09:36:09 YES43 10-JUL-2014 09:36:09 10-JUL-2014 09:51:38 IN-MEMORY

Step 2 Force a log switch to archive the current online redo log file on the primary server.

Example:SQL> ALTER SYSTEM SWITCH LOGFILE;System altered.

Step 3 Verify on the standby, the new redo data that was archived on the standby database. That is, new sequence number withlatest timestamp of redo apply is displayed.



The value of the APPLIED column for the most recently received log file will either be IN-MEMORY or YESif that log file has been applied.

Note

Example:SQL> SELECT sequence#, first_time, next_time, appliedFROM v$archived_logORDER BY sequence#;SEQUENCE# FIRST_TIME NEXT_TIME APPLIED---------- -------------------- -------------------- ---------

41 10-JUL-2014 09:06:27 10-JUL-2014 09:06:53 YES42 10-JUL-2014 09:06:53 10-JUL-2014 09:36:09 YES43 10-JUL-2014 09:36:09 10-JUL-2014 09:51:38 YES44 10-JUL-2014 09:51:38 10-JUL-2014 10:03:48 IN-MEMORY

Enabling Flashback on Standby Server

Follow the steps in Enabling Flashback, on page 20 to enable flashback. Read the standby server as coldstandby server.

Configuring Cold StandbyCold standby refers to a standby server that is made active and switched as a primary when the original primaryserver or site fails due to any unforeseen event. The switchover needs manual intervention and there is somedowntime involved. Typically cold standby servers are placed in a different site other than the primary serversite. There are two options available to configure cold standby:

• Configuring Primary With Only Cold Standby, on page 34

• Configuring Primary With Hot and Cold Standby, on page 34

Configuring Primary With Only Cold StandbyEnsure that you complete the Configuring the Standby Server Setup, on page 4 to create a standby server.No additional steps are required to configure for primary with only cold standby setup.

Configuring Primary With Hot and Cold Standby

Setting up Hot Standby

Complete the Configuring Hot Standby for PMG DB, on page 21 to set up the hot standby server.

Setting Up Additional Standby Server as Cold Standby

This will be an additional standby database added to the primary and should be used as a cold standby serverfor disaster recovery such as site failure recovery. Following configuration steps mentioned assume hot standbyis already created.


Configuring High Availability for the PMG DBConfiguring Cold Standby

Testing Hot StandbyYou can test the following two hot standby processes for the database:

• Testing Failover Process, on page 35

• Testing Switchover Process, on page 42

Testing Failover ProcessTo test the following failover processes, follow these procedures.

• Testing Failover From Primary Database to Standby Database, on page 35

• Testing Failover Revert From New Primary to Original Primary Database, on page 39

Testing Failover From Primary Database to Standby Database

Step 1 Log in to the primary server with the username “oracle”. If you are already logged in, proceed to Step 2.Step 2 Shutdown the primary database and check if standby is implemented.

Example:SQL> select db_unique_name from v$database;DB_UNIQUE_NAME------------------------------PMGDB

SQL> shut abortORACLE instance shut down.

Step 3 Check alert logs on the standby server.TheOracle base directory pathmay vary based on your installation.Note

Example:$ cd /u01/app/oracle/diag/rdbms/pmgdb_stby/PMGDB/trace$ tail -f alert_PMGDB.log -n100…Failover succeeded. Primary database is now PMGDB_STBY.……

Step 4 Check the log status.

Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE FS_FAILOVER_STATUS

FSFO_CURR_TARGET------------------------------ ---------------- ----------- -------------------- ----------------------------------------------------


Configuring High Availability for the PMG DBTesting Hot Standby

PMGDB_STBY PRIMARY 1204588 READ WRITE REINSTATE REQUIREDPMGDB

Step 5 Synchronize failure status by starting up and mounting the database on the original primary server.

Example:SQL> startup mountORACLE instance started.


Step 6 Open a new console on the original primary server, and keep tailing the log file drcPMGDB.log. Wait until the tail outputstops at "DMON: >> DMON Process Shutdown <<".

Example:$cd /u01/app/oracle/diag/rdbms/pmgdb/PMGDB/trace$tail -f drcPMGDB.log<tail output>……014-07-03 21:41:43.825 01001000 787692441 DMON: status from posting instances for Database QUIESCE= ORA-000002014-07-03 21:41:43.825 INSV: Received message for inter-instance publication2014-07-03 21:41:43.825 req ID 1.1.787692441, opcode CTL_QUIESCE, phaseTEARDOWN, flags 10052014-07-03 21:41:43.825 DMON: Releasing (convert to NULL) Health Check Masterlock2014-07-03 21:41:43.825 DMON: Releasing FSFP HOME lock2014-07-03 21:41:43.826 INSV: Reply received for message with2014-07-03 21:41:43.826 req ID 1.1.787692441, opcode CTL_QUIESCE, phaseTEARDOWN2014-07-03 21:41:43.826 01001000 787692441 DMON: Entered rfm_release_chief_lock() for CTL_QUIESCE2014-07-03 21:41:45.827 Fore: FSFO shutting down2014-07-03 21:41:49.879 DMON: Data Guard Broker shutting down2014-07-03 21:41:49.879 DMON: Terminating RSM processes2014-07-03 21:41:49.880 RSM0: delete state object for RSM02014-07-03 21:41:50.879 DMON: RSM0 successfully terminated2014-07-03 21:41:50.880 DMON: Terminating NetSlave processes2014-07-03 21:41:50.880 DMON: Freeing all task elements.2014-07-03 21:41:50.880 DMON: Terminating Instance Slave process2014-07-03 21:41:50.880 INSV: Shutting down2014-07-03 21:41:51.880 DMON: INSV successfully terminated2014-07-03 21:41:51.881 DMON: Zeroing metadata root pointer2014-07-03 21:41:51.881 DMON: Clearing Primary State2014-07-03 21:41:51.881 DMON: Freeing Broker SGA heap2014-07-03 21:41:51.881 DMON: Freeing PGA heap2014-07-03 21:41:51.881 DMON: Removing DMON's state object2014-07-03 21:41:51.881 DMON: Resetting DMON context structure2014-07-03 21:41:51.881 DMON: >> DMON Process Shutdown <<

Step 7 Return to the sql prompt of the original primary server, exit the sql prompt, relogin to the sql prompt, and redo the startupmount.

Example:SQL> exitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing options

$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Thu Jul 3 21:57:39 2014



Configuring High Availability for the PMG DBTesting Failover Process


SQL> startup mountORACLE instance started.


Step 8 Connect the DGMGRL to PMGDB_STBY, which is the primary now and reinstate the failover status.

Example:DGMGRL> connect sys@PMGDB_STBYPassword:Connected

Step 9 View the configuration. Note that the database roles are changed, however, a warning to reinstate standby is displayed.

Example:DGMGRL> show configurationIn case of primary with only hot standby:Configuration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB_STBY - Primary databaseWarning: ORA-16817: unsynchronized fast-start failover configuration

PMGDB - (*) Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated



In case of primary with hot and cold standby:Configuration - dgPMGDB

Protection Mode: MaxAvailabilityDatabases:PMGDB_STBY - Primary databaseWarning: ORA-16817: unsynchronized fast-start failover configuration

PMGDB - (*) Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated

PMGDB_STBY2 - Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated



Step 10 Reinstate the standby database.If Error: ORA-16653: failed to reinstate database is observed in the output, go to the original primary database,exit the sql prompt, re-login to the sql prompt, startup, and mount the database again.

Note

Example:DGMGRL> REINSTATE DATABASE 'PMGDB'Reinstating database "PMGDB", please wait...Reinstatement of database "PMGDB" succeeded

Step 11 View the configuration again to check if the configuration status is a "SUCCESS" and with no errors.



In case of primary with hot and cold standby, the error or warning of cold standby (PMGDB_STBY2) can beignored.

Note

Example:DGMGRL> show configurationIn case of primary with only hot standby:Configuration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB_STBY - Primary databasePMGDB - (*) Physical standby database



Protection Mode: MaxAvailabilityDatabases:PMGDB_STBY - Primary databasePMGDB - (*) Physical standby databasePMGDB_STBY2 - Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated



Step 12 Check the status on the original primary server.


FSFO_CURR_TARGET------------------------------ ---------------- ----------- -------------------- ----------------------------------------------------PMGDB PHYSICAL STANDBY 1205137 MOUNTED SYNCHRONIZED

PMGDB

Step 13 Check the status on the new primary server.


FSFO_CURR_TARGET------------------------------ ---------------- ----------- -------------------- ----------------------------------------------------PMGDB_STBY PRIMARY 1206642 READ WRITE SYNCHRONIZED

PMGDB



Testing Failover Revert From New Primary to Original Primary Database

Step 1 Log in to the primary server with the username “oracle”. If you are already logged in, proceed to Step 2Step 2 Shutdown the new primary (original hot standby) database and check if failover reverts to the original primary database.

Example:SQL> select db_unique_name from v$database;DB_UNIQUE_NAME------------------------------PMGDB_STBY

SQL> shut abortORACLE instance shut down

Step 3 Check the alert logs on the original primary server.TheOracle base directory pathmay vary based on your installation.Note

Example:$ cd /u01/app/oracle/diag/rdbms/pmgdb/PMGDB/trace$ tail -f alert_PMGDB.log -n100…Failover succeeded. Primary database is now PMGDB.……

Step 4 Check the log status on the original primary server.


FSFO_CURR_TARGET------------------------------ ---------------- ----------- -------------------- ----------------------------------------------------PMGDB PRIMARY 1204588 READ WRITEREINSTATE REQUIRED PMGDB_STBY

Step 5 Synchronize the failover status on the original hot standby server, start up, and mount the database.

Example:SQL> startup mountORACLE instance started.


Step 6 Open a new console on the original hot standby server and keep tailing the log file drcPMGDB.log. Wait until the tailoutput stops at "DMON: >> DMON Process Shutdown <<".

TheOracle base directory pathmay vary based on your installation.Note

Example:$cd /u01/app/oracle/diag/rdbms/pmgdb_stby/PMGDB/trace$tail -f drcPMGDB.log



<tail output>…… 2014-07-03 22:27:54.467 02001000 354428689 DMON: status from posting instances for DatabaseQUIESCE = ORA-000002014-07-03 22:27:54.467 DMON: Releasing (convert to NULL) Health Check Masterlock2014-07-03 22:27:54.467 DMON: Releasing FSFP HOME lock2014-07-03 22:27:54.467 INSV: Reply received for message with2014-07-03 22:27:54.468 req ID 2.1.354428689, opcode CTL_QUIESCE, phaseTEARDOWN2014-07-03 22:27:54.469 02001000 354428689 DMON: Entered rfm_release_chief_lock() for CTL_QUIESCE2014-07-03 22:27:56.485 Fore: FSFO shutting down2014-07-03 22:27:59.600 DMON: Data Guard Broker shutting down2014-07-03 22:27:59.601 DMON: Terminating RSM processes2014-07-03 22:27:59.601 RSM0: delete state object for RSM02014-07-03 22:28:00.601 DMON: RSM0 successfully terminated2014-07-03 22:28:00.601 DMON: Terminating NetSlave processes2014-07-03 22:28:00.601 DMON: Freeing all task elements.2014-07-03 22:28:00.601 DMON: Terminating Instance Slave process2014-07-03 22:28:00.602 INSV: Shutting down2014-07-03 22:28:01.602 DMON: INSV successfully terminated2014-07-03 22:28:01.602 DMON: Zeroing metadata root pointer2014-07-03 22:28:01.602 DMON: Clearing Primary State2014-07-03 22:28:01.602 DMON: Freeing Broker SGA heap2014-07-03 22:28:01.603 DMON: Freeing PGA heap2014-07-03 22:28:01.603 DMON: Removing DMON's state object2014-07-03 22:28:01.603 DMON: Resetting DMON context structure2014-07-03 22:28:01.603 DMON: >> DMON Process Shutdown <<

Step 7 Return to the sql prompt of the original hot standby server, exit the sql prompt, relogin to the sql prompt, and redo thestartup mount.


$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Thu Jul 3 21:57:39 2014





Step 8 Connect the DGMGRL to the PMGDB, which is the primary now and reinstate the failover status.

Example:DGMGRL> connect sys@PMGDBPassword:Connected.

Step 9 View the configuration. Note that the database roles are changed, however, a warning to reinstate standby is displayed.

Example:DGMGRL> show configuration

In case of primary with only hot standby:Configuration - dgPMGDB



Protection Mode: MaxAvailabilityDatabases:PMGDB - Primary databaseWarning: ORA-16817: unsynchronized fast-start failover configuration

PMGDB_STBY - (*) Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated




Protection Mode: MaxAvailabilityDatabases:PMGDB - Primary databaseWarning: ORA-16817: unsynchronized fast-start failover configuration

PMGDB_STBY - (*) Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated




Step 10 Reinstate the standby database.If Error: ORA-16653: failed to reinstate database is observed in the output, go to the original hot standbydatabase, exit the sql prompt, relogin to the sql prompt, start up, and mount the database again.

Note

Example:DGMGRL> REINSTATE DATABASE 'PMGDB_STBY'Reinstating database "PMGDB_STBY", please wait...Reinstatement of database "PMGDB_STBY" succeeded

Step 11 View the configuration again to check if the configuration status is a "SUCCESS" and with no errors.Note • Sometimes processes from the previous commandmay not be complete, therefore the “show configuration”

may show a different output. Allow a couple of minutes and execute the command again.

• In case of primary with hot and cold standby, an error or warning of cold standby (PMGDB_STBY2) canbe ignored.

Example:DGMGRL> show configuration

In case of primary with only hot standby:Configuration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby database



Protection Mode: MaxAvailabilityDatabases:



PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databasePMGDB_STBY2 - Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated



Step 12 Check the status on the original hot standby server.


FSFO_CURR_TARGET------------------------------ ---------------- ----------- -------------------- ----------------------------------------------------PMGDB_STBY PHYSICAL STANDBY 1205137 MOUNTED SYNCHRONIZED

PMGDB_STBY

Step 13 Check the status on the original primary server.


FSFO_CURR_TARGET------------------------------ ---------------- ----------- -------------------- ----------------------------------------------------PMGDB PRIMARY 1206642 READ WRITESYNCHRONIZED PMGDB_STBY

Testing Switchover ProcessSwitchover from the primary to secondary or standby database and vice versa is a manual process typicallycarried out during plannedmaintenance or disaster recovery. The following procedures describe the switchoverfrom PMGDB (primary) to PMGDB_STBY (hot standby).

• Testing Switchover From Primary to Standby Database, on page 42

• Testing Switchover Revert From New Primary to Original Primary Database, on page 44

Testing Switchover From Primary to Standby Database

Step 1 Start the DGMGRL utility from the primary or standby server (whichever is up) or from another host where the Oracleclient is installed.



Configuring High Availability for the PMG DBTesting Switchover Process

Copyright (c) 2000, 2009, Oracle. All rights reserved.Welcome to DGMGRL, type "help" for information.

Step 2 Connect to the PMG DB.


Step 3 Switchover to PMGDB_STBY.

Example:DGMGRL> SWITCHOVER TO 'PMGDB_STBY'Performing switchover NOW, please wait...New primary database "PMGDB_STBY" is opening...Operation requires shutdown of instance "PMGDB" on database "PMGDB"Shutting down instance "PMGDB"...ORA-01109: database not open

Database dismounted.ORACLE instance shut down.Operation requires startup of instance "PMGDB" on database "PMGDB"Starting instance "PMGDB"...Unable to connect to databaseORA-12514: TNS:listener does not currently know of service requested in connect descriptorFailed.Warning: You are no longer connected to ORACLE.

Please complete the following steps to finish switchover:start up and mount instance "PMGDB" of database "PMGDB"

Step 4 Exit from the previous sql prompt to start up and mount the PMGDB.


Step 5 Connect to sqlplus.

Example:$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 16:50:03 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to an idle instance.

Step 6 Start up and mount the database.

Example:SQL> startup mountORACLE instance started.Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.

Step 7 Verify that the roles are switched.a) Verify on the original primary (PMGDB).

Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;



DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE------------------------------ ---------------- ----------- --------------------FS_FAILOVER_STATUS FSFO_CURR_TARGET---------------------- ------------------------------PMGDB PHYSICAL STANDBY 1208044 MOUNTEDSYNCHRONIZED PMGDB

b) Verify on the new primary (PMGDB_STBY).


FSFO_CURR_TARGET------------------------------ ---------------- ----------- ------------------------------------------ ------------------------------PMGDB_STBY PRIMARY 1222493 READ WRITE SYNCHRONIZED

PMGDB

Testing Switchover Revert From New Primary to Original Primary Database

Before You Begin

Step 1 Start the DGMGRL utility from the primary or standby server or from another host where the Oracle client is installed.

Example:$ dgmgrlDGMGRL for Linux: Version 11.2.0.1.0 - 64bit ProductionCopyright (c) 2000, 2009, Oracle. All rights reserved.Welcome to DGMGRL, type "help" for information.

Step 2 Connect to PMGDB_STBY.

Example:DGMGRL> connect sys@PMGDB_STBYPassword:Connected.

Step 3 Switchover to PMGDB.

Example:DGMGRL> SWITCHOVER TO 'PMGDB'Performing switchover NOW, please wait...New primary database "PMGDB" is opening...Operation requires shutdown of instance "PMGDB" on database "PMGDB_STBY"Shutting down instance "PMGDB"...ORA-01109: database not open

Database dismounted.ORACLE instance shut down.Operation requires startup of instance "PMGDB" on database "PMGDB_STBY"Starting instance "PMGDB"...Unable to connect to databaseORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Failed.Warning: You are no longer connected to ORACLE.



Please complete the following steps to finish switchover:start up and mount instance "PMGDB" of database "PMGDB_STBY"

Step 4 Exit from previous sql prompt to start up and mount the PMGDB_STBY.


Step 5 Connect to sqlplus.

Example:$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 16:50:03 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to an idle instance.

Step 6 Start up and mount the database.

Example:SQL> startup mountORACLE instance started.Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.

Step 7 Verify that the roles are switched.a) Verify on the hot standby (PMGDB_STBY).

Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE------------------------------ ---------------- ----------- --------------------FS_FAILOVER_STATUS FSFO_CURR_TARGET---------------------- ------------------------------PMGDB_STBY PHYSICAL STANDBY 1938653 MOUNTEDSYNCHRONIZED PMGDB_STBY

b) Verify on the original primary (PMGDB).

Example:SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,CURRENT_SCN,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET FSFO_CURR_TARGET from v$database;DB_UNIQUE_NAME DATABASE_ROLE CURRENT_SCN OPEN_MODE------------------------------ ---------------- ----------- --------------------FS_FAILOVER_STATUS FSFO_CURR_TARGET---------------------- ------------------------------PMGDB PRIMARY 1938910 READ WRITESYNCHRONIZED PMGDB_STBY

Testing Cold StandbyUse these procedures to test cold standby on PMG DB.


Configuring High Availability for the PMG DBTesting Cold Standby

• Testing Site Failure, on page 46

• Recovering Original Primary After Site Failure, on page 48

Testing Site FailureUse the following steps to test unscheduled failure of primary and first hot standby (if available) servers. Toprevent applications (PMG, OpsTools) connection failures without any manual intervention, ensure that thePMGDB configuration script (/rms/app/rms/install/pmgdb_configure.sh) for the Central node is executedwith the standby server configuration as per the instructions in the installation guide.Usage:

pmgdb_configure.sh <Pmgdb_Enabled> <Pmgdb_Primary_Dbserver_Address><Pmgdb_Primary_Dbserver_Port> [<Pmgdb_Stby1_Dbserver_Address>][<Pmgdb_Stby1_Dbserver_Port>] [<Pmgdb_Stby2_Dbserver_Address>][<Pmgdb_Stby2_Dbserver_Port>]If not configured earlier, execute the script again with the appropriate values for database servers and ports.For more details, see the Cisco RAN Management System Installation Guide.

Step 1 To test site failure, verify if the applications connecting to the PMGDB database are able to connect. For example, theOpsTools script getAreas.sh can be executed. Log in to Central node as admin1 user and execute the script.

Example:$ getAreas.sh -key 1001Config files script-props/private/GetAreas.properties or script-props/public/GetAreas.properties notfound.Continuing with default settings.Execution parameters:

key=1001GetAreas processing can take some time please do not terminate.Received areas, total areas 1Writing to file: /home/admin1/getAreas.csvThe report captured in csv file: /home/admin1/getAreas.csv**** GetAreas End Script ***

Step 2 Log in to the hot standby server as an oracle user and shut down the database server.Perform this step only if the hot standby server is configured. Else, proceed to the nextstep.

Note

Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL> shutdown immediateDatabase closed.Database dismounted.ORACLE instance shut down.

Step 3 Log in to the primary server an oracle user and shut down the database server.

Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL> shutdown immediateDatabase closed.


Configuring High Availability for the PMG DBTesting Site Failure

Database dismounted.ORACLE instance shut down.

Step 4 Verify that the applications connecting to the PMGDB database are failing considering that the primary and hot standbyare down. Log in to the Central node as admin1 user and execute the getAreas.sh script.

Example:$ getAreas.sh -key 1001Config files script-props/private/GetAreas.properties orscript-props/public/GetAreas.properties not found. Continuing with default settings.Execution parameters:

key=1001GetAreas processing can take some time please do not terminate.Jul 01, 2014 1:45:14 PM org.apache.tomcat.jdbc.pool.ConnectionPool initSEVERE: Unable to create initial connections of pool.....Failed to Get Areas : Error while connecting to PmgDb, java.sql.SQLException: Listener refused theconnection with the following error:ORA-12514, TNS:listener does not currently know of service requested in connect descriptor....Alternate Output:……Failed to Get Areas : Error while connecting to PmgDb, java.sql.SQLRecoverableException: Io exception:

The Network Adapter could not establish the connection

Step 5 Log in to the cold standby server as an oracle user and log in to sql prompt.

Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaConnected to database

Step 6 Stop the redo apply process.

Example:SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;Database altered.

Step 7 Finish applying all received redo data.

Example:SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH;Database altered.

SQL> ALTER DATABASE ACTIVATE PHYSICAL STANDBY DATABASE;Database altered.

Step 8 Switch the physical standby database to the primary role.

Example:SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN;ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN*ERROR at line 1:ORA-01109: database not open

Step 9 Open the new primary database. This may take some time to complete.


Configuring High Availability for the PMG DBTesting Site Failure

Example:SQL> ALTER DATABASE OPEN;Database altered.

Step 10 Check the status of the database. The switchover status could be FAILED DESTINATION or NOT ALLOWED.

Example:SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE, SWITCHOVER_STATUS from v$database;NAME OPEN_MODE DB_UNIQUE_NAME DATABASE_ROLE SWITCHOVER_STATUS--------- -------------------- ------------------------------ ---------------- --------------------PMGDB READ WRITE PMGDB_STBY2PRIMARY FAILED DESTINATION

Step 11 Verify that applications connecting to the PMGDB database are getting connected now that the cold standby is up as anew primary. Log in to the Central node as admin1 user and execute the getAreas.sh script.

Example:$ getAreas.sh -key 1001Config files script-props/private/GetAreas.properties or script-props/public/GetAreas.properties notfound. Continuing with default settings.Execution parameters:


Recovering Original Primary After Site FailureThe database can be recovered using either the RMAN backup or Flashback database. Following steps describerecovery using RMAN backup.

Converting a Failed Primary into a Standby Database Using RMAN Backups

DETAILED STEPS


Determine the SCN at which the old/cold standby database became the primary database.Step 1

Run the following query on the new primary database to determine the SCN at which the original/cold standby database became the newprimary database:

Step 2

Example:SQL> SELECT TO_CHAR(STANDBY_BECAME_PRIMARY_SCN) FROM V$DATABASE;TO_CHAR(STANDBY_BECAME_PRIMARY_SCN)----------------------------------------1351900

Restore the database with a backup taken before the original primaryhad reached the SCN at which the standby became the new primary

Restore and recover the entire database on the original primary.Step 3


Configuring High Availability for the PMG DBRecovering Original Primary After Site Failure


(standby_became_primary_scn). Then, perform a point-in-time recoveryto recover the original primary to that same point.

Find out the value of the <recovery_scn> from the output from theprevious command. It will be adding 1 to the value ofstandby_became_primary_scn. For example, if previous commandreturns standby_became_primary_scn value as 1351900, <recovery_scn>will be 1351900 + 1 = 1351901.

If the output of command “RMAN> RUN…” shows error“RMAN-20208”, see the Troubleshooting Data Guard on PMGDB, on page 65 to resolve the issue.

Note

Run the following RMAN commands on the original primary.Step 4

Example:$ export ORACLE_SID=PMGDB$ rman target=/Recovery Manager: Release 11.2.0.1.0 - Production on Tue Jul 8 19:30:41 2014



RMAN> startup mountOracle instance starteddatabase mounted

Total System Global Area 1068937216 bytesFixed Size 2220200 bytesVariable Size 289410904 bytesDatabase Buffers 771751936 bytesRedo Buffers 5554176 bytes

RMAN> RUN{SET UNTIL SCN <recovery_scn> ;RESTORE DATABASE;RECOVER DATABASE;}

executing command: SET until clause

Starting restore at 08-JUL-14allocated channel: ORA_DISK_1channel ORA_DISK_1: SID=11 device type=DISK

channel ORA_DISK_1: starting datafile backup set restorechannel ORA_DISK_1: specifying datafile(s) to restore from backup setchannel ORA_DISK_1: restoring datafile 00001 to /u01/app/oracle/oradata/PMGDB/system01.dbfchannel ORA_DISK_1: restoring datafile 00002 to /u01/app/oracle/oradata/PMGDB/sysaux01.dbfchannel ORA_DISK_1: restoring datafile 00003 to /u01/app/oracle/oradata/PMGDB/undotbs01.dbfchannel ORA_DISK_1: restoring datafile 00004 to /u01/app/oracle/oradata/PMGDB/users01.dbfchannel ORA_DISK_1: restoring datafile 00005 to /u01/app/oracle/oradata/PMGDB/mapinfo_ts.dbfchannel ORA_DISK_1: restoring datafile 00006 to /u01/app/oracle/oradata/PMGDB/pmgdb_ts.dbfchannel ORA_DISK_1: reading from backup piece/u01/app/oracle/flash_recovery_area/PMGDB/backupset/2014_07_08/o1_mf_nnndf_TAG20140708T162646_9vqmmgsv_.bkpchannel ORA_DISK_1: piecehandle=/u01/app/oracle/flash_recovery_area/PMGDB/backupset/2014_07_08/o1_mf_nnndf_TAG20140708T162646_9vqmmgsv_.bkptag=TAG20140708T162646channel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 00:00:25




Finished restore at 08-JUL-14

Starting recover at 08-JUL-14using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 24 is already on disk as file/u01/app/oracle/flash_recovery_area/PMGDB/archivelog/2014_07_08/o1_mf_1_24_9vqmn043_.arcarchived log for thread 1 with sequence 25 is already on disk as file/u01/app/oracle/flash_recovery_area/PMGDB/archivelog/2014_07_08/o1_mf_1_25_9vqv5j39_.arcarchived log for thread 1 with sequence 26 is already on disk as file/u01/app/oracle/flash_recovery_area/PMGDB/archivelog/2014_07_08/o1_mf_1_26_9vqw8v1v_.arcarchived log file name=/u01/app/oracle/flash_recovery_area/PMGDB/archivelog/2014_07_08/o1_mf_1_24_9vqmn043_.arcthread=1 sequence=24media recovery complete, elapsed time: 00:00:00Finished recover at 08-JUL-14


Convert original primary database to a physical standby database. Perform the following steps on the original primary database.Step 5

Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 16 17:51:51 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL>

SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;Database altered.

Shut down and start up the original primary.Step 6

Example:SQL> SHUTDOWN IMMEDIATE;ORA-01507: database not mounted

ORACLE instance shut down.



The goal of this step is to synchronize the control file with the databaseby using a dictionary check. After this command, check the alert log

Open the database as read-only.

Example:SQL> ALTER DATABASE OPEN READ ONLY;Database altered.

Step 7

(/u01/app/oracle/diag/rdbms/pmgdb/PMGDB/trace/alert_PMGDB.log)for any actions suggested by the dictionary check. Typically, no useraction is needed if the original primary was not in the middle of addingor dropping data files during the failover.




The log directory may vary based on the installationsetup.

Note

Mount the new standby (original primary) again.Step 8

Example:SQL> SHUTDOWN IMMEDIATE;Database closed.Database dismounted.ORACLE instance shut down.



Log in to the sql prompt on the new primary (old cold standby) and make sure the archive destination is enabled.Step 9

Example:SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;System altered.

Now the failed primary database is restored and is running in the standbyrole.

Start Redo Apply on the new standby (original primary) database.

Example:SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;Database altered.

Step 10

Verify status, it could be SWITCHOVER PENDING or SWITCHOVER LATENT or TO PRIMARY.Step 11

Example:SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE;SWITCHOVER_STATUS-----------------SWITCHOVER PENDING

Verify the status on the new primary as well. A value of TO STANDBY or SESSIONS ACTIVE indicates that the primary database canbe switched to the standby role.

Step 12

Example:SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE;SWITCHOVER_STATUS-----------------TO STANDBY



Converting Failed Primary Into a Standby Database Using RMAN Backups

DETAILED STEPS


Issue the following SQL statement on the primary database to switch it to thestandby role:

Step 1

Example:SQL> select DATABASE_ROLE from v$database;DATABASE_ROLE----------------PRIMARY

SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITHSESSION SHUTDOWN;Database altered.

Shut down and then mount this current primary database.Step 2

Example:SQL> SHUTDOWN IMMEDIATE;ORA-01507: database not mounted


SQL> STARTUP MOUNT;ORACLE instance started.Total System Global Area 1068937216 bytesFixed Size 2220200 bytesVariable Size 289410904 bytesDatabase Buffers 771751936 bytesRedo Buffers 5554176 bytesDatabase mounted.

Check the switchover status on the current primary which now has become coldstandby again. A value of TO PRIMARY or SESSIONS ACTIVE indicates that

Step 3

the standby database is ready. If neither of these values is returned, continue toquery this column until the value returned is either TO PRIMARY or SESSIONSACTIVE.

Example:SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE;SWITCHOVER_STATUS-----------------TO_PRIMARY

Return to the original primary to revert its database role to the primary role.Step 4

Example:SQL> RECOVER MANAGED STANDBY DATABASE FINISH;Media recovery complete.

SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSIONSHUTDOWN;Database altered.




Now open the original primary database.Step 5

Example:SQL> ALTER DATABASE OPEN;Database opened.

Ignore and continue if error"ORA-01153: an incompatiblemedia recovery is active" isdisplayed.

NoteReturn to the new physical standby (cold standby) database and start redo apply.

Example:SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENTLOGFILE DISCONNECT FROM SESSION;Database altered.

Step 6

On original primary:Check status on the original primary and cold standby tomake sure that the databaserole is correct.

Step 7

On cold standby:

Example:SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE,SWITCHOVER_STATUS from v$database;NAME OPEN_MODE DB_UNIQUE_NAMEDATABASE_ROLE--------- -------------------- ----------------------------------------------SWITCHOVER_STATUS--------------------PMGDB READ WRITE PMGDB PRIMARYFAILED DESTINATION

Example:SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE,SWITCHOVER_STATUS from v$database;NAME OPEN_MODE DB_UNIQUE_NAMEDATABASE_ROLE--------- -------------------- ----------------------------------------------SWITCHOVER_STATUS--------------------PMGDB MOUNTED PMGDB_STBY2 PHYSICALSTANDBYSESSIONS ACTIVE

Start up and mount the hot standby if hot standby is present in the setup.Step 8

Example:$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Thu Jul 10 11:39:54 2014




Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1845495840 bytesDatabase Buffers 1476395008 bytes




Redo Buffers 16343040 bytesDatabase mounted.

The status should be a SUCCESS,which indicates reverting to the original

Verify the dgbroker status, log in to DGMGRL either from primary or standby.Note that in case of hot and cold standby to primary setup,WARNING is displayed

Step 9

roles is complete for hot and coldstandby to primary setup.

as hot standby server is not enabled. For only cold standby to primary setup, itshould show SUCCESS which means reverting to the original role is complete.


Note • If the output of commandshows ERROR orWARNING status, seethe TroubleshootingDataGuard on PMG DB, onpage 65 to resolve theissues.

• Sometimes processesfrom the previouscommand may not becomplete, therefore the“show configuration”command may showdifferent output. Allow acouple of minutes andexecute the commandagain.

In case of primary and cold standbysetup:

Copyright (c) 2000, 2009, Oracle. All rights reserved.Welcome to DGMGRL, type "help" for information.


DGMGRL> show configuration

Example:Configuration - dgPMGDBProtection Mode : MaxPerformanceDatabases : PMGDB - Primary database

: PMGDB_STBY - Physical standby databaseFast-Start Failover : DISABLEDConfiguration Status : SUCCESS

Example:Configuration - dgPMGDBProtection Mode: MaxAvailability In case of primary hot and cold standby

setup:Databases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databasePMGDB_STBY2 - Physical standby database


In case of primary hot and cold standbysetup:

Enable cold standby if cold standby (PMGDB_STBY2) is showing disabled orwith warning.

Step 10

Example:DGMGRL> enable database 'PMGDB_STBY2'Enabled.

If the output of commandshows ERRORorWARNINGstatus, see the TroubleshootingData Guard on PMG DB, onpage 65 to resolve the issues.

Note

DGMGRL> show configuration

Example:Configuration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databasePMGDB_STBY2 - Physical standby database






Rolling Back and Cleaning Up Standby and PrimaryConfigurations

This section covers common scenarios for rollback. For specific scenarios, like database corruption and soon, it is recommended that you refer to standard DBA practices or Oracle documentation.

Before starting the rollback, it is recommended that the primary database is backed up.Note

The following procedures describe how to roll back the standby and primary configurations.

• Removing Data Guard Broker Configuration, on page 55

• Removing the Standby Database, on page 60

• Removing the Additional Standby Database, on page 61

Removing Data Guard Broker Configuration

Step 1 Connect to the primary database from DGMGRL.


Step 2 Disable FAST_START FAILOVER.Perform this step only if the setup has hot standbyconfigured.

Note

Example:DGMGRL> DISABLE FAST_START FAILOVER FORCEDisabled.

Step 3 Change the protection mode.Perform this step only if the setup has hot standbyconfigured.

Note


Configuring High Availability for the PMG DBRolling Back and Cleaning Up Standby and Primary Configurations

Example:DGMGRL> edit configuration set protection mode as MaxPerformance;Succeeded.

Step 4 Remove the configuration.

Example:DGMGRL> remove configurationRemoved configuration

Step 5 Check if the configuration is removed.

Example:DGMGRL> show configurationORA-16532: Data Guard broker configuration does not exist

Configuration details cannot be determined by DGMGRL

Alternate OutputORA-16596: database not part of the Data Guard broker configuration

Configuration details cannot be determined by DGMGRL

Removing Configuration Files from Primary Server

Step 1 Log in to the primary server as an oracle user and connect to sqlplus.

Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Wed Jul 2 20:17:31 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL>

Step 2 Stop the Data Guard Broker process.

Example:SQL> alter system set dg_broker_start=FALSE;System altered.

Step 3 Check the path of the Data Guard Broker configuration files.

Example:SQL> select name , value from v$parameter where name like '%dg_broker%';NAME--------------------------------------------------------------------------------VALUE------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------dg_broker_startFALSE

dg_broker_config_file1


Configuring High Availability for the PMG DBRemoving Data Guard Broker Configuration

/u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB.dat

dg_broker_config_file2/u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB.dat



Step 5 Remove dg_broker_config_file1 and dg_broker_config_file2.TheOracle base directory pathmay vary based on your installation.Note

Example:$ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB.dat

$ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB.datThe system responds with a command prompt.

Step 6 Remove the standby control files if they still exist in /tmp directory. Ignore the error if the standby control files do notexist.

Example:$ rm /tmp/pmgdb_stby.ctl

$ rm /tmp/pmgdb_stby2.ctlThe system responds with a command prompt.

Removing Configuration Files From Standby Server

Note • If the setup is only cold standby added to the primary, then this standby is cold standby.

• If the setup is only hot standby added to the primary, then this standby is hot standby.

• If the setup is both hot and cold standby added to the primary, this standby is hot standby.

DETAILED STEPS


Log in to the first standby server as an oracle user and connect to sqlplus.Step 1

Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Wed Jul 2 20:17:31 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL




Stop the redo apply process.Step 2


Stop Data Guard Broker process.Step 3


Check the path of the Data Guard Broker configuration files.Step 4


dg_broker_config_file1/u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB_STBY.dat

dg_broker_config_file2/u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB_STBY.dat

Open the database in read-only mode.Step 5

Example:SQL> alter database open read only;alter database open read only;

Exit from the sql prompt.Step 6


TheOraclebasedirectorypathmayvarybasedon yourinstallation.

NoteRemove the dg_broker_config_file1 and dg_broker_config_file2.

Example:$ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB_STBY.datThe system responds with a command prompt.

$ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB_STBY.datThe system responds with a command prompt.

Step 7



Removing Configuration Files From Additional Standby Server

This section is applicable only to the setup with both hot and cold standby added to primary. This additionalstandby is the cold standby in the setup.

Note

DETAILED STEPS


Log in to the standby server as an oracle user and connect to sqlplus.Step 1

Example:$ export ORACLE_SID=PMGDB$ sqlplus / as sysdbaSQL*Plus: Release 11.2.0.1.0 Production on Wed Jul 2 20:17:31 2014Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsSQL>

Stop the redo apply process.Step 2


Stop the Data Guard Broker process.Step 3


Check the path of the Data Guard Broker configuration files.Step 4


dg_broker_config_file1/u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB_STBY2.dat

dg_broker_config_file2/u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB_STBY2.dat

Open the database in read-only mode.Step 5

Example:SQL> alter database open read only;alter database open read only;

Exit from the sql promptStep 6





TheOraclebasedirectorypathmayvarybasedon yourinstallation.

NoteRemove the dg_broker_config_file1 and dg_broker_config_file2.

Example:$ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr1PMGDB_STBY2.datThe system responds with a command prompt.

$ rm /u01/app/oracle/product/11.2.0/dbhome_1/dbs/dr2PMGDB_STBY2.datThe system responds with a command prompt.

Step 7

Removing the Standby Database

Note • If the setup is only cold standby added to the primary, then this standby is cold standby.

• If the setup is only hot standby added to the primary, then this standby is hot standby.

• If the setup is both hot and cold standby added to the primary, this standby is hot standby.

Step 1 From first standby server console, execute the dbca command.

Example:$ dbca -silent -deleteDatabase -sourceDB PMGDBConnecting to database…100% completeLook at the log file "/u01/app/oracle/cfgtoollogs/dbca/PMGDB.log" for further details.

Step 2 Move the respective directories as backup directories.TheOracle base directory pathmay vary based on your installation.Note

Example:$ mv /u01/app/oracle/oradata/PMGDB /u01/app/oracle/oradata/PMGDB-bakThe system responds with a command prompt.

$ mv /u01/app/oracle/flash_recovery_area/PMGDB /u01/app/oracle/flash_recovery_area/PMGDB-bakThe system responds with a command prompt.

$ mv /u01/app/oracle/flash_recovery_area/PMGDB_STBY /u01/app/oracle/flash_recovery_area/PMGDB-STBY-bakThe system responds with a command prompt.


Configuring High Availability for the PMG DBRemoving the Standby Database

$ mv /u01/app/oracle/admin/PMGDB /u01/app/oracle/admin/PMGDB-bakThe system responds with a command prompt.

Removing the Additional Standby DatabaseThis section is applicable only to the setup with both hot and cold standby added to primary. This additionalstandby is the cold standby in this setup.

DETAILED STEPS


From the standby server console, execute the dbca command.Step 1

Example:$ dbca -silent -deleteDatabase -sourceDB PMGDBConnecting to database…100% completeLook at the log file "/u01/app/oracle/cfgtoollogs/dbca/PMGDB.log" for further details.

Move the respective directories as backup directories.Step 2

Example:$ mv /u01/app/oracle/oradata/PMGDB /u01/app/oracle/oradata/PMGDB-bakThe system responds with a command prompt.

$ mv /u01/app/oracle/flash_recovery_area/PMGDB /u01/app/oracle/flash_recovery_area/PMGDB-bakThe system responds with a command prompt.

$ mv /u01/app/oracle/flash_recovery_area/PMGDB_STBY2/u01/app/oracle/flash_recovery_area/PMGDB-STBY2-bakThe system responds with a command prompt.

$ mv /u01/app/oracle/admin/PMGDB /u01/app/oracle/admin/PMGDB-bakThe system responds with a command prompt.

Cleaning Up the Primary Database

Cleaning Up the Redo Log Files

Step 1 Drop the standby redo log files from the primary database.

Example:SQL> ALTER DATABASE DROP STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo01.log');Database altered.


Configuring High Availability for the PMG DBCleaning Up the Primary Database

SQL> ALTER DATABASE DROP STANDBY LOGFILE ('/u01/app/oracle/oradata/PMGDB/standby_redo02.log');Database altered.



Step 2 Remove the standby redo log files from the directory.

Example:$ rm /u01/app/oracle/oradata/PMGDB/standby_redo01.logThe system responds with a command prompt.

$ rm /u01/app/oracle/oradata/PMGDB/standby_redo02.logThe system responds with a command prompt.



Cleaning Up Initialization ParametersThe initialization parameters can be reset using either one of these methods:

Note • (Recommended) Create spfile from backup pfile that was generated during initial steps. For moreinformation, see Using the Backup Pfile, on page 62.

• If the backup pfile is not available, the parameters can be reset using sql statements. For moreinformation, see Using SQL Statements, on page 63.

Using the Backup Pfile

Use this option to restore the spfile from the backup pfile.

Step 1 Shutdown the database.


Step 2 From the sql prompt, create the spfile from the backup pfile.f the specific path for the initPMGDB_pre_dgsetup.ora file (for example, /backup/initPMGDB_pre_dgsetup.ora)was specified when the file was backed up during the initial steps, specify the full path (for example,/backup/initPMGDB_pre_dgsetup.ora). If the path is not specified, the file will be picked up from the defaultdirectory, that is, $ORACLE_HOME/dbs.

Note



Example:SQL> CREATE SPFILE FROM PFILE='initPMGDB_pre_dgsetup.ora';File created.

Step 3 Start up the database.

Example:SQL> startupORACLE instance started.

Total System Global Area 3340451840 bytesFixed Size 2217952 bytesVariable Size 1828718624 bytesDatabase Buffers 1493172224 bytesRedo Buffers 16343040 bytesDatabase mounted.Database opened.

Using SQL Statements

If the backup pfile is not available, the parameters can be reset through sql statements. This section can beskipped if spfile is already restored from the backup pfile in the previous section.

From the sql prompt, reset the initialization parameters.SQL> ALTER SYSTEM RESET LOG_ARCHIVE_CONFIG;System altered.

SQL> ALTER SYSTEM RESET LOG_ARCHIVE_DEST_2;System altered.

SQL> ALTER SYSTEM RESET LOG_ARCHIVE_DEST_STATE_2;System altered.

SQL> ALTER SYSTEM RESET LOG_ARCHIVE_FORMAT SCOPE=SPFILE;System altered.

SQL> ALTER SYSTEM RESET LOG_ARCHIVE_MAX_PROCESSES;System altered.

SQL> ALTER SYSTEM RESET FAL_SERVER;System altered.

SQL> ALTER SYSTEM RESET DB_FILE_NAME_CONVERT SCOPE=SPFILE;System altered.

SQL> ALTER SYSTEM RESET LOG_FILE_NAME_CONVERT SCOPE=SPFILE;System altered.

SQL> ALTER SYSTEM RESET LOG_ARCHIVE_DEST_3;System altered.

SQL> ALTER SYSTEM RESET LOG_ARCHIVE_DEST_STATE_3;System altered.



Verifying the Database

Before You Begin

Step 1 From the sql prompt, check if the database role is ‘Primary’.

Example:SQL> select name, open_mode, DB_UNIQUE_NAME, DATABASE_ROLE, SWITCHOVER_STATUS from v$database;NAME TYPE VALUE------------------------------------ ----------- ------------------------------service_names string PMGDB_PRIMARY

Step 2 Ensure that the database service PMGDB_PRIMARY is present.

Example:SQL> show parameter serviceNAME TYPE VALUE------------------------------------ ----------- ------------------------------service_names string PMGDB_PRIMARY

Step 3 Verify that applications connecting to the PMGDB database are getting connected. For example, the OpsTools scriptgetAreas.sh can be executed. Log in to the Central node as admin1 user and execute the script.

Example:$ getAreas.sh -key 1001Config files script-props/private/GetAreas.properties or script-props/public/GetAreas.properties notfound.Continuing with default settings.Execution parameters:


Recreating Standby ServersNow that the standby server and primary server configurations are cleaned up, reconfigure the Data GuardBroker. For more information, see the “Setting Up the Oracle Data Guard Broker” procedure on page 58.

Deleting the Primary DatabaseThe primary database is now rolled back to previous state using steps in the previous sections. However, ifthere is a need to clean up all configurations, including the primary database, ensure that you do the following:

• Delete the primary database and create it again using the standard DBA process (for example, usingOracle DBCA utility, and so on).

• Install the PMG DB schema and populate data as per steps mentioned in the install guide. For moreinformation, see, Cisco RAN Management System Installation Guide.


Configuring High Availability for the PMG DBRecreating Standby Servers

• To create the standby servers again, follow the tasks listed in the Configuration Workflow, on page 3.

Troubleshooting Data Guard on PMG DBThis section provides solutions to some of the errors that may be encountered while configuring Data Guardon the PMG DB.

• Reverting Original Primary Database After Site Failure , on page 65

• Verifying the Data Guard Broker Configuration, on page 73

• Reverting From Disk Space Issues, on page 73

Reverting Original Primary Database After Site FailureIf the Data Guard Broker "show configuration" status is not a SUCCESS, check the error code and try toresolve using steps under respective errors. If not connected to DGMGRL tool, connect as sys@PMGDB.

The outputs of the steps are from the setup with primary, hot standby, and cold standby.Note

ORA-16825

ORA-16825: multiple errors or warnings, including fast-start failover-related errors or warnings, detected forthe database

If DGMGRL> show configuration is showing this error for primary database, check verbose view to seeexact errors related to the database.DGMGRL> show database verbose 'PMGDB'Database - PMGDB

Role: PRIMARYIntended State: TRANSPORT-ONInstance(s):PMGDB

Database Error(s):ORA-16628: broker protection mode inconsistent with the database setting

Database Warning(s):ORA-16817: unsynchronized fast-start failover configuration

………ORA-16628

ORA-16628: broker protection mode inconsistent with the database setting

IfDGMGRL> show configuration is showing this error for the primary database, change toMaxAvailabilitymode. Disable “Fast-Start Failover”, if configuration shows it as “ENABLED”.DGMGRL> disable fast_start failoverDisabled.

DGMGRL> edit configuration set protection mode as MaxAvailability;Succeeded.


Configuring High Availability for the PMG DBTroubleshooting Data Guard on PMG DB

Check configuration; if it is a SUCCESS enable fast_start failover, otherwise troubleshootfor the errordisplayed.DGMGRL> show configuration………Fast-Start Failover: DISABLED


DGMGRL> enable fast_start failoverEnabled.

ORA-16810

ORA-16810: multiple errors or warnings detected for the database

If DGMGRL> show configuration is showing this error for standby database, check verbose view to seeexact errors related to the database.DGMGRL> show database verbose 'PMGDB_STBY'Database - PMGDB_STBY

Role: PHYSICAL STANDBYIntended State: APPLY-ONTransport Lag: (unknown)Apply Lag: (unknown)Real Time Query: OFFInstance(s):PMGDB

Database Error(s):ORA-16700: the standby database has diverged from the primary databaseORA-16766: Redo Apply is stopped

………ORA-16700

ORA-16700: the standby database has diverged from the primary database

If DGMGRL> show database verbose is showing this error for the standby database it indicates that the theprimary and standby server files have a mismatch. View alertPMGDB.log of the standby database to see if itis due to ORA-19909 error. If ORA-19909 is in the log, follow the trouble steps for ORA-19909. Followingis sample output of the logs.$ cd /u01/app/oracle/diag/rdbms/pmgdb_stby/PMGDB/trace$ tail -f alert_PMGDB.log -n100Or$ vi alert_PMGDB.log

………MRP0 started with pid=66, OS id=24771MRP0: Background Managed Standby Recovery process started (PMGDB)Serial Media Recovery startedManaged Standby Recovery starting Real Time ApplyWarning: Recovery target destination is in a sibling branchof the controlfile checkpoint. Recovery will only recoverchanges to datafiles.MRP0: Detected orphaned datafiles!Recovery will possibly be retried after flashback...Errors in file /u01/app/oracle/diag/rdbms/pmgdb_stby/PMGDB/trace/PMGDB_mrp0_24771.trc:ORA-19909: datafile 1 belongs to an orphan incarnationORA-01110: data file 1: '/u01/app/oracle/oradata/PMGDB/system01.dbf'


Configuring High Availability for the PMG DBReverting Original Primary Database After Site Failure

Managed Standby Recovery not using Real Time Apply………

The log directory may vary based on the installation setup.Note

ORA-19909

ORA-19909: datafile 1 belongs to an orphan incarnation

If alert_PMGDB.log shows ORA-19909 error, it indicates that the primary database was opened withRESETLOGS after the database recovery that causes the database incarnation to be different between theprimary and standby database. The following steps describe how to avoid re-creating a physical standbydatabase to resolve this issue.

These steps require FLASHBACK enabled. FLASHBACK is enabled as part of configuring the standbyserver setup and is enabled for standby databases. However, for any reason if it is not enabled, the standbydatabase needs to be recreated. To do this, see “ORA-38729”.

Note

1 Before proceeding ensure that the fast_start failover in DGMRL configuration is DISABLED. If notdisabled, disable it.DGMGRL> disable fast_start failoverDisabled.

2 Determine the SCN before the RESETLOGS operation occurred.

• On the primary database, use the following query to obtain the value of the system change number(SCN) that is two SCNs before the RESETLOGS operation occurred on the primary database.SQL> SELECT TO_CHAR(RESETLOGS_CHANGE# - 2) FROM V$DATABASE;TO_CHAR(RESETLOGS_CHANGE#-2)----------------------------------------1245964

3 Obtain the current SCN on the standby database.SQL> SELECT TO_CHAR(CURRENT_SCN) FROM V$DATABASE;TO_CHAR(CURRENT_SCN)----------------------------------------1249692

4 If the value of CURRENT_SCN is larger than the value of resetlogs_change# - 2, issue the followingstatement on the standby database to flash back the standby database.

If the value of CURRENT_SCN is less than the value of the resetlogs_change# - 2, skip to next step.Note

SQL> FLASHBACK STANDBY DATABASE TO SCN 1245964;Flashback complete.

5 Restart Redo Apply on the standby server.



This step may have to be repeated if following step shows configuration status Error or Warning.Note

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;Database altered.

6 Connect to DGMGRL as sys@PMGDB if not connected and view the DGMGRL show configuration; ifthe status is error with error ORA-16766, repeat previous step to restart redo apply and continue with thisstep till no error is displayed.DGMGRL> show configurationConfiguration - dgPMGDB

Protection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - Physical standby databaseError: ORA-16766: Redo Apply is stopped


Fast-Start Failover: DISABLED

Configuration Status:ERROR

7 If the previous step, shows “Warning: ORA-16826: apply service state is inconsistent with the DelayMinsproperty”, use this step on the standby server sql prompt. Else continue with the next step.SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;Database altered.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECTFROM SESSION;Database altered.

DGMGRL> show configurationConfiguration - dgPMGDB

Protection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - Physical standby databasePMGDB_STBY2 - Physical standby database (disabled)ORA-16661: the standby database needs to be reinstated

Fast-Start Failover: DISABLED


8 Now enable fast_start failover and if cold standby is present in the setup, enable cold standby database.DGMGRL> enable fast_start failoverEnabled.

DGMGRL> enable database 'PMGDB_STBY2'Enabled.

9 Verify that errors are resolved and the setup is reverted to the original state successfully.DGMGRL> show configurationConfiguration - dgPMGDB

Protection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databasePMGDB_STBY2 - Physical standby database





RMAN-20208

RMAN-20208: UNTIL CHANGE is before RESETLOGS change

The following command may throw error as shown.RMAN> RUN

{SET UNTIL SCN <recovery_scn> ;RESTORE DATABASE;RECOVER DATABASE;}

Starting restore at 17-JUL-14using target database control file instead of recovery catalogRMAN-00571: ===========================================================RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============RMAN-00571: ===========================================================RMAN-03002: failure of restore command at 07/17/2014RMAN-20208: UNTIL CHANGE is before RESETLOGS changeSolution:

The error RMAN-20208 occurred because the command is trying to go to an SCN (<recovery_scn>) beforethe current incarnation of the database. Follow the steps to resolve this.

1 List the incarnation of the original primary database in RMAN.$ export ORACLE_SID=PMGDB$ rman target=/RMAN> list incarnation of database;using target database control file instead of recovery catalog

List of Database IncarnationsDB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time------- ------- -------- ---------------- --- ---------- ----------1 1 PMGDB 2481454463 PARENT 1 15-AUG-092 2 PMGDB 2481454463 PARENT 945184 17-JUL-143 3 PMGDB 2481454463 PARENT 1243235 17-JUL-144 4 PMGDB 2481454463 CURRENT 1243996 17-JUL-14

2 Nowmark the incarnation having SCN before the <recovery_scn>. In this example, if value of recovery_scnis 1243233, select incarnation 2 with SCN 945184. Reset the database to this incarnation.RMAN> reset database to incarnation 2;database reset to incarnation 2

3 Now execute the RUN command which was failed; it should show output as expected.RMAN> RUN

{SET UNTIL SCN <recovery_scn> ;RESTORE DATABASE;RECOVER DATABASE;}

executing command: SET until clause

………media recovery complete, elapsed time: 00:00:00Finished recover at 08-JUL-14

ORA-38729

ORA-38729: Not enough flashback database log data to do FLASHBACK.



This error may occur while resolving error RMAN-20208. If FLASHBACK DATABASE is run when thetarget SCN is outside the flashback window, then FLASHBACK DATABASE fails with an ORA-38729error. In this case the database will not be changed. The possible reason to get this error could be intentionallyor unintentionally removing/moving some flashback database logs.

To overcome this error, recreate the standby again, that is, clean up as described in Rolling Back and CleaningUp Standby and Primary Configurations, on page 55 and create the standby as described in Configuring thePrimary Server, on page 4 and Configuring the Hot Standby Server, on page 10. Though most of the stepsare same, some additional changes are required. Hence those steps are repeated here.

1 Log in to primary sql prompt and defer LOG_ARCHIVE_DEST_STATESQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=DEFER;System altered.

2 Log in to DGMGRL from primary or hot standby; Disable FAST_START FAILOVER is not disabledalready.DGMGRL> DISABLE FAST_START FAILOVER FORCEDisabled.

3 Change protection mode in Data Guard Broker configuration.DGMGRL> edit configuration set protection mode as MaxPerformance;Succeeded.

4 Remove hot standby from Data Guard Broker configuration.DGMGRL> REMOVE DATABASE 'PMGDB_STBY'Removed database "PMGDB_STBY" from the configuration

5 To remove the configuration files from the hot standby, see .

Redo apply process may have already stopped for the hot standby. Ignore the error if the step "Stop redoapply process" throws any error.

Note

6 To remove the standby database, see Removing the Standby Database, on page 60.

7 Return and log in to the primary sql prompt and enable LOG_ARCHIVE_DEST_STATE.SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;System altered.

8 Back up the primary database, see Backing Up the Primary Database, on page 8.

9 Move existing file as backup if pfile and control file already exist in the /tmp file.$ mv /tmp/pmgdb_stby.ctl /tmp/pmgdb_stby.ctl_back$ mv /tmp/initPMGDB_stby.ora /tmp/initPMGDB_stby.ora_backSystem responds with command prompt.

10 Create the standby control file and PFILE, see the Creating Standby Control File and PFILE, on page 9.

11 Stop the observer by logging in to DGMGRL and connecting as sys@PMGDB if the observer is running.DGMGRL> stop observerDone.

12 Create a hot standby, see Configuring the Hot Standby Server, on page 10.

ORA-16820

ORA-16820: fast-start failover observer is no longer observing this database

This error may be displayed in DGMGRL show configuration output, if the observer is not running or thedrcPMGDB.log (under /u01/app/oracle/diag/rdbms/pmgdb/PMGDB/trace) on primary shows warning as“ObserverHB indicates lack of Observer pings at Standby”.



To overcome this error, stop the observer and run the observer from a different host (primary or standby)other than the one that is running currently.

1 Log in to DGMGRL and connect as [email protected]> stop observerDone.

2 Open another console for primary or standby server and log in as oracle user. Start observer as a backgroundprocess.$ nohup dgmgrl -silent sys/<passoword>@PMGDB "start observer" &$ cat nohup.outObserver started

3 Stop observer again if error persists and start observer from another host.

ORA-16631

ORA-16631: operation requires shutdown of database or instance ""

This error may occur when enabling cold standby database while recovering original primary after site failure:DGMGRL> enable database 'PMGDB_STBY2'Error: ORA-16631: operation requires shutdown of database or instance ""To overcome this error:

1 Shut down and start up mount the cold standby.SQL> SHUTDOWN IMMEDIATE;ORA-01507: database not mounted



2 Attempt to enable the database again from the DGMGRL. If successful, skip the next steps.

3 Verify database parameter FAL_SERVER on cold standby sql prompt if error still persists. If the valueis not “pmgdb”, correct it using the alter system command.SQL> show parameter fal_serverNAME TYPE VALUE------------------------------------ ----------- ------------------------------fal_server string pmgdb

SQL> ALTER SYSTEM SET FAL_SERVER=pmgdbSystem altered.

4 Verify LOG_ARCHIVE_DEST_3 on the primary database. If it is not pointing to PMGDB_stby2 asshown in the output, Set LOG_ARCHIVE_DEST_3 and enable.SQL> show parameter LOG_ARCHIVE_DEST_STATE_3NAME TYPE VALUE-------------------- ----- ------------------------------log_archive_dest_3 string SERVICE=PMGDB_stby2 NOAFFIRM A

SYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PMGDB_STBY2

log_archive_dest_30 stringlog_archive_dest_31 string

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_3='SERVICE=PMGDB_stby2 NOAFFIRM ASYNC



VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PMGDB_STBY2';System altered.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=ENABLE;System altered.

5 Shut down and startup mount the cold standby database.SQL> SHUTDOWN IMMEDIATE;ORA-01507: database not mountedORACLE instance shut down.


6 Remove the cold standby database from DGMGRL. Add standby database to the Data Guard Brokerconfiguration.DGMGRL> remove database 'PMGDB_STBY2';Removed database "PMGDB_STBY2" from the configuration

DGMGRL> add database 'PMGDB_STBY2' as connect identifier is PMGDB_STBY2 maintained asphysical ;Database " PMGDB_STBY2 " added

7 Enable cold standby database and verify that the configuration is showing that it is a SUCCESS.DGMGRL> enable database 'PMGDB_STBY2';Enabled.

DGMGRL> show configurationConfiguration - dgPMGDBProtection Mode: MaxAvailabilityDatabases:PMGDB - Primary databasePMGDB_STBY - (*) Physical standby databasePMGDB_STBY2 - Physical standby database


ORA-12543

ORA-12543: TNS:destination host unreachable

This error may occur while adding or viewing the database from the Data Guard Broker configuration. Thepossible reason could be that the entries in tnsnames.ora, listener.ora are wrong or with syntax error ornetwork/firewall is not allowing the source host to connect to the destination host.

1 Verify if tnsnames.ora on source host (primary/standby) has correct entries as mentioned in this document.The command tnsping can be used to verify:$ tnsping <connection_name>where <connection_name> is the name of an entry in tnsnames.ora file. i.e. either PMGDB, PMGDB_STBYor PMGDB_STBY2.

If connection is working, the output would be OK as shown in following example.$ tnsping PMGDB_STBY2TNS Ping Utility for Linux: Version 11.2.0.1.0 - Production on 07-SEP-2014 19:48:44Copyright (c) 1997, 2009, Oracle. All rights reserved.Used parameter files:



/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/sqlnet.oraUsed TNSNAMES adapter to resolve the aliasAttempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = pmg-db-ha3)(PORT= 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = PMGDB)))OK (0 msec)

2 Verify if listener.ora on the destination host (primary/standby) has correct entries as mentioned in thisdocument. Try to restart listener.$ lsnrctl stop

$ lsnrctl start

3 Verify if any iptables rules on either source or destination host are blocking the connection. Try tnspingby stopping iptables rules, that is, log in as a root and execute:$ service iptables stop

Log in as an oracle user and verify tnsping$ tnsping <connection_name>

ORA-16737

ORA-16737: the redo transport service for standby database "PMGDB_STBY" has an error

The error may be displayed in DGMGRL show configuration. This may be an intermediate error which maygo away if subsequent steps are executed or underlying errors could be different which can be viewed usingthe show database command.DGMGRL> show database verbose 'PMGDB'.

Verifying the Data Guard Broker ConfigurationThe DGMGRL> show configuration command configuration status sometimes shows errors ORA-16610& DGM-17017.

For example:Configuration Status:ORA-16610: command "ENABLE DATABASE PMGDB_STBY2" in progressDGM-17017: unable to determine configuration status

Configuration Status:ORA-16610: command "Broker automatic health check" in progressDGM-17017: unable to determine configuration status

Solution:

Wait for a few minutes and execute the command again.

Reverting From Disk Space IssuesAfter PMG DB is installed and configured, disk space issues may be encountered due to archiving of logsand DB content. This may cause the PMG DB to stop functioning. To verify if PMG DB has stoppedfunctioning, check the PMG DB alert log "alert.PMGDB.log" for the following error.ORA-19815: WARNING: db_recovery_file_dest_size of 4070572032 bytes is 99.56% used, and has17721344remaining bytes available.To revert from the error, increase the DB recovery file destination size to a higher value using the followingcommand:SQL> ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE=8G;


Configuring High Availability for the PMG DBVerifying the Data Guard Broker Configuration

In some cases, following error could be observed.ORA-00845: MEMORY_TARGET not supported on this systemThe possible reason is that the tmpfs directory may be full and it needs to be increased to a higher value usingthe following commands.

• To view tmpfs size:# df -k

• Change the size# mount -t tmpfs shmfs -o size=8g /dev/shm

• Verify the size# df -k

• To apply the above settings permanently to prevent the value from resetting to the default (after reboot),add the following entry in the /etc/fstab:shmfs /dev/shm tmpfs size=8g 0


Configuring High Availability for the PMG DBReverting From Disk Space Issues

Documents

Configuring High Availability for the PMG DB...Configuring High Availability for the PMG DB ThischapterdescribestheprocessofconfiguringhighavailabilityforthePMGDB.Itprovidesthe