4AA2-7717ENW

7/30/2019 4AA2-7717ENW

http://slidepdf.com/reader/full/4aa2-7717enw 1/26

Configuring HP Serviceguard Toolkit forOracle Data Guard

Technical white paper

Table of contents

Introduction ......................................................................................................................................... 2 Terms and definitions ........................................................................................................................... 2 Oracle Data Guard overview ................................................................................................................ 2

Physical and logical standby databases .............................................................................................. 2 Active Data Guard ........................................................................................................................... 3 Data Guard Broker........................................................................................................................... 3 Role transitions ................................................................................................................................ 3

Serviceguard support for Oracle Data Guard .......................................................................................... 3 Dependencies .................................................................................................................................. 4 Supported configurations .................................................................................................................. 4 Continentalclusters environment ......................................................................................................... 6 Metrocluster and EDC configurations .................................................................................................. 8 Metrocluster in Continentalclusters environment .................................................................................. 10 Supporting multiple instances of single-instance ODG configuration ..................................................... 11 HA for Data Guard Broker .............................................................................................................. 11

Installation and configuration of the toolkit ............................................................................................ 12 Setting up the application ............................................................................................................... 12 Setting up the toolkit ....................................................................................................................... 12 ODG package configuration example .............................................................................................. 14

Adding the package to the cluster .................................................................................................... 19 ODG maintenance ......................................................................................................................... 19

Troubleshooting ................................................................................................................................. 20 Known problems and workarounds ...................................................................................................... 21 For more information .......................................................................................................................... 22

Appendix A ...................................................................................................................................... 22 Configuring ODG example ............................................................................................................. 22 Preparing the primary database....................................................................................................... 22 Creating a physical standby database .............................................................................................. 23

7/30/2019 4AA2-7717ENW


2

Introduction

This document describes how the HP Serviceguard Toolkit for Oracle Data Guard assists in easyintegration of Oracle Data Guard (ODG) with HP Serviceguard. ODG is the host-based data-replication software for Oracle Database. It provides management, monitoring, and automationfeatures to create and maintain one or more standby databases to protect data from failures,disasters, human error, and data corruptions. To provide High Availability (HA) so that datareplication continues in the face of failures, ODG can be deployed in a Serviceguard cluster.

Terms and definitions

Term Definition

ASM Automatic Storage Management

ECMT Enterprise Cluster Master Toolkit

EDC Extended Distance Cluster

LVM Logical Volume Manager

MNP Multi-node Package

ODG Oracle Data Guard

RAC Real Application Clusters

Oracle Data Guard overview

Data Guard provides a comprehensive set of services that create, maintain, manage, and monitorone or more standby databases to enable production Oracle databases to survive disasters and datacorruption. Data Guard maintains these standby databases as transactionally consistent copies of theproduction database. If the production database is unavailable in case of a planned or unplannedoutage, then Data Guard can switch any standby database to take over in the production role, thus

reducing the downtime associated with the outage. Data Guard can be used with traditional backup,restoration, and cluster techniques to provide a high level of data protection and data availability. AData Guard configuration consists of one production database, known as the primary database, andup to nine standby databases.

Physical and logical standby databases

A standby database can be either a physical standby database or a logical standby database.

A physical standby database provides a physically identical copy of the primary database, withon-disk database structures that are identical to the primary database on a block-for-block basis. Thedatabase schema, including indexes, is the same. A physical standby database is kept synchronizedwith the primary database, through Redo Apply, which recovers the redo data received from the

primary database and applies the redo to the physical standby database.

A logical standby database contains the same logical information as the production database,although the physical organization and structure of the data can be different. The logical standbydatabase is kept synchronized with the primary database through SQL Apply, which transforms thedata in the redo data received from the primary database into SQL statements and then executes theSQL statements on the standby database. A logical standby database can be used for other businesspurposes in addition to disaster recovery requirements. The users can access a logical standbydatabase for queries and reporting purposes at any time.

7/30/2019 4AA2-7717ENW


3

The standby databases in a Data Guard configuration can be a combination of physical and logicalstandby databases.

Active Data Guard

Active Data Guard is a new feature for Oracle 11g. Oracle Active Data Guard enhances the qualityof service by offloading resource-intensive activities from a production database to one or moresynchronized standby databases. Oracle Active Data Guard enables read-only access to a physicalstandby database for queries, sorting, reporting, Web-based access, and so on, while continuously

applying changes received from the production database. Oracle Active Data Guard also enables theuse of fast incremental backups when offloading backups to a standby database, and can provideadditional benefits of high availability and disaster protection against planned or unplanned outagesat the production site.

Data Guard Broker

The Data Guard broker is a distributed management framework that automates the creation,maintenance, and monitoring of Data Guard configurations. The broker’s interfaces improve usabilityand centralize management and monitoring of the Data Guard configuration.

Role transitions

An Oracle database can be operated in one of two roles: primary or standby. Using Data Guard, therole of a database can be changed using either a switchover or a failover operation.

Switchover

This operation allows the primary database to switch roles with one of its standby databases. There isno data loss during a switchover. After a switchover, each database continues to participate in theData Guard configuration with its new role.

Failover

This operation changes a standby database to the primary role in response to a primary databasefailure. If the primary database was not operating in either maximum protection mode1 or maximumavailability2 mode before the failure, some data loss may occur. If Flashback Database3

Serviceguard support for Oracle Data Guard

is enabled

on the primary database, it can be reinstated as a standby for the new primary database once thereason for the failure is corrected.

Providing HA for ODG is essential for mission-critical business that requires ODG to be deployed inan environment that has a wide range of applications. Integrating ODG with Serviceguard using thetoolkit has the following additional advantages:

1. Provides HA for Data Guard processes and the Data Guard broker (if used) for both primary andthe standby databases. Serviceguard has built-in monitoring capabilities to monitor systemresources like network, volume groups, and filesystems, and to initiate failover following detection.

2. Data Guard provides disaster protection and recovery only for Oracle databases. Data Guardintegration with Serviceguard Extended Distance Cluster (EDC), Metrocluster, orContinentalclusters provides disaster protection and recovery for Oracle databases as well as forthe applications that use the databases.

1 This protection mode ensures that zero data loss occurs if a primary database fails. To provide this level of protection, the redo data needed torecover a transaction must be written to both the online redo log and to at least one synchronized standby database before the transactioncommits.

2 This protection mode provides the highest level of data protection that is possible without compromising the availability of a primary database.3 Flashback Database removes the need to re-create the primary database after a failover. Flashback Database is similar to conventional

point-in-time recovery in its effects, enabling you to return a database to its state at a time in the recent past.

7/30/2019 4AA2-7717ENW


4

Serviceguard has extensive dependencies that allow these application interdependencies to bebuilt in for a complete environment/deployment. This is advantageous from the completeapplication stack, as opposed to accounting for the Oracle Database alone.

3. Serviceguard’s robust failover mechanism can be extended to provide reliable automatic DataGuard role management (failover and switchover) operations.

4. Serviceguard is integrated with Insight Dynamics—VSE for Integrity Servers. This enables workloadmanagement, Instant Capacity, and other VSE technologies to be controlled in concert withServiceguard. The ODG Toolkit integration similarly allows the users to take advantage of these

capabilities for failure scenarios that affect ODG replication.

The Serviceguard Toolkit for Oracle Data Guard consists of a set of shell scripts that are used to start,stop, and monitor ODG primary and standby database instances. In the case of a single-instancedatabase, this toolkit leverages scripts from the ECMT Oracle Toolkit to start, stop, and monitor thedatabase, the listener, and the ASM instances.

Dependencies

The ODG Toolkit requires the ECMT Oracle Toolkit so that together they provide HA to single-instanceODG.

NOTE: For information about supportability and compatibility with various versions of

Serviceguard and HP-UX, please refer to the supportability matrix available athttp://www.hp.com/go/hpux-serviceguard-docs

Supported configurations

Single-instance Oracle database

Figure 1: ODG Toolkit in a single-instance Oracle database environment

The ODG Toolkit can be implemented as a combinational package with the ECMT Oracle Toolkit. Acombinational package is one in which two applications are packaged together by combining theirrespective Serviceguard modules into one package.

The ODG Toolkit has a hard dependency on the ECMT Oracle Toolkit. Hence, customers who do notalready have the ECMT product must purchase it along with the ODG Toolkit.

Node 1 Node 2

Standby

DB Primary

DB

SG Cluster 1 SG Cluster 2

Data Guard replication

Node 1 Node 2

Package forprimary DB

Packagefor standby

DB

http://www.hp.com/go/hpux-serviceguard-docs



7/30/2019 4AA2-7717ENW


5

As a combinational package, the Oracle database will be brought up first using the ECMT OracleToolkit; then the Data Guard processes will be started by the ODG Toolkit; then the applicationis monitored.

Since the Oracle database and the Data Guard are packaged together, the package will failover ifeither the Oracle database or any of the Data Guard processes fail. Note that since the primarydatabase and all the standby databases are configured in separate clusters, separate combinationalpackages must be created for each primary database and standby database.

Configuring an ODG Toolkit involves two scenarios:1. If a customer already has an ECMT Oracle Toolkit package running and wants to convert the

Oracle database into a Data Guard setup, then:

– If the ECMT Oracle package is a legacy-style package, it must be migrated to modular style. Youcan achieve this using the “cmmigratepkg” tool. Subsequently, a new combinational packagemust be created. This is because the ODG Toolkit does not support legacy-style packaging.However, migrating the legacy package to modular style is not the recommended way. Therecommended way is to discard the legacy package and create the Data Guard package afreshin modular style.

– If the ECMT Oracle package is a modular-style package, the Data Guard module can be insertedinto it by using the following command:

cmmakepkg –i <pkg_ascii_file> -m <module_file_name> <output_file_name>

where:

– pkg_ascii_file is the package file of the existing ECMT Oracle package. This can be generatedusing the command “cmgetconf –p <pkg_name> <output_filename>“

– module_file_name is the name of the module to be included in the running package. In case ofData Guard package, its value will be tkit/dataguard/dataguard

– output_file_name is the template file that gets generated with the values of the ECMT Oracledatabase module populated in it. The user must edit this file and enter values for the Data Guardspecific package attributes and then apply the package using the cmapplyconf command.

2. If the customer does not have an ECMT Oracle package running and wants to create the Data

Guard package afresh, then the command to create a combinational package is:

cmmakepkg –m ecmt/oracle/oracle –m tkit/dataguard/dataguard <pkg_file_name>

where:

– ecmt/oracle/oracle is the Oracle toolkit module shipped with ECMT Oracle Toolkit.

– tkit/dataguard/dataguard is the Data Guard toolkit module shipped with the ODG Toolkit.

– pkg_file_name is the template file that gets generated. The user needs to edit this file and entervalues for the ECMT Oracle specific package attributes and also for the Data Guard specificpackage attributes, and then apply the file using cmapplyconf command to create the package.

NOTE: The package parameter, “START_MODE” must be set to “mount” when an ECMT Oracle

Toolkit is used in combination with an ODG Toolkit.NOTE: In the case of Active Data Guard, the standby database will be started up to the “open” state.The user should set the “ACTIVE_STANDBY” parameter to “yes” when he wants to use this feature.

Active Data Guard is supported in Oracle database version 11gR1 or later.

7/30/2019 4AA2-7717ENW


6

Continentalclusters environment

Figure 2: ODG Toolkit in a Continentalclusters environment

Figure 2 depicts a typical Continentalclusters environment, which consists of a Primary Cluster (SGCluster 1) and a Recovery Cluster (SG Cluster 2). Even though both these clusters can have more thantwo nodes, we have used only two nodes in the clusters in our example. In the Primary Cluster, DataGuard is configured using the ODG Toolkit and the Oracle database is placed on a disk that isshared between the nodes in the Primary Cluster. The standby database is created in the RecoveryCluster and is also placed on a shared disk.

A Recovery group is created in the Continentalclusters environment with the following three packages

in it:

1. Primary package: This package is created using the ODG Toolkit on the Primary Cluster. Thispackage will bring up the Oracle database on the Primary Cluster as a primary database and willstart monitoring the primary database processes. If the primary database fails on Node 1, then thePrimary package will be failed over to another node within the Primary Cluster.

2. Data Receiver package: This package is created using the ODG Toolkit on the Recovery Cluster.This package will bring the Oracle database on the Recovery Cluster in Standby mode and willstart monitoring the standby database processes. If the standby database fails, then this packagewill halt the Oracle database and will failover the package to another node within the RecoveryCluster.

3. Recovery package: This package is also created using the ODG Toolkit on the Recovery Cluster. Itwill be configured to bring up the Oracle database on the Recovery Cluster in Primary mode.Initially, this package will be in halted state.

All the above three packages are configured as failover packages.

Continentalcluster

Node 1 Node2

SG Cluster 1( Primary Cluster ) SG Cluster 2( Recovery Cluster )

Primarypackage

DataReceiverpackage Recoverypackage

StandbyDB

PrimaryDB

Recovery group

Node 1 Node 2

7/30/2019 4AA2-7717ENW


7

Initially, the Primary package and the Data Receiver package will be up and running and theRecovery package will be in halted state. The Primary package will bring up the Oracle database onthe Primary Cluster in Primary mode and the Data Receiver package will bring up the Oracledatabase on the Recover Cluster in Standby mode. Thus, a typical Data Guard environment is set upwith data replication done from the primary database on the Primary Cluster to the standby databaseon the Recovery Cluster.

When the primary database fails, Serviceguard configured on the Primary Cluster fails over thedatabase to another node within the Primary Cluster, thus providing HA to the primary database.

Similarly, if the standby database fails, then Serviceguard configured on the Recovery Cluster allowsthe database to failover to another node within the Recovery Cluster.

When the Primary Cluster goes down, the administrator must run the “cmrecovercl” command on theRecovery Cluster to bring up the Recovery package. This command will first halt the Data Receiverpackage, which will halt the standby database, and then bring up the Recovery package, which willbring up the database as a primary database. Note that in this case, role management is handled byODG Toolkit.

Restoring the cluster in a Continentalclusters to its original state is a manual process. The followingsteps must be performed to restore the clusters back to their original state:

1. Halt the Recovery package.

2. Start the Primary package—this will bring up the database on Primary Cluster as a primarydatabase.

3. Start the Data Receiver package—this will bring up the database on the Recovery Cluster as astandby database.

In a Continentalclusters environment, the AUTO_RUN attribute of the packages must be disabled atthe time of bringing up the packages in the Recovery group. This is because Continentalclusters doesnot want the packages to be brought up when the cluster is brought up. But if this attribute is notenabled, the local failover of the package within the cluster at the time of package failure will nothappen. In such a case, package switching can be manually enabled using the “cmmodpkg”command after the packages in the Recovery group are started.

Multiple standby databases cannot be supported in a Continentalclusters setup. This is because therecan be only one Data Receiver package and only one Recovery package within a Recovery group.However, the ODG Toolkit in Continentalclusters will not prevent users from configuring other standbydatabases that are placed outside the Continentalclusters setup.

7/30/2019 4AA2-7717ENW


8

Metrocluster and EDC configurations

Metrocluster is an HP high availability product for Serviceguard customers who require integrateddisaster recovery solutions. HP Metrocluster uses Serviceguard clustering technology to form asingle cluster of systems that are located apart from each other at different data centers atmetropolitan distances.

While Metrocluster provides storage-based replication of data across metropolitan distances,organizations often like to build in additional data replicas, either locally or at other data centers.

Having both storage replication and ODG replication can help to safeguard customer data. Theintegration of ODG with Metrocluster or EDC gives organizations this choice.

Single-instance Data Guard configuration in Metrocluster

Figure 3: Single-instance Data Guard configuration in Metrocluster with standby database residing outside Metrocluster

Metrocluster

Node 1 Node 2 Node 3 Node 4

Data Center 1(Primary site)

Data Center 2(Recovery site)

Primarypackage

PrimaryDB

PrimaryDB

Data replication in Metrocluster

Node 1 Node 2

Third location

(Serviceguard cluster) Standbypackage

StandbyDB

Primary DB

StandbyDB

7/30/2019 4AA2-7717ENW


9

In Figure 3, Metrocluster is configured with two data centers: one as the primary site and the other asthe recovery site. The Data Guard primary database instance is configured in Data Center 1 such thatthe archived logs and the primary database reside on the disk that is shared locally between thenodes in that data center. The primary database instance will be brought up on any one node inData Center 1 while other nodes in the data center will serve as backup nodes for that instance.

Array-based replication technology enables the data written to the storage array in Data Center 1to be replicated to the storage array located in Data Center 2. It is not necessary to configurethe standby database in Data Center 2 because the data replication is taken care of by the

Metrocluster solution.The standby database instance is configured at a third location (say site 3), which is not configuredwithin the Metrocluster environment. The third location is a separate Serviceguard cluster with twonodes and a shared disk. The standby database instance will be brought up on any one node in thethird location, and the database and the archived logs will be located on the shared disk. If highavailability is not desired for the standby database then the third location need not be configured in aServiceguard cluster. In such a situation, there will be only one server at the third location and thestandby database instance has to be brought up manually.

In case of a disaster, if Data Center 1 is down, then the primary database instance will be failed overto Data Center 2 and will continue to function as a primary database. Since Metrocluster hasachieved data replication from Data Center 1 to Data Center 2, there will not be any problems when

starting the primary database at Data Center 2 using the replicated data from the shared disk. Oncethe primary database instance comes up at Data Center 2, it will continue to send the archived logs tothe standby database located at the third location.

Data Guard in EDC environment

Figure 4: ODG in EDC environment in which both the primary and the standby are single-instance databases

EDC


Package forprimary DB

Package forstandby DB

StandbyDB

PrimaryDB

Site A Site B

7/30/2019 4AA2-7717ENW


10

Figure 4 shows a Data Guard setup in an EDC environment. Both the primary and standby databasesare configured as single instance Serviceguard failover packages inside the EDC. All the four nodesare present inside the same Serviceguard cluster. Failover from Site A to Site B and vice versa isnot allowed.

When Site A goes down, the role of Standby on Site B has to be manually changed to primary, sincethe role transitions will not be supported in the first release.

Metrocluster in Continentalclusters environment

Figure 5: Single-instance Data Guard setup in a Continentalclusters environment where the Primary Cluster is configured as aMetroclusters

Figure 5 describes a Continentalclusters setup with two clusters spread over a total of three differentsites. The Primary Cluster is configured as a Metrocluster spread over two different sites that aregeographically dispersed within the confines of a metropolitan area. The Recovery Cluster isconfigured on the third site as a Serviceguard cluster.

The primary database of Data Guard setup is configured as a Primary package in the Primary Cluster(Metrocluster instance). The Primary Cluster has two data centers (sites), each of which have storagearray for storing the data. Metrocluster enables data written to the primary site to be replicated to therecovery site and hence the primary database instance can failover to any node within the PrimaryCluster.

Continentalcluster

Primary Cluster ( Metrocluster )


Data Center 1( Primary site )

Data Center 2( Recovery site )

Primarypackage

PrimaryDB

PrimaryDB

Data replication in Metrocluster

Recovery Cluster ( Third site )

Data Receiverpackage

StandbyDB

Primary DB

StandbyDB

Recoverypackage

Node 1 Node 2

7/30/2019 4AA2-7717ENW


11

The standby database is configured as a Data Receiver package in the Recovery Cluster. It willreceive the archived redo logs sent by the primary database instance irrespective of the node onwhich the primary database is running. Cross subnet failover is allowed across the sites.

When the Primary Cluster fails, the user must run the “cmrecovercl” command to halt the DataReceiver package and start the Recovery package. Once this command is run, the Recovery packagestarts and brings up the Oracle database as a primary database. Thus there will be no instance ofstandby database running once the role of the standby database is changed to primary. Multiplestandby configurations are not supported within a Continentalclusters setup. However, the ODG

Toolkit in Continentalclusters will not prevent customers from configuring standby databases that areplaced outside the Continentalclusters environment.

Supporting multiple instances of single-instance ODG configuration

Figure 6: Multiple Data Guard instances in one Serviceguard cluster

To support multiple Data Guard instances in one Serviceguard cluster, all the instances must beconfigured in such a way that they function independently. This means that they should have differentvolume groups for storing the database and redo logs. They should use different Oracle databaselisteners and IP addresses or ports.

Figure 6 shows a supported configuration where two two-node Serviceguard clusters are configuredto provide HA to the Data Guard configurations. Two single-instance Data Guard primary databasesare configured in Cluster 1 and one standby database is configured for each of the primarydatabases in Cluster 2. Note that both these clusters are independent of each other and there cannot

be package failover between the two clusters. Both the primary database in Cluster 1 and the standbydatabase in Cluster 2 and their corresponding redo logs are located on shared disks such that eachcan be accessed from any nodes in their respective clusters. This will enable failover of the primaryand standby packages within the cluster, thus providing HA for both of them.

HA for Data Guard Broker

HA for Data Guard Broker is not supported on a single-instance database.

Node 1

Node 2

SG Cluster 1 SG Cluster 2

Package 1 Package 4

StandbyDB

PrimaryDB

Primary DB Standby DB

Package 2

Primary DB

PrimaryDB Package 3

Standby DB

StandbyDB

Node 1 Node 2

7/30/2019 4AA2-7717ENW


12

Installation and configuration of the toolkit

Setting up the application

ODG is included with the Enterprise Edition and Standard Edition of the Oracle database software.Oracle must be installed on all the nodes of the cluster, by the user “oracle” and shared storage hasto be configured. For information on configuring ODG, please refer to Appendix A.

NOTE: In the event of package failover to the adoptive node, make sure that the database instanceon the adoptive node will be able to communicate in the Data Guard configuration. This requiresappropriate configuration of the listener and network services.4

Setting up the toolkit

This toolkit has to be used in combination with the ECMT Oracle module for a single-instance Oracledatabase. In this case, ensure that the ECMT B.06.00 has been installed.

After installing the ODG Toolkit, two scripts (hadg.sh and hadg.conf) and a README file will beinstalled in the /opt/cmcluster/toolkit/dataguard directory. Two more scripts (tkit_module.sh andtkit_gen.sh) and one file (dataguard.1), which are used for modular packaging, will be installed inthe /etc/cmcluster/scripts/tkit/dataguard directory and /etc/cmcluster/modules/tkit/dataguard

respectively.These scripts are:

- hadg.conf (user configuration file)

This script contains a list of predefined variables that the user must customize for use with a particulardatabase instance. This is a configuration file that is read by the toolkit script, hadg.sh. The followingvariables are contained in hadg.conf:

TKIT_DIR: This directory is synonymous with the package directory and holds the toolkitconfiguration file. This parameter directs “cmapplyconf” to generate the hadg.conf file under thisdirectory. To put the toolkit into maintenance mode, create the dataguard.debug file under thisdirectory.

ORACLE_HOME: The base directory of Oracle where it is installed.

ORACLE_ADMIN: User name of the Oracle database administrator. This will be used for startingand stopping the database.

For example:

ORACLE_ADMIN=oracle

SID_NAME: The Oracle session name. This uniquely identifies an Oracle database instance.

START_MODE: This parameter determines the startup mode for Oracle database. The default valueis “open.” It can take the options “nomount,” “mount,” or “open.”

NOTE: For the ODG Toolkit, always specify the value of this parameter as “mount.”MAINTENANCE_FLAG: The maintenance flag is used to bring this toolkit into maintenance mode. Ifset to “yes,” this will enable the maintenance feature in the toolkit. The Serviceguard Toolkit for ODGwill look out for a file “dataguard.debug” in the package directory. If the file exists and if themaintenance feature is enabled, monitoring is paused, and database instance may be brought downfor maintenance. The package will not be failed over to the adoptive node even though the instancehas been brought down for maintenance. After the maintenance work, make sure that the instance is

4 Listener and network services can be configured by using Oracle’s netmgr utility.

7/30/2019 4AA2-7717ENW


13

brought up properly. Delete the file “dataguard.debug” from the package directory. This will enablethe toolkit to continue monitoring the database server application.

NOTE: If the maintenance flag is set to “no,” then the above feature will not be available and thetoolkit cannot be brought into maintenance mode.

MONITOR_INTERVAL: The time interval, in seconds, this script will wait between checks to makesure that the Oracle instance is running. The default value is 30 seconds.

TIME_OUT: The time for which the toolkit waits for a completion of a normal shutdown before

initiating forceful halt of the application. The TIME_OUT variable is used to protect against aworst-case scenario where a hung database or ASM instance prevents the halt script from completing,therefore preventing the standby node from starting the instance. The TIME_OUT variable has noeffect on package failover times. The default value is 30 seconds.

ACTIVE_STANDBY: This parameter determines whether the database instance is an active standbyor not. The Active Data Guard Option available with Oracle Database 11g Enterprise Edition enablesyou to open a physical standby database for read-only access. This parameter can be set to “yes” or“no.” The default value is “no.”

DG_BROKER: Specifies whether the ODG broker is to be used or not. ODG Broker management isnot supported in this release hence this parameter must be set to “no.” This can take values “yes” or“no.” The default value is “no.”

START_STANDBY_AS_PRIMARY: Specifies whether the standby database has to be started as theprimary database by failing over the primary database. This parameter can be used in aContinentalcluster environment; the value of this parameter has to be “yes” in the packageconfiguration file of the recovery package.

ALERT_MAIL_ID: This parameter is used to specify the email address for sending alerts.

Main Script (hadg.sh)

This script contains a list of internally used variables and functions that support the starting andstopping of an ODG instance. This script will be called by tkit_module.sh to perform the following:

- On package startup, it starts the Data Guard instance (primary/standby) and launches monitor

processes.

- On package halt, it stops the Data Guard instance and monitor process.

NOTE: The following three scripts are used only during the modular method of packaging.

- Attribute Definition File (Data Guard)

The ADF is used to generate a package ASCII template file.

- Module Script (tkit_module.sh)

This script is called by the Master Control Script and acts as an interface between the Master ControlScript and the Toolkit interface script (hadg.sh). It is also responsible for calling the ToolkitConfiguration File Generator Script (described below).

- Toolkit Configuration File Generator Script (tkit_gen.sh)

This script is called by the Module Script when the package configuration is applied using“cmapplyconf” to generate the user configuration file (hadg.conf) in the package directory

(TKIT_DIR).

7/30/2019 4AA2-7717ENW


14

ODG package configuration example

This section explains an ODG package configuration example.

Follow the instructions in the chapter “Building an HA Cluster Configuration” in the manual“Managing HP Serviceguard“ to create the logical volume infrastructure on shared disks. The diskmust be available to all clustered nodes that will be configured to run the ODG Toolkit. Create filesystems on all logical volumes on the volume groups.

The ODG Toolkit can be configured in one of the following methods:

a) Install directory method: Here the scripts remain in the installation directory.

b) Configuration directory method: Here the user has to copy the scripts from the installation directory(including contents of subdirectories) to the configuration directory and define this location in theparameter “TKIT_DIR” in the package configuration file. The users can modify the scripts in theconfiguration directory to add any specific requirements. Serviceguard first tries to use the hadg.shscript in the configuration directory. If the script is not found in the configuration directory, it takes itfrom the installation directory.

The example configuration uses the installation directory mode operation. This example on ODGPackage Setup and Configuration is for an ODG configuration using LVM.

The example illustrates the creation of a package for ODG in a single-instance environment.a. Creating a package configuration

Create two packages: one for the primary database on the Primary Cluster and the other for thestandby database on the Standby Cluster.

Create a directory in /etc/cmcluster, for example, “dataguard.” This directory will eventually becomeTKIT_DIR and “cd” to this directory.

Run the following commands to create the package configuration file templates. Note that the ODGToolkit has to be used in combination with the ECMT Toolkit (in case of a single-instance database).

A combinational package is the one in which two applications are packaged together by combiningtheir respective Serviceguard modules into one package.

In the example below, the ODG Toolkit module has been used with the ECMT Oracle modules asfollows:

#cmmakepkg -m ecmt/oracle/oracle -m tkit/dataguard/dataguard dgpkg.conf

b. Specifying configuration parameters in the package

Once the package configuration file has been created, the user must specify various parametervalues. Only the parameters that are to be modified in dgpkg.conf specifically for this configurationare shown here.

Note that the package configuration file shown below contains attributes of both the ECMT OracleToolkit and the ODG Toolkit.

NOTE: The following attributes are specific to the Oracle Toolkit in ECMT

#

# “package_name” is the name that is used to identify the package.

# Package names must be unique within a cluster.

#

7/30/2019 4AA2-7717ENW


15

package_name dgpkg

------------------------------------------------------------------------

#

# “package_description” specifies the application that the package runs.

#

package_description “Serviceguard Package”

------------------------------------------------------------------------

#

# “package_type” is the type of package.

# The package_type attribute specifies the behavior for this package.

#

package_type failover

------------------------------------------------------------------------

#

# “run_script_timeout” is the number of seconds allowed for the package to start.

# “halt_script_timeout” is the number of seconds allowed for the package to halt.

#

run_script_timeout 300

halt_script_timeout 610

------------------------------------------------------------------------

# “script_log_file“ is the full path name for the package control script log

# file.

script_log_file /etc/cmcluster/dataguard/pkg.log

---------------------------------------------------------------------------

#

# Define package configuration directory

#

ecmt/oracle/oracle/TKIT_DIR /etc/cmcluster/dataguard

------------------------------------------------------------------------

#

# Define the instance type

#

ecmt/oracle/oracle/INSTANCE_TYPE database

------------------------------------------------------------------------

#

# Define Oracle home

7/30/2019 4AA2-7717ENW


16

#

ecmt/oracle/oracle/ORACLE_HOME /var/orahome

------------------------------------------------------------------------

#

# Define user name of Oracle database administrator

#

ecmt/oracle/oracle/ORACLE_ADMIN oracle

------------------------------------------------------------------------

#

# Define oracle session name

#

ecmt/oracle/oracle/SID_NAME ORCL

------------------------------------------------------------------------

#

# Define Oracle database startup mode

#

ecmt/oracle/oracle/START_MODE mount

------------------------------------------------------------------------

#

# Define whether the database instance is using ASM or not

#

ecmt/oracle/oracle/ASM no

------------------------------------------------------------------------

#

# Define ASM disk groups used by the database instance

#

#ecmt/oracle/oracle/ASM_DISKGROUP

-----------------------------------------------------------------------

#

# Define the volume groups used in the ASM disk groups for the

# database instance

#

#ecmt/oracle/oracle/ASM_VOLUME_GROUP

------------------------------------------------------------------------

#

# The ASM home

7/30/2019 4AA2-7717ENW


17

#

#ecmt/oracle/oracle/ASM_HOME

------------------------------------------------------------------------

#

# Define user name of Oracle ASM administrator

#

ecmt/oracle/oracle/ASM_USER oracle

----------------------------------------------------------------------

#

# The ASM session name

#

#ecmt/oracle/oracle/ASM_SID

------------------------------------------------------------------------------

#

# Define whether the configured listener has to be started with the server

#

ecmt/oracle/oracle/LISTENER yes

------------------------------------------------------------------------

#

# Define the Oracle listener name(s)

#

ecmt/oracle/oracle/LISTENER_NAME LISTENER_ORCL

------------------------------------------------------------------------

#

# Define the listener password(s)

#

#ecmt/oracle/oracle/LISTENER_PASS

------------------------------------------------------------------------

#

#ecmt/oracle/oracle/LISTENER_RESTART

#

#ecmt/oracle/oracle/LISTENER_RESTART

------------------------------------------------------------------------

NOTE: The following are the service commands for the combinational package.

service_name oracle_service_test

service_cmd “$SGCONF/scripts/ecmt/oracle/tkit_module.sh oracle_monitor”

7/30/2019 4AA2-7717ENW


18

service_restart none

service_fail_fast_enabled no

service_halt_timeout 300

service_name oracle_listener_service_test

service_cmd “$SGCONF/scripts/ecmt/oracle/tkit_module.sh oracle_monitor_listener”




service_name oracle_hang_service_test

service_cmd “$SGCONF/scripts/ecmt/oracle/tkit_module.sh oracle_hang_monitor 30 failover”




service_name dataguard_service_test

service_cmd “$SGCONF/scripts/tkit/dataguard/tkit_module.sh dataguard_monitor“




NOTE: The following attributes are specific to the ODG Toolkit.

------------------------------------------------------------------------

#

# Define ACTIVE_STANDBY

#

tkit/dataguard/dataguard/ACTIVE_STANDBY no

------------------------------------------------------------------------

#

# Define DG_BROKER

#

tkit/dataguard/dataguard/DG_BROKER no

------------------------------------------------------------------------

#

# Define START_STANDBY_AS_PRIMARY

#

tkit/dataguard/dataguard/START_STANDBY_AS_PRIMARY no

------------------------------------------------------------------------

#

7/30/2019 4AA2-7717ENW


19

# Define email address for sending alerts

#

#tkit/dataguard/dataguard/ALERT_MAIL_ID

------------------------------------------------------------------------

#

# “vg” is used to specify which volume groups are used by this package.

#

vg vgora

------------------------------------------------------------------------

#

# “fs_name,” “fs_directory,” “fs_mount_opt,” “fs_umount_opt,” “fs_fsck_opt,”

# and “fs_type” specify the filesystems that are used by this package.

#

fs_name /dev/vgora/lvol1

fs_directory /oradb

fs_type “vxfs”

fs_mount_opt “-o rw”

#fs_umount_opt

#fs_fsck_opt

------------------------------------------------------------------------

Adding the package to the cluster

After the setup is complete, add the package to the Serviceguard cluster and start it up.

$ cmapplyconf -P dgpkg.conf

$ cmmodpkg -e -n <node1> -n <node2> dgpkg

$ cmmodpkg -e dgpkg

For more information on adding a package to the cluster, see the “Managing HP Serviceguard”document available at:


ODG maintenance

There might be situations where the ODG database instances have to be taken down for maintenancepurposes like changing configuration without having the instance migrate to standby node. Thefollowing procedure should be followed:

The toolkit maintenance feature is enabled only when the configuration variableMAINTENANCE_FLAG is set to “yes” in the toolkit configuration file. Note that the toolkitmaintenance feature is different from the Serviceguard’s package maintenance feature.




7/30/2019 4AA2-7717ENW


20

Note: The example assumes that the package name is dgpkg, package directory is

/etc/cmcluster/pkg/dgpkg, and the ORACLE_HOME is configured as /orahome.

1. Disable the failover of the package through cmmodpkg command.

$ cmmodpkg -d dgpkg

2. Pause the monitor script.

Create an empty file /etc/cmcluster/pkg/dgpkg/dataguard.debug as shown below:

$ touch /etc/cmcluster/pkg/dgpkg/dataguard.debug

Toolkit monitor scripts (both database instance and listener monitoring scripts), which continuouslymonitor ODG process daemons’ processes, would now stop monitoring these daemon processes. Themessages, “Serviceguard Toolkit for ODG pausing Data Guard monitoring and entering maintenancemode” appears in the Serviceguard Package log file.

3. If required, stop the database instance(s) as shown below:

$ cd /etc/cmcluster/pkg/dgpkg/

$ $PWD/hadg.sh stop

4. Perform maintenance actions (Example: Database maintenance).

5. Restart the Oracle database instance again if you manually stopped it prior to maintenance.

$ cd /etc/cmcluster/pkg/dgpkg/

$ $PWD/hadg.sh start

6. Allow monitoring scripts to continue normally as shown below:

$ rm -f /etc/cmcluster/pkg/dgpkg/dataguard.debug

The message “Starting Oracle Data Guard monitoring again after maintenance” appears in theServiceguard Package Control script log.

7. Enable the package failover.

$ cmmodpkg -e dgpkg

Note: If a package failure occurs during maintenance operations, the package does notautomatically failover to an adoptive node. You must manually start the package on the adoptivenode. For more information on manually starting the package on an adoptive node, see the“Managing HP Serviceguard” guide available at http://www.hp.com/go/hpux-serviceguard-docs

Troubleshooting

This section provides the guidelines to verify if the ODG package has been configured properlyor not.

The following steps help the user to troubleshoot the possible causes for the package failure:1. Verify ODG setup:

To verify a Data Guard configuration, check if the standby database is able to receive the redo logs. Also check if the RFS process is running on the standby database by querying the table“V$MANAGED_STANDBY.” If the standby site is not receiving the logs, obtain information about thearchiving status of the primary database by querying the V$ARCHIVE_DEST view. Check especiallyfor error messages.





7/30/2019 4AA2-7717ENW


21

For example, enter:

SQL> SELECT dest_id “ID”

2> status “DB_status”

3> destination “Archive_dest”

4> error “Error”

5> FROM v$archive_dest;

ID DB_status Archive_dest Error

-- --------- ------------------------------

--------------------------------------

1 VALID /vobs/oracle/work/arc_dest/arc

2 ERROR standby1 ORA-16012: Archivelog standby database

identifier mismatch

3 INACTIVE

4 INACTIVE

5 INACTIVE

5 rows selected

If the output of the query does not help, check the following list of possible issues.

- The service name for the standby instance is not configured correctly in the tnsnames.ora file at theprimary site.

- The service name listed in the LOG_ARCHIVE_DEST_n parameter of the primary initializationparameter file is incorrect.

- The LOG_ARCHIVE_DEST_STATE_n parameter specifying the state of the standby archivingdestination has the value DEFER.

- The listener.ora file has not been configured correctly at the standby site.

- The listener is not started.

- The standby instance is not started.

- You have added a standby archiving destination to the primary initialization parameter file, buthave not yet enabled the change.

- You used an invalid backup as the basis for the standby database (for example, you used a backupfrom the wrong database, or did not create the standby control file using the correct method).

Also, check the Oracle Alert log for errors.

2. Verify the Toolkit setup:Check whether the package configuration attributes specified in the package configuration file arevalid or not.

Known problems and workarounds

1. If the replication does not happen from the primary database to the standby database then checkif the online redo log files are configured or not on the primary as well as the standby.

7/30/2019 4AA2-7717ENW


22

2. In case of a Metrocluster environment where only the primary database is configured within theMetrocluster and the standby database is configured on a third site that is not part of theMetrocluster, the data replication can happen only from the site where the primary is running.Even if the proximity of the other site is better, data replication cannot happen from that site. Thisis a restriction put by Data Guard, and not the ODG Toolkit.

3. The Data Guard broker should not be configured for automatic role transitions. In case ofContinentalclusters scenarios, the role management is handled by the ODG Toolkit and in othercases, if the role of any database is changed, then the ODG Toolkit package will fail.

4. There are use-cases in which only the primary database is configured in a SG cluster and thestandby is not part of any cluster. This is similar to configuring ECMT Toolkit, which can be rundirectly by the user. Legacy-type support is required to support this feature. ODG Toolkit does notsupport legacy-style configuration.

5. Multiple standby database instances cannot be supported in the Continentalclusters setup sincethere can be only one Recovery package and only one Data Receiver package within theRecovery group. Other standby instances can be configured outside the CC environment.

For more information

To learn more, see:

• HP Serviceguard Solutions: http://www.hp.com/go/serviceguardsolutions

• Technical Documentation: www.hp.com/go/hpux-serviceguard-docs

Appendix A

Configuring ODG example

ODG is included with the Enterprise Edition and Personal Edition of the Oracle database software.

The standby database can be created in two ways:

• Physical standby database

• Logical standby database

Configuring the physical standby database in a single-instance Oracle database is described in thissection. Details about creating a logical standby database can be found at the Oracle Web site(http://download.oracle.com/docs/cd/E11882_01/server.112/e10700/create_ls.htm).

If customer already has a database and wants to make use of Data Guard for replication to anotherstandby site, it is the customer’s responsibility to set up the standby in the same state as the primary.Customers have to use third party or Oracle-supplied tools to back up and copy the existing databaseto the standby site. Data Guard replicates only updates to the database.

The following steps describe how to configure the Data Guard. Note that these are generic steps toconfigure the Data Guard. Detailed information about configuring Data Guard can be obtained from

the Oracle Web site (http://www.oracle.com).

Preparing the primary database

Before creating the standby database, the primary database must be configured first. The followingsteps should be followed to create the primary database:

1. Enable force logging:Place the primary database in FORCE LOGGING mode after the database creation using the

http://www.hp.com/go/serviceguardsolutions






http://download.oracle.com/docs/cd/E11882_01/server.112/e10700/create_ls.htm



http://www.oracle.com/







7/30/2019 4AA2-7717ENW


23

following SQL statement:SQL> ALTER DATABASE FORCE LOGGING;

2. Create a password file:Create a password file if one does not already exist. Every database in a Data Guardconfiguration must use a password file, and the password for the SYS user must be identical onevery system for redo data transmission to succeed.

3. Setting the primary database initialization parameters:On the primary database, define initialization parameters that control log transport services while

the database is in the primary role.4. Enable archiving:

If archiving is not enabled, issue the following statements to put the primary database in ARCHIVELOG mode and enable automatic archiving:

SQL> SHUTDOWN IMMEDIATE;

SQL> STARTUP MOUNT;

SQL> ALTER DATABASE ARCHIVELOG;

SQL> ALTER DATABASE OPEN;

5. It is recommended that the online redo log files are created on the shared disk so that when alocal failover of the primary database instance occurs, the online redo logs are also accessible to

the new instance on the adoptive node. Here’s a sample SQL statement that adds a new group ofredo logs to the database:

SQL> ALTER DATABASE ADD LOGFILE (“<oracle_redo_file_1>.rdo,”“<oracle_redo_file_2>.rdo”) SIZE 500K;

Creating a physical standby database

1. Create a backup copy of the primary database data files:You can use any backup copy of the primary database to create the physical standby database aslong as you have the necessary archived redo log files to completely recover the database. Oraclerecommends that you use the Recovery Manager utility (RMAN).

2. Create a control file for the standby database:

If the backup procedure required you to shut down the primary database, issue the followingSQL*Plus statement to start the primary database:

SQL> STARTUP MOUNT;Then create the control file for the standby database, and open the primary database to useraccess, as shown in the following example:

SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS “/tmp/standby.ctl”;

SQL> ALTER DATABASE OPEN;

3. Prepare an initialization parameter file for the standby database.

4. Copy files from the primary system to the standby system:Use an operating system copy utility to copy the following files from the primary system to thestandby system:

Backup data files

Standby control file

Initialization parameter file

5. Set up the environment to support the standby database:

a. Create a password file.Create a password file, and set the password for the SYS user to the same password used bythe SYS user on the primary database. The password for the SYS user on every database ina Data Guard configuration must be identical for redo transmission to succeed.

7/30/2019 4AA2-7717ENW


24

b. Configure listeners for the primary and standby databases. On both the primary and standbysites, use Oracle Net Manager to configure a listener for the respective databases.To restart the listeners (to pick up the new definitions), enter the following LSNRCTL utilitycommands on both the primary and standby systems:

% lsnrctl stop

% lsnrctl start

Ensure that the tnsnames.ora of a node contains the entries of all those nodes that it wants tocommunicate with.

c. Enable broken-connection detection on the standby system.Enable broken-connection detection by setting theSQLNET.EXPIRE_TIME parameter to twominutes in the SQLNET.ORA parameter file on the standby system. For example:

SQLNET.EXPIRE_TIME=2

d. Create Oracle Net service names.On both the primary and standby systems, use Oracle Net Manager to create a networkservice name for the primary and standby databases that will be used by log transportservices.The Oracle Net service name must resolve to a connect descriptor that uses the same protocol,host address, port, and SID that you specified when you configured the listeners for the primaryand standby databases. The connect descriptor must also specify that a dedicated server be

used.e. Create a server parameter file for the standby database.

On an idle standby database, use the SQL CREATE statement to create a server parameter filefor the standby database from the text initialization parameter file

SQL> CREATE SPFILE FROM PFILE=“initstandby.ora”;6. Start the physical standby database:

Perform the following steps to start the physical standby database and Redo Apply.

a. Start the physical standby database.On the standby database, issue the following SQL statements to start and mount the databasein read-only mode:

SQL> STARTUP OPEN READ ONLY;

b. Create a new temporary file for the physical standby database.Creating a new temporary file on the physical standby database at this point rather than lateris beneficial. Temporary files enable disk sorting when the database is open in read-only modeand prepare the database for future role transitions.To add temporary files to the physical standby database, perform the following tasks:1. Identify the table spaces that should contain temporary files. Do this by entering thefollowing command on the standby database:

SQL> SELECT TABLESPACE_NAME FROM DBA_TABLESPACES

2> WHERE CONTENTS = “TEMPORARY”;

TABLESPACE_NAME--------------------------------

TEMP1

TEMP2

Add new temporary files to the standby database.

7/30/2019 4AA2-7717ENW


25

For each table space identified in the previous query, add a new temporary file to the standbydatabase. The following example adds a new temporary file called TEMP1 with size andreuse characteristics that match the primary database temporary files:

SQL> ALTER TABLESPACE TEMP1 ADD TEMPFILE

2> “/arch1/standby/temp01.dbf”

3> SIZE 40M REUSE;

c. Start Redo Apply.

On the standby database, issue the following command to start Redo Apply:SQL> ALTER DATABASE RECOVER MANAGED STANDBYDATABASE DISCONNECT FROM SESSION;

This statement automatically mounts the database. Also, the statement includes theDISCONNECT FROM SESSION option so that Redo Apply runs in a background session.

d. Test archival operations to the physical standby database.The transmission of redo data to the remote standby location does not occur until after a logswitch. A log switch occurs by default when an online redo log file becomes full. To force a logswitch so that redo data is transmitted immediately, use the following ALTER SYSTEM statementon the primary database. For example:

SQL> ALTER SYSTEM SWITCH LOGFILE;

7.

Verify that the physical standby database is performing properly:To see that redo data is being received on the standby database, you should first identify theexisting archived redo log files on the standby database, force a log switch and archive a fewonline redo log files on the primary database, and then check the standby database again. Thefollowing steps show how to perform these tasks.

a. Identify the existing archived redo log files.On the standby database, query the V$ARCHIVED_LOG view to identify existing files in thearchived redo log. For example:

SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME

2 FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

b. Force a log switch to archive the current online redo log file.

On the primary database, issue the ALTER SYSTEM ARCHIVE LOG CURRENT statement toforce a log switch and archive the current online redo log file group:SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

c. Verify that the new redo data was archived on the standby database.On the standby database, query the V$ARCHIVED_LOG view to verify the redo data wasreceived and archived on the standby database:

SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME

2> FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

d. Verify new archived redo log files were applied.On the standby database, query the V$ARCHIVED_LOG view to verify the archived redo logfiles were applied.

SQL> SELECT SEQUENCE#, APPLIED FROM V$ARCHIVED_LOGORDER BY SEQUENCE#;

7/30/2019 4AA2-7717ENW


Further preparations

At this point, the physical standby database is running and can provide the maximum performancelevel of data protection. The following list describes additional preparations you can take on thephysical standby database:

1. Upgrade the data protection mode:The Data Guard configuration is initially set up in the maximum performance mode (the default).This can be change to either maximum protection mode or maximum availability mode.

2. Configure standby redo logs:Standby redo logs are required for standby databases running in the maximum protection modeand maximum availability mode. However, configuring standby redo logs is recommended on allstandby databases, because during a failover, Data Guard can recover and apply more redodata from standby redo log files than from the archived redo log files alone. The standby redologs should exist on both primary and standby databases and have the same size and names.

3. Enable Flashback Database:Flashback Database removes the need to re-create the primary database after a failover.Flashback Database is similar to conventional point-in-time recovery in its effects, enabling you toreturn a database to its state at a time in the recent past. Flashback Database is faster than point-in-time recovery because it does not require restoring data files from backup or the extensive

application of redo data. You can enable Flashback Database on the primary database, thestandby database, or both.

HP welcomes your input. Please give us comments about this white paper, or suggestions for relateddocumentation, through our technical documentation feedback website:http://www.hp.com/bizsupport/feedback/ww/webfeedback.html

To learn more about the HP Serviceguard Toolkit for Oracle Data Guard, visit:http://www.hp.com/go/sgoracledg

Share with colleagues

© Copyright 2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject tochange without notice. The only warranties for HP products and services are set forth in the express warrantystatements accompanying such products and services. Nothing herein should be construed as constituting anadditional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

http://www.hp.com/bizsupport/feedback/ww/webfeedback.html


http://www.hp.com/go/sgoracledg


http://digg.com/submit?phase=2&url=http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA2-7717ENW&title=Configuring%20HP%20Serviceguard%20Toolkit%20for%20Oracle%20Data%20Guard%20(US%20English)

http://www.linkedin.com/shareArticle?mini=true&ro=true&url=http%3A%2F%2Fh20195%2Ewww2%2Ehp%2Ecom%2FV2%2FGetDocument%2Easpx%3Fdocname%3D4AA2-7717ENW&title=Configuring+HP+Serviceguard+Toolkit+for+Oracle+Data+Guard+(US+English)+&armin=armin

http://www.facebook.com/sharer.php?u=http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA2-7717ENW

http://twitter.com/home/?status=Configuring%20HP%20Serviceguard%20Toolkit%20for%20Oracle%20Data%20Guard%20(US%20English)+%40+http%3A%2F%2Fh20195.www2.hp.com%2FV2%2FGetDocument.aspx?docname=4AA2-7717ENW

http://www.hp.com/go/getconnected



Documents

4AA2-7717ENW