Maximum Availability Architecture - Oracle Cloud · Oracle White Paper—Oracle Beehive Maximum Availability Architecture 3 Ancillary Tier, which contains optional Oracle and third

Oracle Beehive Maximum Availability Architecture: Oracle Maximum Availability Architecture White Paper June 2010

Maximum Availability Architecture Oracle Best Practices For High Availability

Oracle White Paper—Oracle Beehive Maximum Availability Architecture

Introduction ....................................................................................... 1

Oracle Beehive Architecture .............................................................. 2

Oracle Beehive Tiers ..................................................................... 2

Determining Your Oracle Beehive High Availability Requirements ..... 3

Oracle Beehive High Availability Architectures .................................. 4

Oracle Beehive with Oracle Real Application Clusters ................... 6

Oracle Beehive with a Secondary Disaster Recovery Site ............. 7

Oracle Beehive Disaster Recovery Plus a Local HA Standby ........ 8

Planned and Unplanned Outage Solutions ...................................... 11

Planned Outages ......................................................................... 12

Unplanned Outages ..................................................................... 13

Deploying Oracle Beehive MAA....................................................... 14

Oracle Database MAA ................................................................. 14

Oracle Beehive Application High Availability Deployment ............ 15

Oracle Beehive Site Target in Oracle Enterprise Manager ........... 17

Oracle Beehive Disaster Recovery Site Deployment ................... 17

Local HA Standby Deployment .................................................... 18

Conclusion ...................................................................................... 18

Appendix ......................................................................................... 19

Example Test Environment .......................................................... 19

Install the first Oracle Beehive Instance ....................................... 20

Post-install Verification ................................................................ 20

Port Configuration ........................................................................ 22

Configure TLS and the XMPP Service ......................................... 23

Sanity Test Install ........................................................................ 25

Load Balancer Configuration ....................................................... 26

Sanity Test with Load Balancer ................................................... 28


Configure ONS ............................................................................ 29

Configure Oracle Beehive Database Services ............................. 30

Clone the Application Tier ............................................................ 33

Oracle Beekeeper Install ............................................................. 38

Oracle Beehive Disaster Recovery Site Setup ............................. 39

Test Role Transitions ................................................................... 48

Oracle Beehive Local HA Standby Setup.................................... 50

Database Configuration Files ...................................................... 52

Create Users Script ..................................................................... 57

Oracle RAC FAN Callout Start Services Script ............................ 60

References ...................................................................................... 63


1

Introduction Oracle Maximum Availability Architecture (MAA) is Oracle's best practices blueprint based on proven Oracle high availability technologies and recommendations. The goal of MAA is to achieve the optimal high availability architecture at the lowest cost and complexity. Papers are published on the Oracle Technology Network (OTN) - http://www.otn.oracle.com/goto/maa.

This paper describes the Oracle Beehive Maximum Availability Architecture (MAA), a best practice blueprint for achieving an optimal Oracle Beehive high availability deployment using Oracle high availability technologies and recommendations.

To achieve Oracle Beehive maximum availability, apply the following high-level recommendations:

1. Deploy the Oracle Beehive database in an Oracle Database MAA configuration.

2. Deploy the Oracle Beehive application in an Oracle Beehive high availability (HA) configuration.

3. Establish a secondary Oracle Beehive site for disaster recovery, testing, and planned maintenance activities.

4. Review the benefits and feasibility of implementing a local HA standby deployment.

This white paper describes each recommendation in detail and the solutions that are available for planned and unplanned outages. The Appendix shows a detailed example of deploying an Oracle Beehive MAA configuration.

This document was written in the context of Oracle Beehive Version 1.5 running on Oracle Database 11g Release 1 (11.1) and, where necessary, this paper references the documentation for these product releases. In most cases, the recommendations and best practices can be applied to later releases, too. If you are running later product versions refer instead to the appropriate version of the product documentation.

http://www.otn.oracle.com/goto/maa�


2

Oracle Beehive Architecture

Built on Java 2 Platform Enterprise Edition (J2EE), Oracle Beehive provides a multitier architecture that leverages proven Oracle technologies, such as Oracle Database and Oracle Application Server, as well as other key Oracle and third-party components.

Oracle Beehive Tiers

The Oracle Beehive architecture is divided into the following logical tiers:

● Client Tier contains Oracle Beehive clients as well as other supported clients and devices, including integration with Microsoft Outlook and a command-line tool for system administration. Oracle Beehive also supports clients and devices that leverage the following standardized protocols: Calendaring Extensions for WebDAV (CalDAV), Extensible Messaging and Presence Protocol (XMPP), File Transfer Protocol (FTP), Internet Message Access Protocol (IMAP), Push Internet Message Access Protocol (P-IMAP), Open Mobile Alliance Data Synchronization (OMA-DS), Simple Mail Transfer Protocol (SMTP), and Web-based Distributed Authoring and Versioning (WebDAV). The open nature of the Oracle Beehive platform also enables developers to build and implement custom clients in the Client Tier using Oracle Beehive Web services.

● Application Tier contains Oracle Beehive services and application server components. The Application Tier is the core of the architecture and includes all Oracle Beehive server components, including interoperable, function-specific services that provide the system's enterprise collaboration features. The Application Tier supports multiple Oracle Beehive server instances. Each Oracle Beehive server instance includes required components of Oracle Application Server 10g, which itself hosts the Oracle Beehive services, including:

• Oracle HTTP Server (OHS): The Web server component of Oracle Application Server 10g. Enables connections between supported clients over HyperText Transport Protocol (HTTP) and Secure HyperText Transport Protocol (HTTPS).

• Oracle Application Server Containers for J2EE (OC4J): J2EE v1.4 -compliant containers that provide an infrastructure for deploying, un-deploying, and redeploying J2EE-compliant applications and modules. Oracle Beehive services are deployed in OC4J containers.

● Data Tier contains the Oracle Database to store system configuration and collaboration data. The Data Tier provides Oracle Database a layer of separation from the other tiers, ensuring, among other things, optimized security and system performance. Only the Database Access Framework (provided by Oracle Beehive) can access the Data Tier. Oracle Beehive services cannot access the Data Tier directly and must make all connections through the schemas provided by the Database Access Framework. For more information on the Database Access Framework, see the Oracle Beehive Concepts [4] guide.


3

● Ancillary Tier, which contains optional Oracle and third-party components that enhance or coexist with certain aspects of the system. Typically, components in this tier are optional because Oracle Beehive already provides many of these capabilities, such as user directories, e-mail, and time management.

For a more detailed description of each tier see the Oracle Beehive Concepts4 [4] guide.

This paper focuses on the MAA deployment in the Application and Data tiers These tiers are outlined in red in Figure 1.

Figure 1 [*Oracle Beehive Concepts]. Oracle Beehive Logical Architecture

Determining Your Oracle Beehive High Availability Requirements

An analysis of business requirements for high availability combined with an understanding of the level of investment required to implement different high availability solutions enables the development of a high availability architecture that will achieve both business and technical objectives. The Oracle Database High Availability Overview [2] guide has a section on “Determining Your High Availability Requirements” that details an analysis framework for determining your HA requirements. A summary of the framework elements:

● Business Impact Analysis This categorizes the business processes based on the severity of the impact of IT-related outages.


4

● Cost of Downtime The business impact analysis provides the insight needed to quantify the cost of unplanned and planned downtime. This cost helps prioritize your HA investment and directly influences the HA technologies that you choose to minimize the downtime risk.

● Recovery Time Objective (RTO) The maximum amount of time that an IT-based business process can be down before the organization starts suffering unacceptable consequences. RTO indicates the downtime tolerance of a business process or an organization in general.

● Recovery Point Objective (RPO) The maximum amount of data that an IT-based business process may lose without harm to the organization. RPO indicates the data-loss tolerance of a business process or an organization in general. This data loss is often measured in terms of time, for example, five hours or two days of data loss.

● Manageability Goal The measurement of the organization's tolerance for complexity in the IT environment. When less complexity is a requirement, simpler methods of achieving high availability are preferred over methods that may be more complex to manage.

● Total Cost of Ownership (TCO) and Return on Investment (ROI) The TCO and ROI are essential to selecting a high availability architecture that also achieves the business goals of your organization. TCO includes all costs (such as acquisition, implementation, systems, networks, facilities, staff, training, and support), over the useful life of the solution chosen. Likewise, the ROI calculation captures all of the financial benefits that accrue to a given high availability architecture.

It is recommended that you understand these elements of the “Analysis Framework for Determining High Availability Requirements” and follow this framework to determine your HA requirements for your Oracle Beehive implementation.

This paper outlines each of the HA architectures and then details the deployment of the “Oracle Beehive Disaster Recovery Plus a Local HA Standby” architecture.

Oracle Beehive High Availability Architectures

Installing Oracle Beehive in a high availability environment involves installing multiple Oracle Beehive instances, installing a third-party load balancer, and configuring the virtual server of each Oracle Beehive instance. Configuring the virtual server ensures that there is a single point of access to your Oracle Beehive application tier. This section describes each of the high availability database architecture options that work with this highly available Oracle Beehive application-tier configuration.


5

All MAA deployment options use multiple Oracle Beehive instances with a load balancer front end. The Oracle Beehive application tier is built on the Oracle Application Server and therefore inherits the Oracle Application Server High Availability Framework features:

● Process management through the Oracle Process Management and Notification (OPMN) Server

o When OPMN detects Oracle Application Server processes that are down, unresponsive, or unreachable, it automatically restarts them. It monitors the processes by pinging them or through notification methods.

o OPMN starts processes in the proper order: processes that depend on other processes are not started up until the dependent processes are started.

● Server load balancing and failover

Additionally, Oracle Beehive makes use of the JDBC Fast Connection Failover (FCF) feature of the Implicit Connection Cache and fully integrates Fast Application Notification (FAN) features.

The benefits you can derive from using Oracle Beehive and FAN and FCF are:

● Failed connections are retried and, if not usable, a good connection is obtained from the connection pool.

● Connection failover delays attributed to standard TCP timeouts are avoided.

● Instead of waiting for the application tier to poll the database tier and find a problem, the Oracle Beehive application tier receives notification of these events (as a result of using FAN and the Oracle Notification Service (ONS)) and reacts immediately to the event.

● Oracle RAC node failures or Oracle RAC resource reorganizations are automatically managed at the application tier.

● Oracle Beehive is balanced across the Oracle RAC database nodes through database services.

Configuring Oracle Beehive to take full advantage of FAN and FCF is detailed in the “Configure Oracle Beehive Database Services” section. Also, see the Oracle Real Application Cluster Administration and Deployment Guide [4] for more information about FCF and FAN.

Optionally, Oracle Beehive can also deploy separate Oracle Beehive Web servers that contain the Oracle HTTP Server in the demilitarized zone (DMZ). The DMZ is a neutral zone between its internal network, private network, and the Internet. A DMZ prevents unauthorized users from accessing systems that contain confidential or proprietary data. In effect, DMZs act as proxy servers and provide another layer of network security, especially when used with firewalls. Use of the DMZ will not be depicted in all options but a DMZ can be deployed with any of these options.


6

Oracle Beehive with Oracle Real Application Clusters

Oracle RAC combine the processing power of multiple interconnected computers to provide system redundancy, scalability, and high availability. Oracle Database with Oracle RAC architecture provides the following benefits:

● Scalability across database instances

● Flexibility to increase processing capacity using commodity hardware, without downtime or changes to the application

● Ability to tolerate and quickly recover from computer and instance failures (measured in seconds)

● Rolling upgrades for system and hardware changes

● Rolling patch upgrades for some interim patches, security patches, CPUs, and cluster software

● Fast, automatic, and intelligent connection and service relocation and failover

● Load balancing advisory and Runtime Connection Load Balancing

● Comprehensive manageability integrating database and cluster features with Grid Plug and Play and policy-based cluster and capacity management

● Workload management, load balancing advisory, and runtime connection load balancing help redirect and balance work across the appropriate resources

● Quality of Service Management reduces the number of outages due to performance problems by identifying and advising on problems that can breach performance objectives

● Oracle Enterprise Management support for Oracle Automatic Storage Management (Oracle ASM) and Cluster File System, Grid Plug and Play, Cluster Resource Management, Oracle Clusterware and Oracle RAC Provisioning and Patching

Figure 2 shows Oracle Beehive deployed with Oracle RAC. This figure shows an Oracle Database with Oracle RAC architecture for a partitioned two-node database. An Oracle RAC database is connected to two instances on different nodes. Each database instance is associated with a set of Oracle Beehive services from each Oracle Beehive Application Server. The database instances monitor each other by checking "heartbeats" on the Oracle RAC interconnect network. Oracle Net provides access from the Application Tier to the Data Tier.


7

Figure 2 Oracle Beehive with Oracle RAC Deployment

Oracle Beehive with a Secondary Disaster Recovery Site

You can achieve the highest level of availability when using Oracle RAC and Oracle Data Guard. Oracle Beehive database connect strings are configured to use these Oracle Database features. The combination of Oracle RAC and Oracle Data Guard provides the most comprehensive architecture for reducing downtime for scheduled outages and preventing, detecting, and recovering from unscheduled outages. This architecture is the recommended configuration for MAA.

To protect against site failures, the MAA recommends that Oracle RAC and Oracle Data Guard reside on separate systems (clusters) and data centers. Figure 3 shows the recommended MAA configuration, with Oracle Database, Oracle RAC, and Oracle Data Guard. Configuring symmetric sites is recommended to ensure that each site can accommodate the performance and scalability requirements of the application after any role transition. Furthermore, operational practices across role transitions are simplified when the sites are symmetric.


8

Note that Figure 3 does not show a DMZ.

Oracle Beehive has no requirement to synchronize application-tier files unless the application-tier logs are needed at the remote site.

Figure 3 Oracle Beehive with a Remote Standby Database on the DR Site

Oracle Beehive Disaster Recovery Plus a Local HA Standby

This architecture is identical to the single-standby database architecture that was described above under “Oracle Beehive with a Secondary Disaster Recovery Site”, except that there is a local standby database in the same Oracle Data Guard configuration that can use the same application-tier nodes in the event it assumes the primary database role.


9

Figure 4 Oracle Beehive with a Local and Remote Standby

The following list describes some of the benefits of a local HA standby database architecture:

● Continuous and transparent disaster or high availability protection if an outage occurs at the primary database because the same application tier nodes can be used with the local standby, and they can remain running and will automatically reconnect

● Improved return on investment with the use of Active Data Guard to facilitate reporting against the local standby

● Synchronous redo transport that transmits to a more local standby database, and asynchronous redo transport that transmits to a more remote standby database for optimum levels of performance and data protection

● Offloading of backups to the local and remote standbys

● Testing and development clones using snapshot standby databases

● Rolling upgrades (this will require Oracle Database 11g release 2 (11.2), which will have the SecureFiles data type support with a logical standby)


10

The following table summarizes and highlights the benefits of using a local standby database in terms of planned outages.

TABLE 1: USING A LOCAL STANDBY DATABASE TO REDUCE PLANNED DOWNTIME

PLANNED MAINTENANCE ORACLE SOLUTION DOWNTIME

Database Interim Patch,

Database Hardware

Upgrade, or Operating

System Upgrade

Oracle RAC & ASM(11G) Rolling Upgrade No downtime

Local Standby Switchover Seconds (Transparent)

Remote Site Switchover Minutes

1. Shutdown

2. Upgrade

3. Startup

Minutes

Database Patch Set

or Upgrade

Local Logical Standby Switchover (Oracle Beehive Release 11.2)

Seconds (Transparent)


1. Shutdown

2. Upgrade

3. Startup

Hours

Database Transition to

Oracle RAC and/or Oracle

ASM, Exadata Storage, or

Upgrading 10g ASM

Local Standby Switchover Seconds (Transparent)


1. Shutdown

2. Upgrade

3. Startup

Hours


11

The following table summarizes and highlights the benefits of using a local standby database in terms of unplanned outages.

TABLE 2: USING A LOCAL STANDBY DATABASE TO REDUCE UNPLANNED DOWNTIME

UNPLANNED OUTAGE ORACLE SOLUTION DOWNTIME

Database Instance

Hang or Failure

Local Standby Failover Seconds (transparent)

Oracle RAC Failover Seconds (transparent)

Remote Site Failover Minutes

Diagnosis and Recovery Hours

Entire Database

Failure or Corruption

Local Standby Failover Seconds (transparent)

Remote Site Failover Minutes

Diagnosis and Recovery Hours

Note that it is possible to convert a physical standby database to a logical standby database or to a snapshot standby database, or you can create additional logical standby databases or snapshot standby databases. Oracle Beehive cannot support the use of a logical standby database until Oracle Database 11g Release 2 (11.2), which includes logical standby support for SecureFiles.

Planned and Unplanned Outage Solutions

The following sections summarize the outages that may occur in a Oracle Beehive environment and the recommended Oracle solution that keeps application downtime to a minimum. All cases focus on Oracle Beehive Application downtime as perceived by the end user, and are not based on the actual downtime of the individual component.


12

Planned Outages

Table 3 summarizes the planned maintenance activities that may typically occur in a Oracle Beehive environment and the recommended technology solutions to keep downtime to a minimum.

TABLE 3. ORACLE BEEHIVE PLANNED OUTAGE SOLUTIONS

MAINTENANCE ACTIVITY SOLUTION ORACLE BEEHIVE OUTAGE

Mid-Tier Operating System or Hardware

Upgrade

Oracle Beehive Load balancing,

distributed services and clustering

No downtime

Beehive Application Upgrades & Patching Oracle Beehive zero downtime upgrade No downtime

Beehive Application Configuration

Change

Oracle Beehive Application Restart 1 to 2 minutes

Database Tier Operating System or

Hardware Upgrade

Oracle RAC No downtime

Oracle Database interim patching Oracle RAC rolling apply No downtime

Oracle Database 11g online patching Online Patching No downtime

Oracle Clusterware upgrade and patches Rolling apply/upgrade No downtime

Database storage migration Oracle ASM No downtime

Oracle ASM upgrade 10g: Oracle Data Guard Seconds to minutes

11g: Rolling upgrade No downtime

Migrating to ASM or migrating a single-

instance database to Oracle RAC

Oracle Data Guard Seconds to minutes

Patch set and database upgrades Oracle Data Guard logical standby (11.2) Seconds to minutes for 11.2 and

11.1 patch sets

Minutes to hour for 11.1 to 11.2


13

Unplanned Outages

Table 4 describes the unplanned outages that may be caused by system or human failures in a Oracle Beehive environment and the recommended technology solutions to keep downtime to a minimum.

TABLE 4. ORACLE BEEHIVE UNPLANNED OUTAGE SOLUTIONS

OUTAGE TYPE ORACLE SOLUTION BENEFITS RECOVERY TIME

Beehive Node or

Component

Load Balancing Surviving nodes pick up the slack No downtime

Database Node or

Instance Failure

Oracle RAC Automatic recovery of failed

nodes and instances, transparent

application and service failover

Beehive instances

transparently fail over with

FAN/FCF

Users see no outage

Site Failure Oracle Data Guard Failover Seconds to 5 minutes

Storage Failure Oracle ASM Mirroring and automatic

rebalancing

No downtime

RMAN with fast recovery

area

Fully managed database recovery

and disk based

Minutes to hours

Oracle Data Guard Failover Seconds to 5 minutes

Human Error Oracle Flashback Database and fine-grained rewind

capability

Minutes

LogMiner Log analysis Minutes to hours

Data Corruption RMAN with fast recovery

area

Online block media recovery and

managed disk-based backups

Minutes to hours

Oracle Data Guard Automatic validation of redo

blocks before they are applied,

fast failover to an uncorrupted

standby database

Seconds to 5 minutes


14

Deploying Oracle Beehive MAA

This section gives an overview of the steps involved in deploying Oracle Beehive in an MAA configuration. A detailed example of the steps can be found in the Appendix. Additionally, you should read the Oracle Beehive Installation Guide [5] that is specific to your platform to prepare for the installation.

The high-level steps to deploy an Oracle Beehive MAA configuration are as follows:

1. Install an Oracle RAC database configured with the database MAA best practices.

2. Install a single Oracle Beehive application node.

3. Install a single node of Oracle Beehive Beekeeper (optional).

4. Configure the load balancer.

5. Clone application-tier nodes.

6. Configure the Oracle Beehive site target in Oracle Enterprise Manager.

7. Install the Oracle Data Guard disaster-recovery site.

8. Install the Oracle Data Guard local HA standby database (optional).

Oracle Database MAA

Before installing the Oracle Beehive application tier, you must have created an Oracle Database. For Oracle Beehive MAA, that database will be an Oracle Real Application Clusters (Oracle RAC) database. Be sure to review chapter 2 of the Oracle Beehive Installation Guide [5] about Oracle Beehive Database requirements. The requirements include:

• Oracle Database 11g release 1 (11.1.0.7 or later), plus the patches listed in the Installation Guide that is specific to your platform.

• The AL32UTF8 (Unicode) character set

• The following database components:

o Oracle XML DB

o Oracle interMedia (Oracle Multimedia in Oracle Database 11g)

o Oracle Text

o Partitioning


15

• Review “Minimum Values of Oracle Database Initialization Parameters” in the Oracle Beehive Installation Guide [5].

• Set the archive log mode of the database to ARCHIVELOG, which enables the archiving of the redo log files

• Use the Oracle Database Configuration Assistant (DBCA) to create the database

• See the Oracle High Availability Best Practices [3] guide, and read the “Configuring for High Availability” chapter.

The database installation steps are not provided in the example installation appendix of this white paper. However, there is a sample of the SPFILE and network administration files (tnsnames.ora and listener.ora) in the Appendix. Note that listener connection rate throttling is configured in the listener.ora file and it is a recommended best practice. For more details about configuring Listener Connection Rate Throttling see Appendix E in the Optimizing Availability During Unplanned Outages Using Oracle Clusterware and RAC white paper.

Oracle Beehive Application High Availability Deployment

Install a single Oracle Beehive application node

After the database is set up, then you can install the first Oracle Beehive application node. Before using the Oracle Beehive Install Wizard, verify that all of the requirements listed in the Oracle Beehive Installation Guide [5] are met. The complete set of steps and sequence of Install Wizard screens are detailed in the Oracle Beehive Installation Guide [5] t.

After the installation completes successfully, then perform the post-installation verification steps. (See the Appendix for example output):

Post-install Verification

1. Verify that all processes are running:

beectl status

2. Verify the default virtual server port setup:

beectl list_properties --component _VIRTUAL_SERVER

3. Verify the HTTP Server configuration:

beectl list_properties –-component <Component Identifier prefixed with “ohs” from the “beectl status” output>

4. Decide on the port configuration and make any necessary changes (see the “Port Configuration” section of this white paper). This example chooses not to use any privileged (< 1024) ports.

http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdf�

http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastRecoveryOracleClusterwareandRAC.pdf�


16

5. Configure TLS and the XMPP Service.

Test the Install

1. Create test user accounts.

2. Check IMAP.

3. Test each Oracle Beehive Client.

Configure the Load Balancer

Complete details for configuring Oracle Beehive with a load balancer are in the Oracle Beehive MAA paper at “Configuring Maximum Availability Architecture for Beehive with F5 BIG-IP Local Traffic Manager: Beehive High Availability.” An example of the Oracle Beehive server side configuration is included in the appendix of this white paper in the “Load Balancer Configuration” section.

Configure Oracle Beehive to Utilize FAN/FCF

This installation step configures Oracle Beehive to take advantage of the Oracle Fast Application Notification (FAN) and Fast Connection Failover (FCF) features to protect the Oracle Beehive application tier and end users from any Oracle RAC database failures. This will also be used later for a local standby failover scenario. The following links high-level steps provide links to details in the appendix:

1. Configure Oracle Notification Service (ONS)

2. Configure Database Services

Clone Application Tier Nodes

It’s recommended to wait until you have a single-node configuration working before cloning other instances. The complete cloning process is detailed in Chapter 30 of the Oracle Beehive Installation Guide [5].

Cloning is the process of copying an existing installation to a different location while preserving its configuration.

The cloning process works by copying all files from the source Oracle home to the destination Oracle home. Hence, the cloning process does not copy any files used by the source instance that are located outside the source Oracle home's directory structure. After the files are copied, a set of beectl commands are used to update the information in key configuration files.

Cloning also enables you to safely modify an existing Oracle Beehive instance in production, such as installing a new patch or making changes to the database. Clone your existing Oracle Beehive instance and apply your changes to the clone. Once you have verified and certified that

http://www.oracle.com/technology/deploy/availability/pdf/maa_wp_beehive_f5_bestpractices.pdf�



17

your changes work as expected, you may safely apply those changes to your Oracle Beehive instance in production.

A cloned installation behaves the same as the source installation. For example, you can uninstall or patch the cloned instance with the Oracle Beehive Install Wizard. You can also use a cloned installation as the source for another cloning operation.

See the Appendix “Clone the Application Tier,” section for an example with the test environment.

Install a Single Node of Oracle Beehive Beekeeper (optional)

Oracle Beekeeper is a secure, browser-based administration client. It provides Oracle Beehive administrators centralized and role-based access to system configuration and management, user and workspace administration, monitoring, and reporting functions. See the “Oracle Beekeeper Install” section for installation details.

Oracle Beehive Site Target in Oracle Enterprise Manager

Oracle Enterprise Manager Grid Control helps you monitor Oracle Beehive components in your enterprise. You can discover Oracle Beehive components in your environment and add them to Grid Control for central monitoring and management. Enterprise Manager Agents will discover and monitor Oracle Beehive sites, services, and service instances. In Grid Control Oracle Beehive sites, services (called applications), and service instances (called application instances) have home pages. For a complete description of target types defined for Oracle Beehive in Grid Control see the “Using Grid Control for Monitoring Oracle Beehive Targets” section under Chapter 12, “Oracle Beehive Management” in the Oracle Enterprise Manager Concepts 10g Release 5 guide [7]. Additionally, in that same section the steps for discovering the Oracle Beehive site target and configuring the Oracle Beehive System and Services are defined.

Oracle Beehive Disaster Recovery Site Deployment

Again, using Oracle Data Guard with your Oracle Beehive deployment ensures high availability, data protection, and disaster recovery for enterprise data. Data Guard provides a comprehensive set of services that create, maintain, manage, and monitor one or more standby databases to enable production Oracle databases to survive disasters and data corruptions and to minimize downtime for planned outages. Data Guard maintains these standby databases as copies of the production database. Then, if the production database becomes unavailable because of a planned or an unplanned outage, Data Guard can switch any standby database to the production role, minimizing the downtime associated with the outage.

The complete steps for configuring the Oracle Beehive disaster-recovery site are included in Chapter 16 of the Oracle Beehive Administrator’s Guide [8]. For an example setup, see the “Oracle


18

Beehive Disaster Recovery Site Setup” section in the appendix. The high-level steps for setting up an Oracle Beehive disaster-recovery (DR) site are:

1. Clone the Oracle Beehive DR site application tier.

2. Create a DR site physical standby database.

3. Configure Oracle Beehive connections to include the DR site.

4. Add the DR site application nodes to the WAN traffic manager.

5. Test role transitions, failover and switchover.

Local HA Standby Deployment

As described earlier in the “Oracle Beehive Disaster Recovery Plus a Local HA Standby” section, deploying a local high availability standby provides various benefits. The details for deploying a local HA standby are in Chapter 16 of the Oracle Beehive Administrator’s Guide [8]. An example of that deployment is in the appendix under “Oracle Beehive Local HA Standby Setup.”

Deploying a local standby uses the same set of application-tier nodes as the primary database. It also requires setting up the Oracle FAN ONS publisher program. By calling the FAN Publisher program based on a trigger enabled on the DB_ROLE_CHANGE system event, the Oracle Beehive application-tier instances can be notified of any primary site failure and instructed to reconnect to the new primary database. This is described generically for JDBC clients in the “Client Failover Best Practices for Highly Available Oracle Databases” MAA white paper.

Additionally, a demonstration of a local standby failover can be found on the MAA Web site [1]: Oracle Beehive 1.5 - Automatic Failover of Oracle Beehive to a Data Guard 11g Standby Database.

Conclusion

Choosing and implementing the architecture that best fits the availability requirements of a business can be a daunting task. This architecture must encompass appropriate redundancy, provide adequate protection from all types of outages, ensure consistent high performance and robust security, while being easy to deploy, manage, and scale. Needless to mention, this architecture should be driven by well-understood business requirements. Configuring Oracle Beehive Maximum Availability Architecture will eliminate or significantly reduce unplanned and planned downtime, thereby enabling you to meet or exceed your business service levels.

http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_ClientFailoverBestPractices.pdf�

http://download.oracle.com/technology/deploy/availability/viewlets/beehive-local-standby/oracle_maa_beehive_local_standby_failover4_viewlet_swf.html�

http://download.oracle.com/technology/deploy/availability/viewlets/beehive-local-standby/oracle_maa_beehive_local_standby_failover4_viewlet_swf.html�


19

Appendix

Example Test Environment

Database

Operating System: Oracle Enterprise Linux 5, Linux x86-64, kernel 2.6.18-53.1.21.2.1 OS User: oracle ASM Disk Groups: +DATA, +FRA Oracle Managed Files being used (db_create_file_dest=’+DATA’ and db_recovery_file_dest=’+FRA’) CRS Oracle home: /u01/app/11.1.0/crs ASM Oracle home: /u01/app/oracle/product/11.1.0/asm_1 Database Oracle home: /u01/app/oracle/product/11.1.0/db_1

Primary Site Cluster with 2 Hosts: dscbac05 (10.204.32.205), dscbac06 (10.204.32.206) VIPs: dscbac05-vip, dscbac06-vip DB_UNIQUE_NAME=bhdb

Standby Site Cluster with 2 Hosts: dscbac07 (10.204.32.207), dscbac08 (10.204.32.208) VIPs: dscbac07-vip, dscbac08-vip DB_UNIQUE_NAME=bhdc.

Application Tier

Operating System: Oracle Enterprise Linux 5, Linux x86-64, kernel 2.6.18-53.1.21.2.1 OS User: bhmt Oracle Beehive Oracle home: /u01/app/oracle/product/1.5.1.2.0/beehive_1 Beekeeper Oracle home: $ORACLE_BASE/product/1.5.1.3.0/beekeeper_1

Primary Site 2 nodes:

1. dscbac09, beehive_instance_maaXtst.dscbac09.us.oracle.com 2. dscbac10, beehive_instance_maa2.dscbac10.us.oracle.com

Standby Site 2 nodes (node name, Beehive instance name)

1. dscbac11, beehive_instance_maa3.dscbac11.us.oracle.com 2. dscbac12, beehive_instance_maa4.dscbac12.us.oracle.com


20

Load Balancer

This test environment used a F5 Big-IP model 1500. The best practices in the Configuring Maximum Availability Architecture for Beehive with F5 BIG-IP Local Traffic Manager: Beehive High Availability MAA paper were followed.

Enterprise Manager

Oracle Enterprise Manager 10g Release 5 Grid Control (10.2.0.5)

Install the first Oracle Beehive Instance

When entering the database information for Oracle RAC, separate the host:port by a ‘^’ character. For example:

dscbac05-vip.us.oracle.com:1523^dscbac06-vip.us.oracle.com:1523

Service name: bhdb.us.oracle.com

Post-install Verification

1. Check that all processes are running: beectl status -----------------------------------------+----------------+--------------------- Component identifier | Component type | Status -----------------------------------------+----------------+--------------------- BTI_maaXtst.dscbac09.us.oracle.com | Bti | RUNNING -----------------------------------------+----------------+--------------------- oc4j_soa_maaXtst.dscbac09.us.oracle.com | ManagedOc4j | RUNNING -----------------------------------------+----------------+--------------------- BEECORE_maaXtst.dscbac09.us.oracle.com | ManagedOc4j | RUNNING -----------------------------------------+----------------+--------------------- BEEAPP_maaXtst.dscbac09.us.oracle.com | ManagedOc4j | RUNNING -----------------------------------------+----------------+--------------------- BEEMGMT_maaXtst.dscbac09.us.oracle.com | ManagedOc4j | RUNNING -----------------------------------------+----------------+--------------------- ohs_maaXtst.dscbac09.us.oracle.com | HttpServer | RUNNING -----------------------------------------+----------------+--------------------- 6 Record(s) displayed.

2. Check virtual server port setup beectl list_properties --component _VIRTUAL_SERVER --------------------+----------------------------------------------------------- Property name | Property value --------------------+----------------------------------------------------------- Alias | _VIRTUAL_SERVER --------------------+----------------------------------------------------------- BtiClientPort | 21401 --------------------+----------------------------------------------------------- BtiSecureClientPort | 21451 --------------------+----------------------------------------------------------- FtpPort | 2121 --------------------+-----------------------------------------------------------





21

HttpPort | 7777 --------------------+----------------------------------------------------------- HttpSslEnabled | true --------------------+----------------------------------------------------------- HttpSslPort | 443 --------------------+----------------------------------------------------------- IPAddress | --------------------+----------------------------------------------------------- ImapPort | 143 --------------------+----------------------------------------------------------- ImapSslEnabled | false --------------------+----------------------------------------------------------- ImapSslPort | 993 --------------------+----------------------------------------------------------- ServerName | dscbac09.us.oracle.com --------------------+----------------------------------------------------------- SmtpAuthRequired | false --------------------+----------------------------------------------------------- SmtpPort | 25 --------------------+----------------------------------------------------------- SmtpSslEnabled | false --------------------+----------------------------------------------------------- SmtpSslPort | 465 --------------------+----------------------------------------------------------- XmppPort | 5222 --------------------+----------------------------------------------------------- XmppSslEnabled | false --------------------+----------------------------------------------------------- XmppSslPort | 5223 --------------------+----------------------------------------------------------- 19 Record(s) displayed.

3. List HTTP (ohs) setup. beectl list_properties –component ohs_maaXtst.dscbac09.us.oracle.com

-------------------+------------------------------------------------------------ Property name | Property value -------------------+------------------------------------------------------------ Alias | ohs_maaXtst.dscbac09.us.oracle.com -------------------+------------------------------------------------------------ BeehiveInstance | beehive_instance_maaXtst.dscbac09.us.oracle.com -------------------+------------------------------------------------------------ DocRootLimitExcept | GET POST -------------------+------------------------------------------------------------ HttpListenPort | 7777 -------------------+------------------------------------------------------------ HttpServerCluster | 3f00bdb9-4aa6-4b43-a1a7-5a9a823d4f34 -------------------+------------------------------------------------------------ HttpSslListenPort | 443 -------------------+------------------------------------------------------------ KeepAliveTimeout | 15 -------------------+------------------------------------------------------------ LogLevel | notice -------------------+------------------------------------------------------------ MaxClients | 150 -------------------+------------------------------------------------------------ OsGroup | oinstall -------------------+------------------------------------------------------------ OsUser | bhmt -------------------+------------------------------------------------------------ SSLLogLevel | warn


22

-------------------+------------------------------------------------------------ ServerAdmin | " " -------------------+------------------------------------------------------------ ServerRoot | "/u01/app/oracle/product/1.5.1.5.0/beehive_1/Apache/Apache" -------------------+------------------------------------------------------------ Status | ENABLED -------------------+------------------------------------------------------------ Timeout | 60 -------------------+------------------------------------------------------------ 16 Record(s) displayed.

Port Configuration

Set the ports to unprivileged (> 1024) port numbers and in preparation for load balancer configuration. Note that these ports also match the ports in the “Configuring Maximum Availability Architecture for Beehive with F5 BIG-IP Local Traffic Manager: Beehive High Availability” paper. beectl modify_property --component _VIRTUAL_SERVER --name SmtpPort --value 2225 beectl modify_property --component _VIRTUAL_SERVER --name ImapPort --value 5143 beectl modify_property --component _VIRTUAL_SERVER --name BtiSecureClientPort --value 5224 beectl modify_property --component _EmailService:SMTPProperties --name Port --value 2225 beectl modify_property --component _EmailService:IMAPProperties --name Port --value 5143 beectl modify_property --component SitewideBtiConfiguration --name SecureClientPort --value 5224

This is the port of the load balancer for HTTP. Note that the HttpListenPort is the listening port on the mid-tiers. beectl modify_property --component _VIRTUAL_SERVER --name HttpPort --value 80 beectl modify_property --component ohs_maaXtst.dscbac09.us.oracle.com--name HttpListenPort --value 7777 beectl modify_property --component _VIRTUAL_SERVER --name HttpSslEnabled --value true beectl list_properties --component _VIRTUAL_SERVER --------------------+----------------------------------------------------------- Property name | Property value --------------------+----------------------------------------------------------- *ImapPort | 5143 --------------------+----------------------------------------------------------- *SmtpPort | 2225 --------------------+----------------------------------------------------------- Alias | _VIRTUAL_SERVER --------------------+----------------------------------------------------------- BtiClientPort | 21401 --------------------+----------------------------------------------------------- *BtiSecureClientPort | 5224 --------------------+----------------------------------------------------------- FtpPort | 2121 --------------------+----------------------------------------------------------- *HttpPort | 80 --------------------+----------------------------------------------------------- *HttpSslEnabled | true




23

--------------------+----------------------------------------------------------- HttpSslPort | 443 --------------------+----------------------------------------------------------- IPAddress | --------------------+----------------------------------------------------------- ImapSslEnabled | false --------------------+----------------------------------------------------------- ImapSslPort | 993 --------------------+----------------------------------------------------------- ServerName | dscbac09.us.oracle.com --------------------+----------------------------------------------------------- SmtpAuthRequired | false --------------------+----------------------------------------------------------- SmtpSslEnabled | false --------------------+----------------------------------------------------------- SmtpSslPort | 465 --------------------+----------------------------------------------------------- XmppPort | 5222 --------------------+----------------------------------------------------------- XmppSslEnabled | false --------------------+----------------------------------------------------------- XmppSslPort | 5223 --------------------+----------------------------------------------------------- NOTE:- * indicates that property value is changed and change is not yet activated. beectl activate_configuration

beectl modify_local_configuration_files

Configure TLS and the XMPP Service

Configure TLS

1. Enable auto login for default wallet on each mid-tier. orapki wallet create -wallet $ORACLE_HOME/Apache/Apache/conf/ssl.wlt/default/ -auto_login -pwd welcome

See the Oracle Beehive Install guide5 for replacing test certificates and configuring the TLS password the Oracle Wallet, “Configuring TLS with Oracle Wallet”.

2. Get instance names. beectl list_components --type BeehiveInstance -----------------+------------------------------------------------------------ Component type | Component identifier -----------------+------------------------------------------------------------ BeehiveInstance | beehive_instance_maaXtst.dscbac09.us.oracle.com -----------------+------------------------------------------------------------

3. Set instance property. beectl modify_property \ --component beehive_instance_maaXtst.dscbac09.us.oracle.com \ --name WalletDir \ --value /u01/app/oracle/product/1.5.1.5.0/beehive_1/Apache/Apache/conf/ssl.wlt/default

4. Verify settings. beectl list_properties --component beectl list_properties --component beehive_instance_maaXtst.dscbac07.us.oracle.com --name WalletDir

http://www.oracle.com/pls/bee2/to_URL?remark=ranked&urlname=http:%2F%2Fdownload.oracle.com%2Fdocs%2Fcd%2FE16671_01%2Fbh.200%2Fe16641%2Fwallet.htm%23sthref381�


24

Configure the XMPP Service

1. View settings beectl list_properties --component _XmppService --------------------------+----------------------------------------------------- Property name | Property value --------------------------+----------------------------------------------------- Alias | _XmppService --------------------------+----------------------------------------------------- Database | --------------------------+----------------------------------------------------- DomainName | example.com --------------------------+----------------------------------------------------- DomainNames | example.com --------------------------+----------------------------------------------------- Language | en --------------------------+----------------------------------------------------- LightweightThreadCount | 10 --------------------------+----------------------------------------------------- LightweightThreadPriority | 5 --------------------------+----------------------------------------------------- MessagesOnInvalidAction | --------------------------+----------------------------------------------------- RetryCount | 6 --------------------------+----------------------------------------------------- RetryTimeout | 5 --------------------------+----------------------------------------------------- SearchFields | first , last , email --------------------------+----------------------------------------------------- ServiceApplication | svcapp_xmpp --------------------------+----------------------------------------------------- ServiceInstances | instance_xmpp_BEEAPP_maaXtst.dscbac09.us.oracle.com --------------------------+----------------------------------------------------- Site | _CURRENT_SITE --------------------------+----------------------------------------------------- Status | ENABLED --------------------------+----------------------------------------------------- SupportedAgents | uds --------------------------+----------------------------------------------------- VersionRules | --------------------------+----------------------------------------------------- XmppPort | 5222 --------------------------+----------------------------------------------------- XmppSslPort | 5223 --------------------------+----------------------------------------------------- XmppTimerKeepAliveTime | 5 --------------------------+-----------------------------------------------------

2. Change domain, if desired. beectl modify_property --component _XmppService \ --name DomainName --value oracle.com beectl modify_property --component _XmppService \ --name DomainNames --value oracle.com -–activate_configuration

3. View the changes. --------------------------+----------------------------------------------------- Property name | Property value --------------------------+----------------------------------------------------- Alias | _XmppService


25

--------------------------+----------------------------------------------------- Database | --------------------------+----------------------------------------------------- *DomainName | oracle.com --------------------------+----------------------------------------------------- *DomainNames | oracle.com --------------------------+----------------------------------------------------- Language | en --------------------------+----------------------------------------------------- LightweightThreadCount | 10 --------------------------+----------------------------------------------------- LightweightThreadPriority | 5 --------------------------+----------------------------------------------------- MessagesOnInvalidAction | --------------------------+----------------------------------------------------- RetryCount | 6 --------------------------+----------------------------------------------------- RetryTimeout | 5 --------------------------+----------------------------------------------------- SearchFields | first , last , email --------------------------+----------------------------------------------------- ServiceApplication | svcapp_xmpp --------------------------+----------------------------------------------------- ServiceInstances | instance_xmpp_BEEAPP_maa2.dscbac08.us.oracle.com , i | nstance_xmpp_BEEAPP_maaXtst.dscbac07.us.oracle.com , | instance_xmpp_BEEAPP_maa4.dscbac06.us.oracle.com , | instance_xmpp_BEEAPP_maa3.dscbac05.us.oracle.com --------------------------+----------------------------------------------------- Site | _CURRENT_SITE --------------------------+----------------------------------------------------- Status | ENABLED --------------------------+----------------------------------------------------- SupportedAgents | uds --------------------------+----------------------------------------------------- VersionRules | --------------------------+----------------------------------------------------- XmppPort | 5222 --------------------------+----------------------------------------------------- XmppSslPort | 5223 --------------------------+----------------------------------------------------- XmppTimerKeepAliveTime | 5 --------------------------+----------------------------------------------------- 20 Record(s) displayed. 1) Activate the configuration beectl activate_configuration beectl modify_local_configuration_files

Sanity Test Install

1. Create test user accounts.

Using the bulk_create_users.sh script in the Appendix create some Oracle Beehive test users. For example, to create 5 users, bhtest1-bhtest5 run this command:

./bulk_create_users.sh -bh /u01/app/oracle/product/1.5.1.2.0/beehive_1 \ -prefix bhtest -passwd Welcome1 -domain oracle.com \ -startId 1 -endId 5 -includeDomainInName


26

2. Check IMAP.

See My Oracle Support Note 317307.1 for details. For example: a. > telnet dscbac09.us.oracle.com 5143

Trying 144.25.145.30... Connected to dscbac09.us.oracle.com. Escape character is '^]'. * OK server ready. Unauthorized Access Prohibited.

b. A1 login [email protected] Welcome1 A1 OK LOGIN completed

c. A2 list "" "*" * LIST () "/" "Documents" * LIST () "/" "INBOX" * LIST () "/" "Public Documents" A2 OK LIST completed

d. A3 select "INBOX" * 0 EXISTS * 0 RECENT * OK [UIDVALIDITY 1] UID validity status * OK [UIDNEXT 1] Predicted next UID * FLAGS (\Seen \Deleted \Answered \Forwarded \Redirected \Flagged \Hidden \Draft $MDNSent) * OK [PERMANENTFLAGS (\Seen \Deleted \Answered \Forwarded \Redirected \Flagged \ Hidden \Draft $MDNSent)] Permanent flags A3 OK [READ-WRITE] SELECT completed

3. Test some Oracle Beehive clients.

http://dscbac09.us.oracle.com:7777/bcentral http://dscbac09.us.oracle.com:7777/caldav http://dscbac09.us.oracle.com:7777/content/dav

In addition to these URLs the Oracle Beehive Extensions for Outlook, Oracle Beehive Extensions for Explorer, Oracle Beehive Conferencing, and Oracle Beehive Instant Messaging clients could be setup and tested.

Load Balancer Configuration

At this stage there is only a single Oracle Beehive application node but the load balancer configuration can still be done at this point so that all clones will have the same settings. The Oracle Beehive MAA paper at “Configuring Maximum Availability Architecture for Beehive with F5 BIG-IP Local Traffic Manager: Beehive High Availability” has the complete details for configuring the F5 Big-IP Local Traffic Manager (LTM) and the corresponding Oracle Beehive server configuration. This configuration will include terminating SSL at the Big-IP LTM and redirecting non-SSL traffic through the SSL ports. Below are the Oracle Beehive server commands that go with this test setup.

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=317307.1�

http://dscbac09.us.oracle.com:7777/bcentral�

http://dscbac09.us.oracle.com:7777/caldav�

http://dscbac09.us.oracle.com:7777/content/dav�




27

Set the Virtual Server Name and Other Properties

beectl modify_property --component _VIRTUAL_SERVER --name ServerName --value beehive.oracle.com beectl modify_property --component _VIRTUAL_SERVER --name HttpPort --value 443 beectl modify_property --component _VIRTUAL_SERVER --name HttpSslEnabled --value true beectl list_properties --component _VIRTUAL_SERVER --------------------+----------------------------------------------------------- Property name | Property value --------------------+----------------------------------------------------------- *ServerName | beehive.oracle.com --------------------+----------------------------------------------------------- Alias | _VIRTUAL_SERVER --------------------+----------------------------------------------------------- BtiClientPort | 21401 --------------------+----------------------------------------------------------- BtiSecureClientPort | 5224 --------------------+----------------------------------------------------------- FtpPort | 2121 --------------------+----------------------------------------------------------- *HttpPort | 443 --------------------+----------------------------------------------------------- *HttpSslEnabled | true --------------------+----------------------------------------------------------- HttpSslPort | 443 --------------------+----------------------------------------------------------- IPAddress | --------------------+----------------------------------------------------------- ImapPort | 5143 --------------------+----------------------------------------------------------- ImapSslEnabled | false --------------------+----------------------------------------------------------- ImapSslPort | 993 --------------------+----------------------------------------------------------- SmtpAuthRequired | false --------------------+----------------------------------------------------------- SmtpPort | 2225 --------------------+----------------------------------------------------------- SmtpSslEnabled | false --------------------+----------------------------------------------------------- SmtpSslPort | 465 --------------------+----------------------------------------------------------- XmppPort | 5222 --------------------+----------------------------------------------------------- XmppSslEnabled | false --------------------+----------------------------------------------------------- XmppSslPort | 5223 --------------------+----------------------------------------------------------- NOTE:- * indicates that property value is changed and change is not yet activated.

Set Oracle Beehive HTTP Server for SSL Termination

1. Set the SslTerminatedByLoadBalancer property of the HttpServerCluster component to true: beectl modify_property --component _current_site:HttpServerCluster \ --name SslTerminatedByLoadBalancer --value true


28

2. Review the changes:

beectl list_properties --component _CURRENT_SITE:HttpServerCluster -----------------------------+--------------------------------- Property name | Property value -----------------------------+--------------------------------- Alias | -----------------------------+--------------------------------- * HttpServerSslEnabled | true -----------------------------+---------------------------------

Activate the Changes beectl activate_configuration

beectl modify_local_configuration_files

Sanity Test with Load Balancer

1. Check IMAP.

See My Oracle Support Note 317307.1 for details. For example: a. > telnet beehive.oracle.com 143

Trying 144.25.145.30... Connected to beehive.oracle.com. Escape character is '^]'. * OK server ready. Unauthorized Access Prohibited.

b. A1 login [email protected] Welcome1 A1 OK LOGIN completed

c. A2 list "" "*" * LIST () "/" "Documents" * LIST () "/" "INBOX" * LIST () "/" "Public Documents" A2 OK LIST completed

d. A3 select "INBOX" * 0 EXISTS * 0 RECENT * OK [UIDVALIDITY 1] UID validity status * OK [UIDNEXT 1] Predicted next UID * FLAGS (\Seen \Deleted \Answered \Forwarded \Redirected \Flagged \Hidden \Draft $MDNSent) * OK [PERMANENTFLAGS (\Seen \Deleted \Answered \Forwarded \Redirected \Flagged \ Hidden \Draft $MDNSent)] Permanent flags A3 OK [READ-WRITE] SELECT completed

2. Test some Oracle Beehive clients.

http://beehive.oracle.com/bcentral http://beehive.oracle.com/caldav http://beehive.oracle.com/content/dav In addition to these URLs the Oracle Beehive Extensions for Outlook, Oracle Beehive Extensions for Explorer, Oracle Beehive Conferencing, and Oracle Beehive Instant Messaging clients could be setup and tested. The complete list of port mappings can be

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=317307.1�

http://beehive.oracle.com/bcentral�

http://beehive.oracle.com/caldav�

http://beehive.oracle.com/content/dav�


29

found in Table 2 and Table 5 in the “Configuring Maximum Availability Architecture for Beehive with F5 BIG-IP Local Traffic Manager: Beehive High Availability” paper.

Configure ONS

This step sets up Oracle Beehive to run with an Oracle RAC database and take advantage of the FAN/FCF features that are integrated into Oracle Beehive.

1. Identify the ONS port.

a. Change to the Oracle CRS home bin directory > cd /u01/app/11.1.0/crs/bin

b. List the current settings on the database Oracle RAC nodes > ./onsctl ping Number of onsconfiguration retrieved, numcfg = 2 onscfg[0] {node = dscbac09, port = 6251} Adding remote host dscbac09:6251 onscfg[1] {node = dscbac10, port = 6251} Adding remote host dscbac10:6251

ons is running ...

2. Modify the Oracle Beehive server database configuration ONS entries. beectl modify_database --database _current_site:Database \ --ons_entry dscbac09.us.oracle.com:6251 \ --ons_entry dscbac10.us.oracle.com:6251

3. Activate the changes. beectl activate_configuration beectl modify_local_configuration_files beectl restart --all

4. View the changes. beectl list_properties --component _current_site:Database \ --name OnsNodeConfiguration

---------------------+-------------------------------------------------------- Property name | Property value ---------------------+-------------------------------------------------------- OnsNodeConfiguration | dscbac09.us.oracle.com:6251 , dscbac10.us.oracle.com:6251 ---------------------+--------------------------------------------------------

1 Record(s) displayed.




30

Configure Oracle Beehive Database Services

Configure Oracle Beehive Bootstrap Service

1. Create a CRS managed Oracle Beehive bootstrap database service on all Oracle RAC nodes and start. srvctl add service -d bhdb -s beehive_bs -r bhdb1,bhdb2 srvctl start service -d bhdb -s beehive_bs -i bhdb1 srvctl start service -d bhdb -s beehive_bs -i bhdb2

2. Update the Oracle Beehive server to use the CRS managed database bootstrap service beectl modify_bootstrap_configuration --connect_string \ '(DESCRIPTION =(ADDRESS_LIST=( (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac05-vip)(PORT = 1523)) (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac06-vip)(PORT = 1523))) (LOAD_BALANCE = yes) (ENABLE=BROKEN) (FAILOVER=on) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = beehive_bs.us.oracle.com)))'

3. View the change beectl list_bootstrap_configuration -------------------------------+---------------------------------------------- Property Name | Property Value -------------------------------+---------------------------------------------- *ConnectString | (DESCRIPTION =(address_list= | (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac05 | -vip)(PORT = 1523)) | (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac06 | -vip)(PORT = 1523))) | (LOAD_BALANCE = yes) | (ENABLE=BROKEN) | (FAILOVER=on) | (CONNECT_DATA = | (SERVER = DEDICATED) | (SERVICE_NAME = beehive_bs.us.oracle.com)) | ) -------------------------------+------------------------------------------------ SchemaName | BEE_CODE_01142010 -------------------------------+------------------------------------------------ SchemaPassword | [Protected Value not displayed] -------------------------------+------------------------------------------------ Oc4jNameWhereMgmtsvcIsDeployed | BEEMGMT -------------------------------+------------------------------------------------ MgmtsvcEJBUser | oc4jadmin -------------------------------+------------------------------------------------ MgmtsvcEJBPassword | [Protected Value not displayed] -------------------------------+------------------------------------------------


31

BeehiveInstanceID | 150a51af-29e8-4e98-b5ab-c1bc9b35f5eb -------------------------------+------------------------------------------------ LangPackLocalCache | /u01/app/oracle/product/1.5.1.2.0/beehive_1/bee | hive/seed/translation -------------------------------+------------------------------------------------ MasterKey | [Protected Value not displayed] -------------------------------+------------------------------------------------ BeehiveStopMode | FORCEFULL -------------------------------+------------------------------------------------ 10 Record(s) displayed.

Configure Oracle Beehive Affinity Services

1. Create one affinity DB service per Oracle RAC node and start. srvctl add service -d bhdb -s bhaffsvc1 -r bhdb1 srvctl add service -d bhdb -s bhaffsvc2 -r bhdb2 srvctl start service -d bhdb -s bhaffsvc1 -i bhdb1 srvctl start service -d bhdb -s bhaffsvc2 -i bhdb2

2. Set the Oracle Beehive server AffinityServiceNames property. beectl modify_property --component _CURRENT_SITE:Database \

--name AffinityServiceNames \

--value ‘((DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dscbac05-

vip.us.oracle.com)(PORT=1523))(LOAD_BALANCE=yes)(FAILOVER=on)(CONNECT_DATA=(SE

RVER=DEDICATED)(SERVICE_NAME=bhaffsvc1.us.oracle.com)))’ \

--value ‘((DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dscbac06-

vip.us.oracle.com)(PORT=1523))(LOAD_BALANCE=yes)(FAIOVER=on)(CONNECT_DATA=(SER

VER=DEDICATED)(SERVICE_NAME=bhaffsvc2.us.oracle.com)))’

3. View the change. beectl list_properties --component _CURRENT_SITE:Database \

> --name AffinityServiceNames

---------------------+----------------------------------------------------------

Property name | Property value

---------------------+----------------------------------------------------------

*AffinityServiceNames| (DESCRIPTION= (ADDRESS=(PROTOCOL=TCP)(HOST=dscbac05-vip.u

| s.oracle.com)(PORT=1523))LOAD_BALANCE=yes)(FAILOVER=on)(C

| ONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=bhaffsvc1.us.

| oracle.com))),(DESCRIPTION= (ADDRESS=(PROTOCOL=TCP)(HOST=

| dscbac06-vip.us.oracle.com)(PORT=1523))LOAD_BALANCE=yes)(

| FAILOVER=on)(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME

| =bhaffsvc2.us.oracle.com)))

---------------------+----------------------------------------------------------



32

Set the Oracle Beehive Server Outbound Connect Timeout

Connect Timeout: The Oracle Beehive JDBC clients must also insure that after they have been notified of a failure they are not subject to TCP timeouts if they subsequently attempt to reconnect to a failed host. This is accomplished using SQLNET outbound connect timeout such that clients will quickly attempt to connect to the next host in an address list if the first is not available. For the Oracle Beehive server, set the outbound connect timeout, as follows: beectl modify_property --component _current_site:Database:DefaultNonXaPool \ --name ConnectionProperties \ --value oracle.net.ns.SQLnetDef.TCP_CONNTIMEOUT_STR:3000 beectl modify_property --component _current_site:Database:AffinityPool \ --name ConnectionProperties \ --value oracle.net.ns.SQLnetDef.TCP_CONNTIMEOUT_STR:3000

Configure Oracle Beehive Search Service

If using an isolated Oracle Beehive Search service then create a CRS managed service and update the Oracle Beehive server search service connect string. The nodes may overlap with the bootstrap and affinity service connect strings, or the Search database instances can be on separate instances from the Oracle Beehive services. This example shows an overlap as the Search service is designated to node 2, dscbac10, as its primary node and node 1, dscbac09, is the backup node.

1. Create a CRS managed Oracle Beehive search database service on all Oracle RAC nodes and start. srvctl add service -d bhdb -s search -r bhdb2 –a bhdb1 srvctl modify service -d bhdb -s search -n -i bhdb2 -a bhdb1 srvctl start service -d bhdb -s search -i bhdb2 srvctl config service -d bhdb -s search search PREF: bhdb2 AVAIL: bhdb1

2. Update the Oracle Beehive server to use the CRS managed database search service. beectl modify_property --component _CURRENT_SITE:Database \ --name SearchServiceName \ --value '(DESCRIPTION=(ADDRESS_LIST=\ (ADDRESS=(PROTOCOL=TCP)(HOST=dscbac05-vip.us.oracle.com)(PORT=1523)) (ADDRESS=(PROTOCOL=TCP)(HOST=dscbac06-vip.us.oracle.com)(PORT=1523))) (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=search.us.oracle.com)))'

Activate the Changes beectl activate_configuration beectl modify_local_configuration_files

Install an Oracle RAC FAN Callout Script to Start Services

FAN callouts are server-side executables that Oracle RAC executes immediately when high availability events occur. For more details about FAN callouts, see the “Using Fast Application


33

Notification Callouts” section of the Oracle Real Application Clusters Administration and Deployment guide [6].

For Oracle Beehive we install an Oracle RAC callout script that will start all services when the database is opened. The script used is in the appendix in the “Oracle RAC FAN Callout Start Services Script” section. This script is a modification of the Start Services on Instance Up script at http://www.oracle.com/technology/sample_code/products/rac/index.html.

After ensuring the script has the correct environmental settings, mainly ORACLE_HOME and CRS_HOME then copy the script to each Oracle RAC node into the CRS_HOME/racg/usrco directory.

Test Oracle RAC Failover

For a demonstration of Oracle RAC failover see Oracle Beehive 1.5 - Oracle RAC Instance Failover. You can test that there is no Oracle Beehive service lost when a Oracle RAC database instance or node fails by having a group of users logged on and using Oracle Beehive as follows:

1. Startup only a single instance of the Oracle RAC database, such as bhdb1.

2. Start the Oracle Beehive application node.

3. Startup Oracle Beehive test clients.

4. Start the other Oracle RAC database instance, bhdb2.

5. Issue the SHUTDOWN ABORT command on Oracle RAC instance bhdb1.

6. Note that the application node continues to function and there are no interruptions to client functionality.

Clone the Application Tier

The complete cloning process is detailed in chapter 30 of the Oracle Beehive Installation Guide [5]. For example:

1. Shut down the mid-tier: beectl stop --all

2. Prepare the ORACLE_HOME to be cloned: beectl clone_preparation –file /u01/app/oracle/stage/bhmt/clone/clone_dscbac09_100114.files

Note : All validators registered for "clone_preparation" command will be executed now. Note : The validation results will be accumulated and analyzed at the end. Executing "Ias Instance Name" validator ...

http://www.oracle.com/technology/sample_code/products/rac/zip/startsrvonup.zip�

http://www.oracle.com/technology/sample_code/products/rac/index.html�

http://download.oracle.com/technology/deploy/availability/viewlets/beehive-rac-failover/beehiveracfailover_viewlet_swf.html�

http://download.oracle.com/technology/deploy/availability/viewlets/beehive-rac-failover/beehiveracfailover_viewlet_swf.html�


34

Successfully executed all validators registered for "clone_preparation" command. Now analyzing the validation results ... "Ias Instance Name" validator has passed with the following information : INFO : Ias instance name from /u01/app/oracle/product/1.5.1.2.0/beehive_1/config/ias.properties : maaXtst.dscbac09.us.oracle.com INFO : Ias instance name from /u01/app/oracle/product/1.5.1.2.0/beehive_1/opmn/conf/opmn.xml : maaXtst.dscbac09.us.oracle.com INFO : Ias instance name from SystemModel : maaXtst.dscbac09.us.oracle.com INFO : Ias instance name from /u01/app/oracle/product/1.5.1.2.0/beehive_1/clone/config/source.properties : maaXtst.dscbac09.us.oracle.com INFO : Ias instance name is same in all the configuration files. Preparing the oracle home for cloning ... This process will take some time, Please be patient. See the following log file for recent updates ... Log file : "/u01/app/oracle/product/1.5.1.2.0/beehive_1/beehive//logs/config/PrepareClone/<date>/PrepareClone_<time>.log"

[WARNING] [2010-01-14 13:30:55] [CloneImpl::executeIasPrepareCloneScripts Line 635 ] [Note : Ias clone scripts do not set the valid exit codes after completing the execution.] [WARNING] [2010-01-14 13:30:55] [CloneImpl::executeIasPrepareCloneScripts Line 636 ] [Note : So, Please check the ias clone script output message carefully.] [WARNING] [2010-01-14 13:30:55] [CloneImpl::getExcludedDirsAndExcludedFiles Line 936 ] [/u01/app/oracle/product/1.5.1.2.0/beehive_1/beehive/tmp/exported_configuration/ - Invalid entry in exclude file - /u01/app/oracle/product/1.5.1.2.0/beehive_1/beehive/conf/scripts/exclude_while_cloning.txt.] [WARNING] [2010-01-14 13:30:55] [CloneImpl::getExcludedDirsAndExcludedFiles Line 936] [/u01/app/oracle/product/1.5.1.2.0/beehive_1/beehive/zocsspool/ - Invalid entry in exclude file - /u01/app/oracle/product/1.5.1.2.0/beehive_1/beehive/conf/scripts/exclude_while_cloning.txt.] [WARNING] [2010-01-14 13:30:55] [CloneImpl::getExcludedDirsAndExcludedFiles Line 936] [/u01/app/oracle/product/1.5.1.2.0/beehive_1/beehive/conf/scripts/clone/conf/clonestate.xml - Invalid entry in exclude file - /u01/app/oracle/product/1.5.1.2.0/beehive_1/beehive/conf/scripts/exclude_while_cloning.txt.] Successfully prepared the oracle home for cloning.

3. Verify that hasbind is owned by the Oracle Beehive software owner: ls -l $ORACLE_HOME/beehive/bin/hasbind -rwx------ 1 oracle dba 32534 Jan 23 14:22 /u01/app/oracle/product/1.5.0.0.0/beehive_1/beehive/bin/hasbind

4. Create the Clone Image: cd $ORACLE_HOME export STAGE=/u01/app/oracle/stage/bhmt/clone export LOG=$STAGE/clone_`date +%Y%m%d_%H%M%S`.log


35

date > $LOG; tar –c –T $STAGE/clone_dscbac09_100114.files –zf $STAGE/clone_image.tar.gz >> $LOG;date >> $LOG

5. Copy over the image file: mkdir -p /u01/app/oracle/product/1.5.1.2.0/beehive_1 export ORACLE_HOME=/u01/app/oracle/product/1.5.1.2.0/beehive_1 cd $STAGE scp $STAGE/clone_image.tar.gz dscbac05:`pwd` clone_image.tar.gz 100% 4777MB 56.2MB/s 01:25

6. Verify Oracle Beehive prerequisites (see chapter 1 of the Oracle Beehive Installation Guide [5]) are met

This can be done manually or you can run the installer up through the “Prerequisite check” and then exit after the prerequisites are all met.

For now, you should minimally run the rpm command and verify the minimal set of required packages against the list in the installation guide. rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' \ gcc gcc-c++ setarch ksh sysstat gdbm libstdc++ libstdc++-devel compat-libstdc++-296 \ compat-db control-center glibc-common binutils make

7. Ensure OS groups Match Source System:

1. View groups on source system dscbac09: groups oinstall dba

2. View groups on target system, dscbac10 groups bhmt dba

3. Add to any missing groups As root: usermod –a –G oinstall bhmt

4. Re-login to OS user

8. On target system, dscbac10, extract Files from the image file: mkdir –p /u01/app/oracle/product/1.5.1.2.0/beehive_1 cd /u01/app/oracle/product/1.5.1.2.0/beehive_1 tar -xvzf $STAGE/clone_image.tar.gz

9. Set PERLSLIB environment:

export PERL5LIB=$ORACLE_HOME/perl/lib/5.8.3/i686-linux-thread-

multi:$ORACLE_HOME/perl/lib/5.8.3:$ORACLE_HOME/perl/lib/site_perl/5.8.3/i686-

linux-thread-multi/

10. If this is the first Oracle installation on this host then run the orainstRoot.sh script:

This step is necessary only if it does not already exist and if you want the mid-tiers to have the same file locations for everything, including the Oracle Inventory:

http://st-doc.us.oracle.com/beehive/1/bh.100/e14830/checklist.htm#CIHDCIGF�


36

a. Login as root.

b. Change to the Oracle home directory: cd $ORACLE_HOME

c. Run the script ./orainstRoot.sh

11. Obfuscate the site_key and database schema passwords

a. Obfuscate for the db_schema_password: beectl obfuscate --expiration_time_in_minutes 0

Specify the value for secure option --password (The text you type will not be displayed as it is entered): Enter value for password: Specify the value for secure option --password (The text you type will not be displayed as it is entered): Confirm value for password: Successfully obfuscated the string. uIb5fBeekm0GzQcCQgr8PjcgF9S3HzA76fsyixLYp+Y=

b. Obfuscate for the site_key: beectl obfuscate --expiration_time_in_minutes 0 Specify the value for secure option --password (The text you type will not be displayed as it is entered): Enter value for password: Specify the value for secure option --password (The text you type will not be displayed as it is entered): Confirm value for password: Successfully obfuscated the string. GpQpKOzEZ2z8tlQWZBdUzYYua0KBa7pSeC65bWN6BW3AqSyFzmL4Bw15tdSkNep6

12. Set ORACLE_HOME path.

It is recommended to use the same ORACLE_HOME path across all application tiers. $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/beehive/bin/beectl modify_beectl --new_oracle_home /u01/app/oracle/product/1.5.1.2.0/beehive_1

13. Clone the mid-tier: date > $LOG; beectl clone_midtier --ias_instance_name maa4 --host_name dscbac06.us.oracle.com --db_schema_password uIb5fBeekm0GzQcCQgr8PjcgF9S3HzA76fsyixLYp+Y= --site_key GpQpKOzEZ2z8tlQWZBdUzYYua0KBa7pSeC65bWN6BW3AqSyFzmL4Bw15tdSkNep6 --oui_inv_ptr_loc "/etc/oraInst.loc" --do_not_start_at_end –-obfuscated | tee –a $LOG; date >> $LOG

Fri Jan 15 14:14:11 PST 2010

Note : All validators registered for "clone_midtier" command will be executed now. Note : The validation results will be accumulated and analyzed at the end.


37

Executing "HostName" validator ... Executing "Ias Instance Name" validator ... Successfully executed all validators registered for "clone_midtier" command. Now analyzing the validation results ... "Ias Instance Name" validator has passed with the following information : INFO : Ias instance name from /u01/app/oracle/product/1.5.1.2.0/beehive_1/config/ias.properties : maaXtst.dscbac09.us.oracle.com INFO : Ias instance name from /u01/app/oracle/product/1.5.1.2.0/beehive_1/opmn/conf/opmn.xml : maaXtst.dscbac09.us.oracle.com INFO : Ias instance name from SystemModel : maaXtst.dscbac09.us.oracle.com INFO : Ias instance name from /u01/app/oracle/product/1.5.1.2.0/beehive_1/clone/config/source.properties : maaXtst.dscbac09.us.oracle.com INFO : Ias instance name is same in all the configuration files. Cloning oracle home ... This process will take some time, Please be patient. See the following log file for recent updates ... Log file : "/u01/app/oracle/product/1.5.1.2.0/beehive_1/beehive//logs/config/Clone/<date>/Clone_<time>.log" [WARNING] [2010-01-15 14:14:16] [DataProvider::readIasClonePorts Line 1001 ] [Moving on ...] [WARNING] [2010-01-15 14:14:16] [DataProvider::getOsUser Line 600 ] [getlogin() returned null or empty , Trying with getpwuid(2000) ] [WARNING] [2010-01-15 14:14:16] [DataProvider::getOsUser Line 600 ] [getlogin() returned null or empty , Trying with getpwuid(2000) ] [WARNING] [2010-01-15 14:16:01] [CloneImpl::executeIasCloneScripts Line 503 ] [It is assumed that this is a UNIX environment] [SEVERE ] [2010-01-15 14:18:44] [CloneImpl::executeIasCloneScripts Line 551 ] [] [SEVERE ] [2010-01-15 14:18:44] [CloneImpl::executeIasCloneScripts Line 552 ] [] [SEVERE ] [2010-01-15 14:18:44] [CloneImpl::executeIasCloneScripts Line 553 ] [] [SEVERE ] [2010-01-15 14:18:44] [CloneImpl::executeIasCloneScripts Line 554 ] [----------------------------------------------------------------------------] [WARNING] [2010-01-15 14:18:44] [CloneImpl::executeIasCloneScripts Line 555 ] [Note : Ias clone scripts do not set the valid exit codes after completing the execution.] [WARNING] [2010-01-15 14:18:44] [CloneImpl::executeIasCloneScripts Line 556 ] [Note : So, Please check the ias clone script output message carefully.] [SEVERE ] [2010-01-15 14:18:44] [CloneImpl::executeIasCloneScripts Line 558 ] [----------------------------------------------------------------------------] [SEVERE ] [2010-01-15 14:18:44] [CloneImpl::executeIasCloneScripts Line 559 ] [] [SEVERE ] [2010-01-15 14:18:44] [CloneImpl::executeIasCloneScripts Line 560 ] [] [SEVERE ] [2010-01-15 14:18:44] [CloneImpl::executeIasCloneScripts Line 561 ] [] [SEVERE ] [2010-01-15 14:18:44] [CloneImpl::validateBeehiveInstance Line 581 ] [] [SEVERE ] [2010-01-15 14:18:44] [CloneImpl::validateBeehiveInstance Line 582 ] [----------------------------------------------------------------------------] [WARNING] [2010-01-15 14:18:44] [CloneImpl::validateBeehiveInstance Line 583 ] [Note : It is possible that even after a successful ias cloning, some configurations files may be corrupted.]


38

[WARNING] [2010-01-15 14:18:44] [CloneImpl::validateBeehiveInstance Line 584 ] [Note : Now going to check if ias instance name is updated properly in all configuration files.] [SEVERE ] [2010-01-15 14:18:44] [CloneImpl::validateBeehiveInstance Line 585 ] [----------------------------------------------------------------------------] [SEVERE ] [2010-01-15 14:18:44] [CloneImpl::validateBeehiveInstance Line 586 ] [] [WARNING] [2010-01-15 14:18:48] [CloneImpl::validateBeehiveInstance Line 610 ] [] [WARNING] [2010-01-15 14:18:48] [CloneImpl::validateBeehiveInstance Line 611 ] [----------------------------------------------------------------------------] [WARNING] [2010-01-15 14:18:48] [CloneImpl::validateBeehiveInstance Line 612 ] [Ias cloning was successful and all configuration files are ok.] [WARNING] [2010-01-15 14:18:48] [CloneImpl::validateBeehiveInstance Line 613 ] [Moving on.] [WARNING] [2010-01-15 14:18:48] [CloneImpl::validateBeehiveInstance Line 614 ] [----------------------------------------------------------------------------] [WARNING] [2010-01-15 14:18:48] [CloneImpl::validateBeehiveInstance Line 615 ] [] Successfully cloned the oracle home. Fri Jan 15 14:19:06 PST 2010

Note:

Ignore the SEVERE errors shown in the previous example.

14. As the root user, run the root.sh script from the newly cloned ORACLE_HOME.

15. Modify the new Oracle Beehive instance’s Oracle Wallet settings: beectl modify_property \ --component beehive_instance_maa2.dscbac10.us.oracle.com \ --name WalletDir \ --value /u01/app/oracle/product/1.5.1.5.0/beehive_1/Apache/Apache/conf/ssl.wlt/default beectl activate_configuration beectl modify_local_configuration_files

16. If not already done, add this newly cloned Oracle Beehive application node to the load balancer configuration.

Oracle Beekeeper Install

The steps to install Oracle Beekeeper are in the Oracle Beehive Install guide5. In addition to those steps, if you are installing multiple Beekeeper nodes then you will also need to configure the Beekeeper virtual server as follows:

1. Edit the file <Oracle Beekeeper home>/j2ee/home/config/default-web-site.xml and specify the virtual host name and port number in the <frontend> child element of <web-site>, as follows: <web-site xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation= "http://xmlns.oracle.com/oracleas/schema/11/web-site-11_1.xsd"


39

port="7778" secure="false" protocol="http" display-name="Default Web Site" schema-major-version="11" schema-minor-version="1"> ... <frontend host="beekeeper.oracle.com" port="80" /> ... </web-site>

In this example, beekeeper.oracle.com is the host name of the BigIP LTM virtual host and 80 is the port number.

2. Restart beekeeper: $ORACLE_HOME/opmn/bin/opmnctl stopall $ORACLE_HOME/opmn/bin/opmnctl startall

Oracle Beehive Disaster Recovery Site Setup

The complete set of steps for configuring the Oracle Beehive disaster recovery site are included in the Oracle Beehive Administrator’s Guide [8] in Chapter 16. The example using the aforementioned test environment is included for clarity.

Clone Oracle Beehive Application Tier

When cloning the Oracle Beehive application instances for DR purposes, think of these application nodes as disabled nodes that are part of the same Oracle Beehive configuration that are not enabled until the DR site assumes the primary role. Thus, when creating the DR application instances, use the do_not_start_at_end flag on the clone_midtier command. Follow the same steps listed under “Clone the Application Tier,” with the following changes:

1. Clone the Oracle Beehive application-tier: beectl clone_midtier --ias_instance_name maa3 --host_name dscbac11.us.oracle.com --db_schema_password uIb5fBeekm0GzQcCQgr8PjcgF9S3HzA76fsyixLYp+Y= --site_key q8dzSzBZhQ/z9KprGd73fGrXQOGlxE4zbpl5tXamT1Ix5OtL7++iTg==--oui_inv_ptr_loc "/etc/oraInst.loc" --do_not_start_at_end –-obfuscated

Then repeat this for the second DR application node on dscbac12: beectl clone_midtier --ias_instance_name maa4 --host_name dscbac12.us.oracle.com --db_schema_password uIb5fBeekm0GzQcCQgr8PjcgF9S3HzA76fsyixLYp+Y= --site_key q8dzSzBZhQ/z9KprGd73fGrXQOGlxE4zbpl5tXamT1Ix5OtL7++iTg==--oui_inv_ptr_loc "/etc/oraInst.loc" --do_not_start_at_end –-obfuscated

2. Disable the new DR application-tier instances:

a. List all Oracle Beehive instances: beectl list_components --type BeehiveInstance -----------------+-------------------------------------------------------- Component type | Component identifier


40

-----------------+-------------------------------------------------------- BeehiveInstance | beehive_instance_maa2.dscbac10.us.oracle.com -----------------+-------------------------------------------------------- BeehiveInstance | beehive_instance_maa3.dscbac11.us.oracle.com -----------------+-------------------------------------------------------- BeehiveInstance | beehive_instance_maa4.dscbac12.us.oracle.com -----------------+-------------------------------------------------------- BeehiveInstance | beehive_instance_maaXtst.dscbac09.us.oracle.com -----------------+--------------------------------------------------------


b. Disable each DR application-tier instance: beectl modify_property --component beehive_instance_maa3.dscbac11.us.oracle.com --name Status --value DISABLED Changes to configuration repository are not activated. Successfully stored the property for component id f8fdf6f8-d490-4319-82d6-29e769789317. beectl modify_property --component beehive_instance_maa4.dscbac12.us.oracle.com --name Status --value DISABLED Changes to configuration repository are not activated. Successfully stored the property for component id 402e69b7-fcb2-423f-8fb8-a540ef7b35a3.

c. Activate the changes beectl activate_configuration Now attempting to get writable configuration with maximum wait time 30 seconds. Got writable configuration successfully. Now attempting to activate writable configuration and releasing the lock. Updated new configuration repository successfully. Proposed configuration is saved successfully and activated now.

Physical Standby Setup

This example uses the Oracle Recovery Manager (RMAN) DUPLICATE command to create the Data Guard physical standby database over the network from an active primary database. See the Oracle Database Backup and Recovery User's Guide [9]. There are various options for creating the physical standby database that are documented in the following resources:

• Oracle Documentation: Oracle Data Guard Concepts and Administration

• Oracle Documentation: Oracle Data Guard Broker (see the “Scenarios Using Oracle Enterprise Manager” / Scenario 2: Creating a Configuration or Adding an Additional Standby Database section)

• MAA Web site: “Rapid Oracle RAC Standby Deployment: Oracle Database 11g Release 2”

For this example the standby database unique name will be bhdc as detailed under the previous “Example Test Environment” section. This example will not use the Data Guard broker.

http://www.oracle.com/pls/db111/lookup?id=SBYDB�

http://www.oracle.com/pls/db111/lookup?id=DGBKR�

http://www.oracle.com/pls/db111/lookup?id=DGBKR410�

http://www.oracle.com/pls/db111/lookup?id=DGBKR410�

http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm�

http://www.oracle.com/technology/deploy/availability/pdf/maa_wp_11g_rac_standby.pdf�

http://www.oracle.com/technology/deploy/availability/pdf/maa_wp_11g_rac_standby.pdf�


41

3. Configure the database network files.

Note that the listener.ora must have one and only one address descriptor to run the RMAN network duplicate feature.

Add a static entry to each listener.ora file. For example, for node 1, dscbac05: SID_LIST_LISTENER_DSCBAC05 = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME=bhdb.us.oracle.com) (SID_NAME = bhdb1) (ORACLE_HOME = /u01/app/oracle/product/11.1.0/db_1) ) )

See the “Database Configuration Files” section for complete file details.

4. Copy the database password file from the primary to the standby nodes: export STBY_ORACLE_SID=bhdc1 scp $ORACLE_HOME/dbs/orapw${ORACLE_SID} dscbac07: $ORACLE_HOME/dbs/orapw${STBY_ORACLE_SID} export STBY_ORACLE_SID=bhdc2 scp $ORACLE_HOME/dbs/orapw${ORACLE_SID} dscbac08: $ORACLE_HOME/dbs/orapw${STBY_ORACLE_SID}

5. Create a starter init.ora file on the standby database:

In $ORACLE_HOME/dbs create init${ORACLE_SID}.ora with only the DB_NAME defined. Set this to the same name as the DB_NAME on the primary. For example:

$ cat initbhdc1.ora *.db_name='bhdb'

6. Start instance with NOMOUNT: SQL> startup nomount pfile='/u01/app/oracle/product/11.1.0/db_1/dbs/initbhdc1.start' SQL> create spfile='+DATA/bhdc/spfilebhdc.ora' from pfile='/u01/app/oracle/product/11.1.0/db_1/dbs/initbhdc1.start';

7. Test the primary to standby connection

Make sure you can connect from the primary to the standby as in the following example: sqlplus “sys/welcome1@BHDC1_DGMGRL as sysdba”

This should connect. If your connections are blocked, then you most likely have more than one address descriptor in your listener.ora file.


42

8. Verify that there are no primary database corruptions (optional): export NLS_LANG=AMERICAN_AMERICA.WE8DEC export NLS_DATE_FORMAT="MM/DD/YYYY HH24:MI:SS" rman target / | tee -a rmanBkupValidate`date +%Y%m%d_%H%M%S`.log set echo on run { allocate channel prmy1 type disk; allocate channel prmy2 type disk; allocate channel prmy3 type disk; allocate channel prmy4 type disk; BACKUP VALIDATE DATABASE; }

Or a slower more encompassing method that does archive logs as well as a logical check: export NLS_LANG=AMERICAN_AMERICA.WE8DEC export NLS_DATE_FORMAT="MM/DD/YYYY HH24:MI:SS" rman target / | tee -a rmanBkupValidate`date +%Y%m%d_%H%M%S`.log set echo on run { allocate channel prmy1 type disk; allocate channel prmy2 type disk; allocate channel prmy3 type disk; allocate channel prmy4 type disk; BACKUP VALIDATE CHECK LOGICAL DATABASE ARCHIVELOG ALL; }

9. Run RMAN Network Duplicate.

If the previous sqlplus login check works correctly, then run the following RMAN script (this is an example, modifications may be required):

export NLS_LANG=AMERICAN_AMERICA.WE8DEC export NLS_DATE_FORMAT="MM/DD/YYYY HH24:MI:SS" $ rman target / auxiliary sys/welcome1@BHDC1_DGMGRL | tee -a rmanLog_`date +%Y%m%d_%H%M%S`.txt set echo on run { allocate channel prmy1 type disk; allocate channel prmy2 type disk; allocate channel prmy3 type disk; allocate channel prmy4 type disk; allocate auxiliary channel stby type disk; duplicate target database for standby from active database db_file_name_convert '+DATA/bhdb','+DATA/bhdc' spfile set db_unique_name='bhdc' set control_files='+DATA/bhdc/controlfile/control01.dbf' set instance_number='1' set audit_file_dest='/u01/app/oracle/admin/bhdc/adump' set remote_listener='LISTENERS_bhdc' nofilenamecheck; }


43

10. Create the standby SPFILE:

By default the RMAN duplicate command will create an spfile for the instance that is located in $ORACLE_HOME/dbs directory.

This file will contain entries that refer to the instance names on the primary database. As part of this creation process the database name is being changed to reflect the DB_UNIQUE_NAME for the standby database, and as such the spfile created is essentially worthless. A new spfile is created using the contents of the primary database’s spfile.

1. Verify the location of the control file on the standby: SQL> show parameter control_files NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ control_files string +DATA/bhdc/controlfile/control01.dbf

2. Create and edit the standby PFILE. a. On the primary, create a PFILE for editing:

SQL> create pfile='?/dbs/initbhdc1.pfile' from spfile; File created.

b. Copy the PFILE from the primary to the standby. scp $ORACLE_HOME/dbs/initbhdc1.pfile \ dscbac07:$ORACLE_HOME/dbs/initbhdc1.pfile

c. Edit the PFILE, $ORACLE_HOME/dbs/initbhdc1.pfile, and globally change all primary instance references to the new standby instance names:

For example, see the example standby spfile (changes are shown in red) bhdc2.local_listener='LISTENER_BHDC2' bhdc1.local_listener='LISTENER_BHDC1'

d. Create the init.ora startup file: $ cat initbhdc1.spfile SPFILE='+DATA/bhdc/spfilebhdc.ora'

3. Create the SPFILE: SQL> create SPFILE='+DATA/bhdc/spfilebhdc.ora' from pfile='?/dbs/initbhdc1.pfile'; File created.

4. Remove the RMAN created spfile from $ORACLE_HOME/dbs located on standby node 1 rm $ORACLE_HOME/dbs/spfilebhdc1.ora

5. Setup the default init.ora file and remount the database with it: $ cd $ORACLE_HOME/dbs $ cp initbhdc1.spfile initbhdc1.ora $ scp initbhdc1.ora dscbac08:`pwd`/initbhdc2.ora

Standby node 1 (dscbac07): SQL> shutdown immediate


44

SQL> startup mount

Standby node 2 (dscbac08):

SQL> startup mount

11. Create secondary control files: SQL> shutdown immediate SQL> startup nomount SQL> alter system set control_files='+DATA', '+FRA' scope=spfile; System altered. Use RMAN to duplicate the existing control file: $ rman target / Recovery Manager: Release 11.1.0.7.0 - Production on Tue Aug 11 07:57:25 2009 Copyright (c) 1982, 2007, Oracle. All rights reserved. connected to target database: BHDB (not mounted) RMAN> restore controlfile from '+DATA/bhdc/controlfile/control01.dbf'; Starting restore at 11-AUG-09 using target database control file instead of recovery catalog allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=137 instance=bhdc1 device type=DISK channel ORA_DISK_1: copied control file copy output file name=+DATA/bhdc/controlfile/current.275.694598303 output file name=+FRA/bhdc/controlfile/current.256.694598303 Finished restore at 11-AUG-09 SQL> startup force mount SQL> show parameter control_files NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ control_files string +DATA/bhdc/controlfile/current.275.694598303, +FRA/bhdc/controlfile/current.256.694598303

12. Ensure server side load balancing is configured:

See the tnsnames.ora file for entries. You should perform this task in the spfile creation steps. SQL> show parameter list NAME TYPE VALUE ------------------------------------ ----------- ----------------------------- local_listener string LISTENER_BHDC1 remote_listener string LISTENERS_bhdc

13. Register the new standby database with CRS: srvctl add database –d bhdc –o $ORACLE_HOME –m us.oracle.com –p “+DATA/bhdc/spfilebhdc.ora” –n bhdb –r physical_standby srvctl add instance –d bhdc –i bhdc1 –n dscbac07 srvctl add instance –d bhdc –i bhdc2 –n dscbac08


45

14. Create standby redo logs on the primary and standby:

If the primary already had standby redo logs, then the new standby will also have them.

1. Determine if the standby redo logs already exist on the standby: SQL> col member format A45 select * from v$logfile where type='STANDBY';

GROUP# STATUS TYPE MEMBER IS_ ---------- ------- ------- --------------------------------------------- --- 5 STANDBY +DATA/bhdb/onlinelog/group_5.294.706417063 NO 5 STANDBY +FRA/bhdb/onlinelog/group_5.4940.706417065 YES 6 STANDBY +DATA/bhdb/onlinelog/group_6.296.706417067 NO 6 STANDBY +FRA/bhdb/onlinelog/group_6.4863.706417071 YES 7 STANDBY +DATA/bhdb/onlinelog/group_7.305.706417073 NO 7 STANDBY +FRA/bhdb/onlinelog/group_7.4834.706417075 YES 8 STANDBY +DATA/bhdb/onlinelog/group_8.302.706417079 NO 8 STANDBY +FRA/bhdb/onlinelog/group_8.4832.706417081 YES 9 STANDBY +DATA/bhdb/onlinelog/group_9.304.706417085 NO 9 STANDBY +FRA/bhdb/onlinelog/group_9.4831.706417087 YES 10 rows selected.

2. If the standby redo logs do not exist, then add them and make them the same size as the online redo logs: ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 5 SIZE 1G, GROUP 6 SIZE 1G, GROUP 7 SIZE 1G; ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 8 SIZE 1G, GROUP 9 SIZE 1G, GROUP 10 SIZE 1G;

3. If the standby redo logs do not exist on the primary, then also add them to the primary and make them the same size as the online redo logs. This is necessary to prepare for a switchover or failover.

15. Set up Data Guard parameters.

Note: At this stage, you can set up the Data Guard broker, if desired. This example does not use the Data Guard broker. For details on setting up the Data Guard Broker, see the Oracle Data Guard Broker documentation.

Both Databases alter system set log_archive_config=’dg_config=(bhdb,bhdc)’; alter system set standby_file_management=’AUTO’;

Primary Database alter system set fal_client=’bhdb’; alter system set fal_server=’bhdc’; alter system set log_archive_dest_2=’service=bhdc db_unique_name=bhdc VALID_FOR=(ALL_LOGFILES,PRIMARY_ROLE) ‘; alter database set standby database to maximize availability;

http://www.oracle.com/pls/db111/lookup?id=DGBKR�


46

If using different data file and/or log file paths: alter system set db_file_name_convert='+DATA/bhdc','+DATA/bhdb'; alter system set log_file_name_convert='+FRA/bhdc','+FRA/bhdb';

Standby Database

This could’ve been done in the spfile creation steps: alter system set fal_client=’bhdc’; alter system set fal_server=’bhdb’; alter system set log_archive_dest_2=’service=bhdb db_unique_name=bhdb VALID_FOR=(ALL_LOGFILES,PRIMARY_ROLE) ‘; If using different data file and/or log file paths: alter system set db_file_name_convert='+FRA/bhdb','+FRA/bhdc' alter system set log_file_name_convert='+FRA/bhdb','+FRA/bhdc'

16. Set the data protection mode on the primary. For example: alter database set standby database to maximize availability;

17. Turn on Flashback Database on primary and standby:

If flashback database is not already on then: SQL> alter database flashback on;

18. Enable real-time apply on the standby database: recover managed standby database using current logfile disconnect;

19. Enable redo transport on the primary database: alter system set log_archive_dest_state_2=enable;

20. Verify redo transport and apply are working:

See the following sections in the Oracle Data Guard Concepts and Administration 11g Release 1 guide: 1. Monitoring redo transport: 6.3 Monitoring Redo Transport Services 2. Monitoring redo apply: 7.3.3 Monitoring Redo Apply on Physical Standby Databases

Configure the Oracle Beehive Application Tier with a Physical Standby

1. Ensure that the same CRS managed services and Oracle RAC FAN callout exist on the standby.

Follow same steps as when creating the primary database services, just using the new DB NAME.

a. Verify the primary database, bhdb, services: srvctl config service -d bhdb beehive_bs PREF: bhdb1 bhdb2 AVAIL: bhaffsvc1 PREF: bhdb1 AVAIL: bhaffsvc2 PREF: bhdb2 AVAIL: search PREF: bhdb2 AVAIL: bhdb1

b. Add the same services to the standby: srvctl add service -d bhdc -s beehive_bs

http://www.oracle.com/pls/db111/lookup?id=SBYDB�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28294/log_transport.htm#BABDGGID�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28294/log_apply.htm#BGBHGCED�


47

srvctl add service -d bhdc -s bhaffsvc1 -r bhdc1 srvctl add service -d bhdc -s bhaffsvc2 -r bhdc2 srvctl add service -d bhdc -s search -r bhdc2 –a bhdc1

c. Set up the Oracle RAC callout:

Copy the RAC FAN Callout Start Services Script. to the CRS Home racg/usrco directory on each node: export ORACLE_CRS_HOME=/u01/app/11.1.0/crs cp /tmp/startSRVonUp.pl $ORACLE_CRS_HOME/racg/usrco

2. Update the Database System Object with ONS Port to include each standby node.

Follow same steps as when configuring ONS on the primary, the ONS ports should be the same.

Modify the Oracle Beehive server’s database system object's configuration to add the ONS remote ports that are listening on your standby Oracle RAC nodes:

beectl modify_database --database _current_site:Database \ --ons_entry dscbac05.us.oracle.com:6251 \ --ons_entry dscbac06.us.oracle.com:6251 \ --ons_entry dscbac07.us.oracle.com:6251 \ --ons_entry dscbac08.us.oracle.com:6251

3. Modify the bootstrap connect string.

Make sure to include primary and standby database nodes: beectl modify_bootstrap_configuration --connect_string \ "(DESCRIPTION=(ADDRESS = (PROTOCOL = TCP)(HOST = dscbac05-vip)(PORT = 1523))(ADDRESS = (PROTOCOL = TCP)(HOST = dscbac06-vip)(PORT = 1523))(ADDRESS = (PROTOCOL = TCP)(HOST = dscbac07-vip)(PORT = 1523))(ADDRESS = (PROTOCOL = TCP)(HOST = dscbac08-vip)(PORT = 1523))(LOAD_BALANCE = yes)(ENABLE=BROKEN)(FAILOVER=on)(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = beehive_bs.us.oracle.com)))"

4. Set up affinity service connect string.

Make sure to pair each primary service with a corresponding standby node: beectl modify_property --component _CURRENT_SITE:Database \ --name AffinityServiceNames \ --value ‘((DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dscbac05-vip.us.oracle.com)(PORT=1523))(ADDRESS=(PROTOCOL=TCP)(HOST=dscbac07-vip.us.oracle.com)(PORT=1523))(LOAD_BALANCE=yes)(FAILOVER=on)(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=bhaffsvc1.us.oracle.com)))’ \ --value ‘((DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dscbac06-vip.us.oracle.com)(PORT=1523))(ADDRESS=(PROTOCOL=TCP)(HOST=dscbac08-vip.us.oracle.com)(PORT=1523))(LOAD_BALANCE=yes)(FAILOVER=on)(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=bhaffsvc2.us.oracle.com)))’

5. Update Search connect string.

If using isolated Oracle Beehive Search database instances then update the Search service accordingly: beectl modify_property --component _CURRENT_SITE:Database --name SearchServiceName \


48

--value '(DESCRIPTION=(ADDRESS_LIST=\ (ADDRESS=(PROTOCOL=TCP)(HOST=dscbac05-vip.us.oracle.com)(PORT=1523)) (ADDRESS=(PROTOCOL=TCP)(HOST=dscbac06-vip.us.oracle.com)(PORT=1523)) (ADDRESS=(PROTOCOL=TCP)(HOST=dscbac07-vip.us.oracle.com)(PORT=1523)) (ADDRESS=(PROTOCOL=TCP)(HOST=dscbac08-vip.us.oracle.com)(PORT=1523))) (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=search.us.oracle.com)))'

6. Modify outbound connect timeout to 3 seconds: beectl modify_property --component _current_site:Database:DefaultNonXaPool --name ConnectionProperties \ --value oracle.net.ns.SQLnetDef.TCP_CONNTIMEOUT_STR:3000 beectl modify_property --component _current_site:Database:AffinityPool --name ConnectionProperties \ --value oracle.net.ns.SQLnetDef.TCP_CONNTIMEOUT_STR:3000

7. Activate the changes: beectl activate_configuration beectl modify_local_configuration_files

Load Balancer Configuration

As stated earlier, the DR site application server nodes are disabled until the DR site assumes the primary role. In most DR cases, the DR site is over a WAN and any role transitions require use of a WAN Traffic Manager. The configuration of a WAN traffic manager is outside the scope of this paper. The high-level architecture with a WAN traffic manager is illustrated earlier in Figure 3 Oracle Beehive with DR Remote StandbyOracle Beehive. The WAN traffic manager serves the purpose of transparently failing over clients to the new primary application tier. At this stage of the DR site setup, ensure that the DR site application nodes are configured into the WAN traffic manager.

For a complete discussion of the WAN traffic manager role in a site role transition see the Oracle Database High Availability Best Practices [2] guide in section 4.2.1.3 Repair Solution.

Test Role Transitions

Switchover to Remote Standby

1. Shutdown all other instances on primary and standby.

Only a single instance should be up on each database.

2. Shutdown all primary mid-tiers: beectl stop –all on all mid-tiers

3. Perform a switchover to transition the primary database to a standby: ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN; SHUTDOWN IMMEDIATE; STARTUP MOUNT;

4. Switchover the standby database to a primary:

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/outage.htm#BABHDCAJ�


49

ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN; ALTER DATABASE OPEN;

5. Disable old primary database mid-tiers: beectl list_components --type BeehiveInstance For each old mid-tier instance: beectl modify_property --component <component_id> --name Status --value DISABLED

6. Enable new primary mid-tiers: beectl list_components --type BeehiveInstance For each new primary mid-tier instance: beectl modify_property --component <component_id> --name Status --value ENABLED

7. Startup all new primary site mid-tiers:

On all new primary site mid-tiers, enter the following: beectl start –all

8. Start applying redo on the new standby database: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;

Failover to the Remote Standby Database

1. Kill the primary by issuing the SHUTDOWN ABORT command on each instance.

2. Perform a database failover:

Issue the following SQL*Plus statements: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH; ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN; ALTER DATABASE OPEN; srvctl start instance -d bhdc -i bhdc2

3. Stop primary mid-tiers:

If you can reach primary mid-tiers then stop each one: beectl stop –all

4. Disable old mid-tiers: beectl list_components --type BeehiveInstance For each old mid-tier instance: beectl modify_property --component <component_id> --name Status --value DISABLED


50

5. Enable new mid-tiers: beectl list_components --type BeehiveInstance For each new mid-tier instance: beectl modify_property --component <component_id> --name Status --value ENABLED

6. Start new primary mid-tiers: beectl start --all

7. Reinstate the original primary database as a standby database:

a. On the new primary database, get the flashback SCN: SELECT TO_CHAR(STANDBY_BECAME_PRIMARY_SCN) FROM V$DATABASE;

b. On the old primary database: startup mount FLASHBACK DATABASE TO SCN (standby_became_primary_scn SCN from #1) ; ALTER DATABASE CONVERT TO PHYSICAL STANDBY; SHUTDOWN IMMEDIATE; startup mount recover managed standby database through all switchover using current logfile disconnect

Oracle Beehive Local HA Standby Setup

These steps are very similar to the DR site setup except that the same application tiers are used during a role transition (failover or switchover) and are not required to be restarted for a failover. Thus, there is no cloning of application-tier nodes required.

This example shows having only a local standby database. It is possible to have a local and a remote standby, or even more standby databases. For details about working with multiple standby databases, see the “Managing Data Guard Configurations Having Multiple Standby Databases - Best Practices” MAA white paper.

Perform the following set up steps.

1. Create the physical standby using the same steps you performed in the “Physical Standby Setup” section. The example in the following steps use the same database unique name for the local standby, bhdc.

2. Set up the application tier to work with the new database as detailed under “Configure the Oracle Beehive Application Tier with a Physical Standby” section.

http://www.oracle.com/technology/deploy/availability/pdf/maa10gr2multiplestandbybp.pdf�

http://www.oracle.com/technology/deploy/availability/pdf/maa10gr2multiplestandbybp.pdf�


51

3. Set up the Oracle FAN ONS Publisher:

Note that FAN/FCF is only used when the same mid-tiers are kept active. For example, for a local standby failover. In a remote DR failover when a new set of mid-tiers is used, then FAN/FCF is not used.

a. Configure the FAN ONS Publisher Configuration File:

$ORACLE_HOME/dbs/cfo${ORACLE_SID}.ora bhdb peer=bhdc bhdc peer=bhdb bhdb service=beehive_bs location=dscbac05,bhdb1:dscbac06,bhdb2 bhdb service=search location=dscbac06,bhdb2 bhdb service=bhaffsvc1 location=dscbac05,bhdb1 bhdb service=bhaffsvc2 location=dscbac06,bhdb2 bhdc service=beehive_bs location=dscbac07,bhdc1:dscbac08,bhdc2 bhdc service=search location=dscbac08,bhdc2 bhdc service=bhaffsvc1 location=dscbac07,bhdc1 bhdc service=bhaffsvc2 location=dscbac08,bhdc2

b. Copy the cfo file to each node:

scp $ORACLE_HOME/dbs/cfo${ORACLE_SID}.ora

dscbac06:$ORACLE_HOME/dbs/cfobhdb2.ora


dscbac07:$ORACLE_HOME/dbs/cfobhdc1.ora


dscbac08:$ORACLE_HOME/dbs/cfobhdc2.ora

c. Build a wrapper script around the Publisher program: $ORACLE_HOME/bin/cfo.sh #!/bin/ksh export TZ=PST8PDT export ORACLE_SID=bhdb1 export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 export LD_LIBRARY_PATH=$ORACLE_HOME/lib export PATH=$ORACLE_HOME/bin:$PATH $ORACLE_HOME/bin/cfo r echo `date` "executed $0" > /tmp/cfo.log

d. Copy the cfo.sh script to each node: scp $ORACLE_HOME/bin/cfo.sh dscbac06:$ORACLE_HOME/bin/cfo.sh scp $ORACLE_HOME/bin/cfo.sh dscbac07:$ORACLE_HOME/bin/cfo.sh scp $ORACLE_HOME/bin/cfo.sh dscbac08:$ORACLE_HOME/bin/cfo.sh

e. Set the appropriate ORACLE_SID in cfo.sh on each node: dscbac06: export ORACLE_SID=bhdb2 dscbac07: export ORACLE_SID=bhdc1 dscbac08: export ORACLE_SID=bhdc2


52

4. Create a database role change trigger on the primary database: CREATE OR REPLACE TRIGGER ons_JDBCpublish AFTER DB_ROLE_CHANGE ON DATABASE BEGIN dbms_scheduler.create_job( job_name=>'publish_events', job_type=>'executable', job_action=>'/u01/app/oracle/product/11.1.0/db/bin/cfo.sh', enabled=>TRUE ); END;

5. Test role transitions:

6. Follow the same steps as described in the “Test Role Transitions” section, except (because this is a local standby) it is not required to enable/disable the application-tier nodes. Additionally, for a failover, you do not need to stop and start the application-tier nodes. For example, the application tier nodes should remain up throughout a failover.

Database Configuration Files

Primary Site Server Parameter File

*.audit_file_dest='/u01/app/oracle/admin/bhdb/adump' *.audit_trail='db' *.cluster_database_instances=2 *.cluster_database=TRUE *.compatible='11.1.0.7.0' *.control_files='+DATA/bhdb/controlfile/control01.dbf' *.db_block_size=8192 *.db_create_file_dest='+DATA' *.db_domain='us.oracle.com' *.db_name='bhdb' *.db_recovery_file_dest='+FRA' *.db_recovery_file_dest_size=53687091200 *.db_writer_processes=4 *.dg_broker_start=FALSE *.diagnostic_dest='/u01/app/oracle' *.dispatchers='(PROTOCOL=TCP) (SERVICE=bhdbXDB)' *.event='' *.fal_client='bhdb' *.fal_server='bhdc' *.fast_start_mttr_target=10 *.fast_start_parallel_rollback='HIGH' bhdb1.instance_number=1 bhdb2.instance_number=2 *.java_pool_size=52428800 *.job_queue_processes=10 bhdb2.local_listener='LISTENER_BHDB2' bhdb1.local_listener='LISTENER_BHDB1' *.log_archive_config='dg_config=(bhdb,bhdc)' *.log_archive_dest_2='service=bhdc db_unique_name=bhdc VALID_FOR=(ALL_LOGFILES,PRIMARY_ROLE) ' *.log_archive_dest_state_2='ENABLE'


53

*.log_archive_trace=8191 *.open_cursors=300 *.parallel_execution_message_size=65535 *.pga_aggregate_target=11889803264 *.processes=1000 *.remote_listener='LISTENERS_BHDB' *.remote_login_passwordfile='exclusive' *.service_names='bhdb.us.oracle.com' *.sga_target=1610612736 *.standby_file_management='AUTO' *.streams_pool_size=104857600 bhdb2.thread=2 bhdb1.thread=1 *.undo_retention=3600 bhdb2.undo_tablespace='UNDOTBS2' bhdb1.undo_tablespace='UNDOTBS1'

Primary Site listener.ora Files

Node 1, dscbac05

# listener.ora.dscbac05 Network Configuration File: /u01/app/oracle/product/11.1.0/asm_1/network/admin/listener.ora # Generated by Oracle configuration tools. LISTENER_DSCBAC05 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac05-vip) (PORT = 1523)(RATE_LIMIT=YES)(QUEUESIZE=1024)(IP = FIRST)) ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.204.32.205) (PORT = 1523)(IP = FIRST)) ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC)) ) ) ) SID_LIST_LISTENER_DSCBAC05 = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME=bhdb.us.oracle.com) (SID_NAME = bhdb1) (ORACLE_HOME = /u01/app/oracle/product/11.1.0/db_1) ) ) # Set listener connection rate throttling CONNECTION_RATE_LISTENER_DSCBAC05=30


54

Node 2, dscbac06

LISTENER_DSCBAC06 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac01-v6)(PORT = 1523)(RATE_LIMIT=YES)(QUEUESIZE=1024)(IP = FIRST)) ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.204.32.206)(PORT = 1523)(IP = FIRST)) ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC)) ) ) ) SID_LIST_LISTENER_DSCBAC06 = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME=bhdb.us.oracle.com) (SID_NAME = bhdb2) (ORACLE_HOME = /u01/app/oracle/product/11.1.0/db_1) ) ) CONNECTION_RATE_LISTENER_DSCBAC06=30

tnsnames.ora File for All Sites

# tnsnames.ora Network Configuration File # (same on all nodes, primary & standby) # /u01/app/oracle/product/11.1.0/db_1/network/admin/tnsnames.ora BHDB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac05-vip)(PORT = 1523)) (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac06-vip)(PORT = 1523)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = bhdb.us.oracle.com) ) ) BHDC = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac09-vip)(PORT = 1523)) (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac08-vip)(PORT = 1523)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = bhdc.us.oracle.com) )


55

) LISTENERS_BHDB = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac05-vip)(PORT = 1523)) (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac06-vip)(PORT = 1523)) ) LISTENERS_BHDC = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac09-vip)(PORT = 1523)) (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac08-vip)(PORT = 1523)) ) LISTENER_BHDB1 = (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac05-vip)(PORT = 1523)) LISTENER_BHDB2 = (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac06-vip)(PORT = 1523)) LISTENER_BHDC1 = (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac09-vip)(PORT = 1523)) LISTENER_BHDC2 = (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac08-vip)(PORT = 1523)) # Connect string for RMAN Network duplicate BHDC1_DGMGRL = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac09-vip)(PORT = 1523)) (CONNECT_DATA = (SERVER = DEDICATED) (SID = bhdc1) ) )

Standby Site Server Parameter File

*.audit_file_dest='/u01/app/oracle/admin/bhdc/adump' *.audit_trail='db' *.cluster_database_instances=2 *.cluster_database=true *.compatible='11.1.0.7.0' *.control_files='+DATA/bhdc/controlfile/control01.dbf' *.db_block_size=8192 *.db_create_file_dest='+DATA' *.db_domain='us.oracle.com' *.db_name='bhdb' *.db_recovery_file_dest='+FRA' *.db_recovery_file_dest_size=5368709120000 *.db_unique_name='bhdc' *.db_writer_processes=4 *.dg_broker_start=FALSE *.diagnostic_dest='/u01/app/oracle' *.dispatchers='(PROTOCOL=TCP) (SERVICE=bhdbXDB)' *.fal_client='bhdc'


56

*.fal_server='bhdb' *.fast_start_mttr_target=10 *.fast_start_parallel_rollback='HIGH' bhdc1.instance_number=1 bhdc2.instance_number=2 *.java_pool_size=52428800 *.job_queue_processes=0 *.large_pool_size=1073741824 bhdc1.local_listener='LISTENER_BHDC1' bhdc2.local_listener='LISTENER_BHDC2' *.log_archive_config='dg_config=(bhdb,bhdc)' *.log_archive_dest_2='service=bhdb db_unique_name=bhdb VALID_FOR=(ALL_LOGFILES,PRIMARY_ROLE) ' *.log_archive_dest_state_2='ENABLE' *.log_archive_trace=8191 *.open_cursors=300 *.parallel_execution_message_size=65535 *.pga_aggregate_target=11889803264 *.processes=1000 *.remote_listener='LISTENERS_bhdc' *.remote_login_passwordfile='exclusive' *.service_names='bhdc.us.oracle.com' *.sga_target=3145728000 *.standby_file_management='AUTO' bhdc1.thread=1 bhdc2.thread=2 *.undo_retention=3600 bhdc1.undo_tablespace='UNDOTBS1' bhdc2.undo_tablespace='UNDOTBS2'

Standby Site listener.ora Files

Node 3, dscbac07

LISTENER_DSCBAC07 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac01-v7)(PORT = 1523)(RATE_LIMIT=YES)(QUEUESIZE=1024)(IP = FIRST)) ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.204.32.207)(PORT = 1523)(IP = FIRST)) ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC)) ) ) ) SID_LIST_LISTENER_DSCBAC07 = (SID_LIST = (SID_DESC = (SID_NAME = bhdc1) (ORACLE_HOME = /u01/app/oracle/product/11.1.0/db_1)


57

) ) CONNECTION_RATE_LISTENER_DSCBAC07=30

Node 4, dscbac08

LISTENER_DSCBAC08 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dscbac01-v8)(PORT = 1523)(RATE_LIMIT=YES)(QUEUESIZE=1024)(IP = FIRST)) ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.204.32.208)(PORT = 1523)(IP = FIRST)) ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC)) ) ) ) SID_LIST_LISTENER_DSCBAC08 = (SID_LIST = (SID_DESC = (SID_NAME = bhdc2) (ORACLE_HOME = /u01/app/oracle/product/11.1.0/db_1) ) ) CONNECTION_RATE_LISTENER_DSCBAC08=30

Create Users Script

This script can be used to automate the creation of Oracle Beehive users for testing purposes. Script is named bulk_create_users.sh bulk_create_users.sh -help usage : ./bulk_create_users.sh [-bh <beehive_home] [-prefix <user_prefix>] [-passwd <uniform_pasword>] [-domain <email_domain>] [-startId <start_id>] -endId <end_id> [-p <parallel_degree] [-binDir <dir_of_scripts>] [ -includeDomainInName] -bh: required if BEEHIVE_HOME not set. default is BEEHIVE_HOME -prefix: optional. default is bhtest -passwd: optional. default is Welcome1 -domain: optional. default is oracle.com -startId: optional: default is 1 -endId: required -p: parellel_degree. optional. default is 5


58

bulk_create_users.sh Script # # Script to create multiple Oracle Beehive users # echoUsage() { echo "usage : $0 [-bh <beehive_home] [-prefix <user_prefix>] [-passwd

<uniform_pasword>] [-domain <email_domain>] [-startId <start_id>] -endId <end_id> [-p <parallel_degree] [-binDir <dir_of_scripts>] [ -includeDomainInName]"

echo " -bh: required if BEEHIVE_HOME not set. default is BEEHIVE_HOME" echo " -prefix: optional. default is bhtest" echo " -passwd: optional. default is Welcome1" echo " -domain: optional. default is oracle.com" echo " -startId: optional: default is 1" echo " -endId: required " echo " -p: parellel_degree. optional. default is 5" echo "Example: " echo " $0 -prefix bhtest -passwd Welcome1 -domain oracle.com -startId 1 -

endId 10" } beehiveHome="" userprefix="bhtest" passwd="Welcome1" emaildomain="oracle.com" startid=1 stopid=0 NUM_PARALLEL=5 binDir="." includeDomainOpt="" if [[ "$1" = "-help" ]]; then echoUsage exit 1 fi while [[ $1 = -* ]]; do case $1 in -bh ) beehiveHome=$2 shift;; -enprId ) enprId=$2 shift;; -prefix ) userprefix=$2 shift;; -passwd ) passwd=$2 shift;; -domain ) emaildomain=$2 shift;; -startId ) startid=$2 shift;; -endId ) stopid=$2 shift;; -outputFile ) tmpfile=$2 shift;; -binDir ) binDir=$2 shift;; -p ) NUM_PARALLEL=$2 shift;; -includeDomainInName ) includeDomainOpt=" -includeDomainInName" ;; * ) echo "ERROR: invalid option: $1.";echoUsage


59

exit 1 esac shift done if [[ "$beehiveHome" = "" ]]; then if [ "$BEEHIVE_HOME" = "" ]; then echo "-bh option is required or define BEEHIVE_HOME environment variable." exit 1 fi else export BEEHIVE_HOME=$beehiveHome fi if [[ $stopid -lt $startid ]]; then echo "ERROR: -endId is required or given endId is invalid." echoUsage exit 1 fi START_PARALLEL=0 let "NO_USERS_EACH=($stopid-$startid+1)/$NUM_PARALLEL" if [[ $NO_USERS_EACH -lt 1 ]]; then NUM_PARALLEL=1 let "NO_USERS_EACH=($stopid-$startid+1)" fi let "LEFT=$TOTALUSER-$NO_USERS_EACH*$NUM_PARALLEL" let "currentstopid=$startid-1" echo "Getting enterprise collabId..." ECID="`$BEEHIVE_HOME/beehive/bin/beectl list_properties --component

_CURRENT_ENTERPRISE --no_wrap --no_stagger |grep CollabId | awk '{print $3}'`" echo "Enterprise CollabId: $ECID" if [[ "$ECID" = "" ]]; then echo "ERROR: Enterprise CollabId is unexpectedly empty. Exiting.." exit 1 fi startTime="`date`" sigFile="/tmp/bhCreateUsers`date +%N`.tmp" while [ $START_PARALLEL -lt $NUM_PARALLEL ] do let "currentstartid=$currentstopid+1" let "currentstopid=$currentstartid+$NO_USERS_EACH-1" if [ $LEFT -gt 0 ] then let "currentstopid+=1" let "LEFT-=1" fi if [ $currentstopid -gt $stopid ] then let "currentstopid=$stopid" fi echo "Creating $userprefix$currentstartid to $userprefix$currentstopid ..." $binDir/create_users.sh -bh $BEEHIVE_HOME -enprId $ECID -prefix $userprefix

-passwd $passwd -domain $emaildomain -startId $currentstartid -endId $currentstopid -sigFile $sigFile $includeDomainOpt &


60

sleep 10 let "START_PARALLEL=$START_PARALLEL+1" done let "expectedNums=$NUM_PARALLEL+$NUM_PARALLEL" while [ "$end" != "true" ] do num=`/bin/cat $sigFile | /usr/bin/wc -l` if [[ $num -ge $expectedNums ]]; then end="true" fi sleep 10 done echo "Exit timestamp `date` (started at $startTime)"

Oracle RAC FAN Callout Start Services Script #!/usr/bin/perl -w # Callout program that will, on an INSTANCE UP event start any services defined against # this database. This is to address the issue of INSTANCE STOP setting non-uniform service state # to OFFLINE. # Note: Running services will not be relocated. # # CHANGE History: 013-JUN-2006 TANTHONY Created use strict; # Replace the following variables with appropriate values my $CRS_HOME="/u01/app/11.1.0/crs"; my $ORACLE_HOME="/u01/app/oracle/product/11.1.0/db_1"; my $GetHOST = "/bin/hostname -s"; # TMP refers to the log location only my $TMP = "/tmp"; # # # Logging enabled my $LOGFILE = "$TMP/SRV_co.log"; # my $instance; my $database; my $host; my $service; my $state; my $reason; my $card; my $status; my ($key,$value) = ""; my $myHost = ""; my ($myServ) = ""; my ($mysrvlist) = ""; # # Open a logfile local *LOG_FILE;


61

open (LOG_FILE, ">>$LOGFILE") or do { print "Cannot open $LOGFILE\n"; exit(1); }; # Determine this host system("$GetHOST > $TMP/myhost"); local *TEMP_FILE; open ( TEMP_FILE,"$TMP/myhost") or do { print "Cannot determine hostname\n"; exit(1); }; while (<TEMP_FILE>) { chomp; $myHost = $_; }; close(TEMP_FILE); # Uncomment these lines if only interested in specific events if ($ARGV[0] ne "INSTANCE") { exit(0); }; #if ($ARGV[0] ne "SERVICEMEMBER") { exit(0); }; #if ($ARGV[0] ne "SERVICE") { exit(0); }; #if ($ARGV[0] ne "NODE") { exit(0); }; for (my $i=0; $i <= $#ARGV; $i++) { print LOG_FILE "$i $ARGV[$i]\n"; if ($ARGV[$i] =~ m#=#) { ($key,$value) = (split /=/, $ARGV[$i]); print LOG_FILE "Key = $key Value = $value\n"; if ($key eq "service") { $service = $value; } elsif ($key eq "instance") { $instance = $value; $ENV{ORACLE_SID} = $value; } elsif ($key eq "database") { $database = $value; } elsif ($key eq "host") { $host = $value; } elsif ($key eq "card") { $card = $value; } elsif ($key eq "status") { $status = $value; } elsif ($key eq "reason") { $reason = $value; } } } print LOG_FILE "Host=$host DB=$database Inst=$instance Service=$service Status=$status Reason=$reason MyHost=$myHost\n"; # # The following function will set service state such that they will restart # if ($host eq "$myHost") { if ($status eq "up" && $ARGV[0] eq "INSTANCE") { print LOG_FILE "Attempting set of service state for database: $database for instance $instance\n";


62

# Determine services associated with this database srvMap($database, $instance); } else { } } else { print LOG_FILE "Event generated on a different node\n"; } # Sub routine to start services defined against a particular database sub srvMap { my ($dbIn, $instanceIn) = @_; local*SRVFILE; #print LOG_FILE "In srvMap subroutine for $dbIn\n"; system("date +'%D %H:%M:%S.%N' >> /tmp/SRV_co.log") ; # # Identify services defined for this database print LOG_FILE "DB=$dbIn ORACLE_HOME=$ORACLE_HOME \n"; system("$ORACLE_HOME/bin/srvctl config service -d $dbIn > $TMP/serviceMap-$dbIn.out"); open (SRVFILE,"$TMP/serviceMap-$dbIn.out") or do { print "Cannot open SRVFILE\n"; exit(1); }; while (<SRVFILE>) { chomp; if ($_ =~ m#$instanceIn #) { ($mysrvlist) = ($_ =~ m#PREF: (.*) AVAIL#); #print LOG_FILE "$mysrvlist"; if (($mysrvlist eq $instanceIn) || ($mysrvlist =~ m#$instanceIn #) || ($mysrvlist =~ m#$instanceIn$#)) { ($myServ) = ($_ =~ m#^([\w]+) #); print LOG_FILE "Starting Service $myServ on instance $instanceIn \n"; system("$ORACLE_HOME/bin/srvctl start service -d $dbIn -s $myServ -i $instanceIn"); } } }; system("date +'%D %H:%M:%S.%N' >> /tmp/SRV_co.log") ; print LOG_FILE "Routine complete\n"; }


63

References

1. Oracle Maximum Availability Architecture Web site http://www.otn.oracle.com/goto/maa

2. Oracle Database High Availability Overview

● 11.1 (Part #B14210) http://www.oracle.com/pls/db111/lookup?id=HAOVW

● 11.2 (Part # E10804-02 ) http://www.oracle.com/pls/db112/lookup?id=HAOVW

3. Oracle Database High Availability Best Practices (Part B25159) http://www.oracle.com/pls/db111/lookup?id=HABPT

4. Oracle Beehive Concepts (Part # E16644-02) http://download.oracle.com/docs/cd/E16671_01/bh.200/e16644/toc.htm

5. Oracle Beehive Installation Guide for Linux (Part Number E16641-02) http://download.oracle.com/docs/cd/E16671_01/bh.200/e16641/toc.htm

6. Oracle Real Application Clusters Administration and Deployment Guide 11g Release 1 (11.1) (Part Number B28254-07) http://www.oracle.com/pls/db111/lookup?id=RACAD

7. Oracle Enterprise Manager Concepts 10g Release 5 (10.2.0.5) (Part Number B31949-10) http://download.oracle.com/docs/cd/B16240_01/doc/em.102/b31949/oracle_site_management.htm#OEMCN9327

8. Oracle Beehive Administrator’s Guide Release 2 (2.0) (Part Number E16648-04) http://download.oracle.com/docs/cd/E16671_01/bh.200/e16648/toc.htm

9. Oracle Database Backup and Recovery User's Guide (11.1) (Part Number B28270-03) http://www.oracle.com/pls/db111/lookup?id=BRADV

http://www.otn.oracle.com/goto/maa�

http://www.oracle.com/pls/db111/lookup?id=HAOVW�

http://www.oracle.com/pls/db112/lookup?id=HAOVW�

http://www.oracle.com/pls/db111/lookup?id=HABPT�

http://download.oracle.com/docs/cd/E16671_01/bh.200/e16644/toc.htm�


http://www.oracle.com/pls/db111/lookup?id=RACAD�

http://download.oracle.com/docs/cd/B16240_01/doc/em.102/b31949/oracle_site_management.htm#OEMCN9327�


http://www.oracle.com/pls/db111/lookup?id=BRADV�

Oracle White Paper Title: June 2010 Author: Ray Dutcher Contributing Authors: MAA team Editor: Viv Schupmann Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA 94065 U.S.A. Worldwide Inquiries: Phone: +1.650.506.7000 Fax: +1.650.506.7200 oracle.com

Copyright © 2010, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

0109