SOA Cloud Services Disaster Recovery · Oracle Service-Oriented Architecture Cloud Service (SOACS) is a Platform as a Service (PaaS) computing platform solution for running Oracle

SOA Cloud Service Disaster Recovery on OCI

Production and DR in the Cloud

O R A C L E W H I T E P A P E R | D E C E M B E R 2 0 1 9

SOA CLOUD SERVICE DISASTER RECOVERY ON OCI

Table of Contents

Introduction 1

Disaster Protection Considerations for Oracle SOA Cloud Service 3

Service Level Requirements 5

Security Requirements 5

Configuration Requirements 6

SOACS/MFTCS DR Deployment 8

Set up Details 10

1. Assign virtual hostname for the frontend OTD/LBaaS/OCILBS and update

frontend address in primary site 10

2. Update t3/RMI urls (if used) 12

3. Provision the SOACS System in secondary location pointing to snapshot

database 14

4. Assign virtual hostname for the frontend OTD/LBaaS/OCILBS in the

secondary site 17

5. Download and run the Disaster Recovery Setup (DRS) tool 17

6. OPTIONAL: Verify full switchover process 19

SOACS/MFTCS DR Lifecycle Procedures 20

Switchover 20

Failover 20

Applying domain configuration changes to the system 21

a) Repeating domain configuration changes in both sites 21

b) Using DBFS for configuration propagation 22

SOA CLOUD SERVICES DISASTER RECOVERY ON OCI

Scaling the SOACSDR system 27

Conclusion 28

Appendix A - Setting up Data Guard manually 29

Appendix B – Cloud backups in DB Systems 34

Appendix C – Using a Single Tenancy to create a DR system 36

Appendix D – Using Enterprise Manager Site Guard to manage the SOACS DR

switchovers 38

1 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI

Introduction

Oracle’s Maximum Availability Architecture (Oracle MAA) is the best practices blueprint for data

protection and availability of Oracle products (Database, Fusion Middleware, Applications) deployed on

on-premises, private, public or hybrid clouds. Implementing Oracle Maximum Availability Architecture

best practices is one of the key requirements for any Oracle deployment. Oracle Fusion Middleware and

Oracle Databases include an extensive set of high availability features, such as process death detection

and restart, clustering, server migration, clusterware integration, GridLink, load balancing, failover,

backup and recovery, rolling upgrades, and rolling configuration changes, which protects application

deployments from unplanned downtime and minimize planned downtime.

Oracle Service-Oriented Architecture Cloud Service (SOACS) is a Platform as a Service (PaaS)

computing platform solution for running Oracle SOA Suite in the cloud. Oracle Managed File Transfer

Cloud Service (MFTCS) is a high performance, standards-based, end-to-end managed file gateway.

These two computing platforms use Oracle Compute Cloud Service, Oracle Database Cloud Service

and Oracle Java Cloud Service as its basic infrastructure. Both require an Oracle Database to store

Oracle Platform Security Services information, instance tracking, composite and document metadata

and other Oracle FMW Infrastructure schemas. In a typical Oracle SOA deployment the application data

(such as application-specific schemas, jms stores etc.) and the SOA-specific schemas are stored in the

same database for transactional consistency and simplified administration reasons. In a SOACS/MFTCS

instance an Oracle Database Cloud Service instance is used to store these schemas. All Oracle SOA

deployments need protection from unforeseen disasters and natural calamities. This protection is also

required for systems deployed in the Cloud and it needs to address both the middle tier (Oracle SOA

Suite Cloud Service) and the data tier (Database Cloud Service). The solution involves setting up a

standby system at a geographically different Oracle cloud data center than the production site. The

standby system may have equal or fewer services and resources compared to the production site. The

standby system is normally in a passive mode and is activated when the primary site becomes

unavailable. This deployment model is sometimes referred to as an active-passive model. This model is

usually adopted when the two sites are connected over a WAN and network latency does not allow

clustering across the two sites.

Oracle SOA Cloud Service (SOACS) provides the best Recovery Time Objective (RTO) and Recovery

Point Objective (RPO) by utilizing high availability and disaster protection capabilities provided by Oracle

Fusion Middleware and Oracle Database. While there are some unique considerations to a cloud

disaster recovery (DR) configuration, it follows the same Oracle MAA best practices as any Oracle

Fusion Middleware (FMW) and Oracle Database deployment. This Oracle MAA blueprint details Oracle


MAA Best Practices and provides a procedural overview for deploying DR for SOA Cloud Service. Oracle

SOA Cloud Service Disaster Recovery solution is achieved by replicating a limited set of configuration

files that are required to bootstrap SOA components. The application may require additional

configuration files to be replicated. Options are provided in this paper to suit different application

paradigms. Disaster protection for Oracle Database Cloud Service used by Oracle SOA Cloud Service

is provided through Oracle Data Guard.

In this paper, Oracle Cloud Infrastructure is used for SOACS and Db Systems. The configuration screens

and setup steps provided in the different sections are based on the capabilities provided by this new

IaaS platform. Refer to the OCI Classic SOACSDR whitepaper for enabling disaster protection in older

IaaS versions.

This paper is intended for a technical audience having knowledge of Oracle WebLogic Server, Oracle

FMW SOA, Oracle Database, Data Guard and Oracle Database backup and recovery. This paper also

assumes a basic understanding of services offered on the Oracle Cloud1.

1 https://cloud.oracle.com/home

https://cloud.oracle.com/en_US/cloud-infrastructure

https://www.oracle.com/technetwork/database/availability/maa-soacs-dr-3125213.pdf

https://cloud.oracle.com/home


Disaster Protection Considerations for Oracle SOA Cloud Service

Oracle SOA Cloud Service uses DBCS (Oracle Database Cloud Service) to host the metadata and SOA instance

information. There are different database models and services available in Oracle Cloud. This paper precisely focus

and uses Database Systems VM on OCI for the examples and the configuration used. There are two different

Software Editions available for protecting Oracle Database Cloud Services against disasters:

Data Guard utilizing Enterprise Edition Service or High Performance Service.

Active Data Guard utilizing the Extreme Performance Service.

Oracle Active Data Guard allows, besides providing all the features included with Data Guard, the use of a standby

database for read operations. For more information on Data Guard and Active Data Guard please refer to the Data

Guard home page on the Oracle Technology Network and the Active Data Guard white paper2.

Oracle FMW SOA Suite can take advantage of the automatic block repair, fast incremental backups, fast rolling

upgrades, Far Sync, and most features of Active Data Guard. However, SOA servers cannot, however, use Oracle

Active Data Guard ‘s read-only query capabilities. This is because Oracle SOA Servers cannot run in read-only

mode. As soon as a SOA Server starts up, it connects to the database and tries to process business flows (i.e. they

“write” to the database right away). The only way to start the SOA servers in the standby site is by either 1)

performing a switchover/failover of the database or 2) by converting the standby database to a snapshot standby

which makes the standby read-writable temporarily for testing purposes. Redo received from the primary are stored

but not applied. The changes that are done are discarded at the end of testing where it can resume applying the

redo from the primary database. It needs to be noted that additional storage is required for the fast recovery area

when a standby is in snapshot mode (to hold archived redo received from the primary production database for later

use and current redo and flashback logs generated by the snapshot standby).

In a SOACS Disaster Recovery (DR) set up, a snapshot standby can be used to replicate configuration changes that

were applied in the production site when WLS domain configuration is not changed too frequently. The procedure

(which will be described later on in detail) consists in converting the standby database to a snapshot standby and

applying again the changes using the WLS Admin Console in the secondary location. This updates the file system

artifacts (ear files, deployment plans etc.) to be the same as in the primary site. It is also a best practice to

periodically place the standby in read/write mode (using Data Guard Snapshot Standby) to validate its readiness to

support read-write production workloads. The snapshot standby, however, needs to be used carefully with SOA

servers since undesired processing of pending instances may occur when the SOA servers have access to a

dehydration store. A snapshot standby may also be used for a final level of pre-production functional and

performance testing of patches and upgrades since the DR system is frequently sized similar to the production

system. Please refer to the Oracle documentation for additional details on Data Guard Snapshot Standby3.Beyond

this, the Oracle Cloud provides all backend infrastructure and capabilities required for disaster recovery should the

primary database become unavailable for any reason. This includes:

1. Ability to monitor the standby database and alert on major issues.

2. Ability to activate the standby to validate DR readiness and then convert back to a synchronized standby.

3. Utilization of essentially the same Oracle MAA best practices as on-premises deployments. This paper

describes the few considerations that are specific to cloud deployments.

2 http://www.oracle.com/technetwork/database/availability/active-data-guard-wp-12c-1896127.pdf 3 http://www.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/12.2/sbydb&id=GUID-B140C38B-DE01-4252-8422-7154018DDFEC

http://www.oracle.com/technetwork/database/options/active-data-guard/overview/index.html

http://www.oracle.com/technetwork/database/options/active-data-guard/overview/index.html

http://www.oracle.com/technetwork/database/availability/active-data-guard-wp-12c-1896127.pdf

http://www.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/12.2/sbydb&id=GUID-B140C38B-DE01-4252-8422-7154018DDFEC

http://www.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/12.2/sbydb&id=GUID-B140C38B-DE01-4252-8422-7154018DDFEC


4. Ability to switchover (planned event) or failover (unplanned event) production to the standby database in the

cloud during a planned maintenance or an unplanned outage. Once the failed primary database is repaired, it

can be resynchronized with the new production database in the cloud and then roles can be switched back to

the original status.

5. Ability to failover both the SOA/middle tier and database to enable production applications to run in the Oracle

Cloud when there is a complete site outage.

In the middle tier, the architecture of a SOACS/MFTCS DR system consists of two SOACS instances deployed in

the same sites as the Database Service instances. Each SOACS instance uses a SOA Cluster, an Administration

Server and OTD/LBaaS/OCILBS4 as front-end load balancer. A single Database (Database Cloud Service) instance

is used to host both the SOACS schemas (MDS, composite deployment, SOAINFRA schemas etc.) the JMS

persistent stores, the Transaction Logs persistent stores and any application specific data. Figure 1 describes this

architecture:

Figure 1: SOACS/MFTCS DR architecture

This topology can be created in different ways. Each approach has some implications. This paper describes the set

up process that has been considered to provide the greatest advantages from the Availability, Provisioning overhead

and Lifecycle perspective. Refer to the following sections for the recommend approach.

4 Although the examples and tests in this paper use Oracle Traffic Director (OTD) as front end load balancer, the same virtual server, certificates and

routing changes described in the document apply to Load Balancer Classic (LbaaS) and Oracle Cloud Infrastructure Load Balancing service (OCILBS)

PROVIDED SOACS supports that precise Web Tier component.


Service Level Requirements

Oracle SOA Cloud Service is a user-managed environment. The user must determine service level expectations for

availability, data protection, and performance that are practical for a given configuration and application. Service

Levels must be established for each of three dimensions relevant to disaster recovery that are applicable to any

Data Guard configuration:

Availability: Recovery Time Objective (RTO) describes the maximum acceptable downtime should an

outage occur. This includes time required to detect the outage and to failover the database, the Web tier

and SOA servers so that service is resumed.

Data Protection: Recovery Point Objective (RPO) describes the maximum amount of data loss that can

be tolerated. In SOA’s case this is especially related to transaction logs, JMS messages and SOA instance

information which all resides in the same database. The actual achievable RPO depends upon:

» Available network bandwidth.

» Network reliability.

» Data Guard transport method used: either asynchronous for near-zero data loss protection, or

synchronous for zero data loss protection.

Performance: Database and Middle Tier response time may be different after failover if less capacity –

compute, memory, I/O, etc., are provisioned at the standby system than in the primary system. This occurs

when users purposefully under-configure standby resources to reduce cost (accepting reduced service

level while in DR mode). MAA best practices recommend configuring symmetrical capacity at both primary

and standby in the web, application and database tiers so there is no change in response time after

failover. Rapid provisioning available with the cloud can enable a middle ground where less capacity is

initially deployed, but where the new primary is rapidly scaled-up should a failover be required.

Note: Independent of the service levels related to DR, all database instances created in the Oracle cloud conform to

the service descriptions defined by the applicable Database Cloud Service5.

Security Requirements

Oracle MAA best practice recommends using Oracle Transparent Data Encryption (TDE) to encrypt primary and

standby databases at rest. Conversion to TDE enables automatic encryption at rest for all DATA/INDEX tablespaces

and encryption-in-flight of user data redo changes during replication to the cloud. Oracle Net encryption is also

required for encryption-in-flight for other redo changes that are not encrypted by TDE (e.g. data from unencrypted

tablespaces such as SYSTEM and SYSAUX). When DBFS is used to maintain WLS domain configuration, using the

appropriate encryption at the DB level guarantees also the security of configuration propagation across sites.

5 http://www.oracle.com/us/corporate/contracts/paas-iaas-public-cloud-2140609.pdf


Note: Data Guard and Active Data Guard use redo-based replication – a process that transmits redo generated by a

primary database and applies those changes to a standby database using continuous media recovery. This means

that primary and standby databases are block for block identical copies of each other. Using TDE to encrypt a

standby database also requires that the primary database be encrypted with TDE.

Using TDE to protect data is an important part of improving the security of the system. Users should however be

aware of certain considerations when using any encryption solution, including:

Additional CPU overhead: Encryption requires additional CPU cycles to calculate encrypted and

decrypted values. TDE minimizes the overhead by taking advantage of database caching capabilities and

leveraging hardware acceleration for AES on Intel and SPARC CPUs. Most TDE users see little

performance impact on their production systems after enabling TDE. Please refer to the Oracle Database

Advanced Security Guide6 for more details.

Lower data compression: Encrypted data compresses poorly because it must reveal no information

about the original plain text data. Thus, any compression applied to TDE encrypted data will have low

compression ratios. Hence, redo transport compression should not be enabled when TDE encryption is

used. However, when TDE is used in conjunction with Oracle database compression technologies such as

Advanced Compression or Hybrid Columnar Compression, compression is performed before the

encryption occurs, and the benefits of compression and encryption are both achieved.

Key management: Encryption is only as strong as the key used to encrypt. Furthermore, the loss of the

encryption key is tantamount to losing all data protected by that key. If encryption is enabled on a few

databases, keeping track of the key and its lifecycle is relatively easy. As the number of encrypted

databases grows, managing keys becomes an increasingly difficult problem. For users with a large

number of encrypted databases, it is recommended that Oracle Key Vault7 on-premise be used to store

and manage TDE master keys.

Configuration Requirements

Beyond these runtime-related aspects, the following considerations are key to the SOACS/MFTCS DR set up:

The SOACS instance name in primary & standby should be the same. The instance name is used to

construct the hostnames, WLS server names and WLS domain name where SOACS will run. Using the

same instance name guarantees consistency and is required for the recovery of JMS messages and

Tlogs. It also simplifies customizations and operations in both sites. However, in Oracle SOA Cloud

Service, the same instance name cannot be used more than once in a single tenancy for the same service

type. i.e. it is not possible to use two different SOACS instances with the same name in the same account.

If the system is being created from scratch and there is flexibility in the name that will be used for the

primary SOACS instance, an instance name sharing the first 8 characters in the primary and standby

instances can be used to overcome this limitation. (SOACS WebLogic domain and WebLogic server

names are limited to eight characters. This limitation, however, does not exist for the SOACS Cloud

instance name). In this case the automated Data Guard configuration provided by DBCS and a single

tenancy can be used to configure a SOACS DR system- Refer to Appendix B for an example of the

instance names and approach that can be used to avoid the need of two different tenancies. In all other

6 http://www.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/12.2/asoag&id=GUID-81BF34FA-6044-47D4-BF31-12DEF178BA6B

7 http://www.oracle.com/us/products/database/security/key-vault/overview/index.html

http://www.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/12.2/asoag&id=GUID-81BF34FA-6044-47D4-BF31-12DEF178BA6B


http://www.oracle.com/technetwork/database/options/compression/overview/index.html

http://www.oracle.com/us/products/database/security/key-vault/overview/index.html



cases (i.e. when the primary system is not using an instance name of 8 or more than 8 characters) the

following will apply TO CONFIGURE SOACS/MFTCS DR:

1. TWO DIFFERENT TENANCIES NEED TO BE USED

2. THE AUTOMATED DATAGUARD CONFIGURATION OPTION PROVIDED BY DBCS CAN

NOT BE USED

A database instance is created automatically when you create the Oracle Database Cloud Service that

SOACS needs. On the standby site, the database initially created cannot be used as a Data Guard

standby database. This database will be deleted during the DR set up process.

MDS, Composite deployments and policies are automatically synchronized across sites by Data Guard

since they are stored in the database.

Most of the WLS domain configuration that SOACS uses is synced initially across sites using DBFS with

the following considerations:

1. Each SOACS system will maintain the original JDBC URLs used to connect to their local DB

even after the DR set up has completed. Only the schema prefix will be altered so that both

locations point to the same schemas

2. All the configuration under weblogc_domain_name/config is automatically distributed by the WLS

Infrastructure to the other nodes in the same site by the WLS infrastructure.

3. Custom application deployments (workflow task ears, custom ear files, deployment plan updates,

JMS resources, redeployments) and everything residing under the Administration Server WLS

domain directory (except temp data) is synced initially across sites. Data residing in other node’s

or outside the domain directory for the WebLogic Administration Server, Will have to be manually

copied to the secondary location.

The SOACS/MFTCS DR configuration makes failover/switchover transparent to end users or systems

accessing the services by making SOA endpoints agnostic to the site that is being used. For this to

be possible, the front end address of the existing SOA/MFT Cluster is altered to use a virtual hostname

that can be assigned to either DR location’s front end load balancer once the configuration is completed.

Appropriate DNS services (Oracle Cloud DNS, commercial DNS, local DNS server or manual file

hostname resolution) need to be used for the front end url to be mapped to either site. If any composites

are deployed/used before this front end address change is performed, it may be required to alter the

endpoint address to match this new virtual hostname (NOTE: By default end point addresses are

constructed based on the SOA/WLS cluster’s HTTP front end parameter).

It is possible to use the existing system’s front-end host name address (if such exists already) as the

virtual host address for disaster protection. i.e. if the original system was using mysoacs.example.com as

the vanity url for primary, this same virtual hostname can be mapped to the secondary system after a

switchover.

The initial configuration in the OTD/LbaaS/ OCILBS tier is sufficient to sustain the default routing, virtual

server and pools needed for the SOACS service. Custom configuration in OTD/LbaaS/ OCILBS is not

replicated across sites and any load balancer configuration change applied in one site, needs to be

repeated in the other-


Once Data Guard is configured for the databases used by the primary and secondary SOACS systems,

converting the “physical standby” to “snapshot standby” should be done only without starting the SOA

servers in the standby site or after “draining/truncating” the SOA database. This is required to eliminate

pending messages that could be re-executed.8

The primary and secondary databases need to communicate with each other over their listener port for

redo transport. Middle tiers need to communicate with the remote database for the initial set up also. This

communication can happen over an Internet Gateway (Oracle Net’s traffic is encrypted). Alternatively, if

the PaaS service supports it, private networks and Dynamic Routing Gateways between primary and

standby can be used and is the recommended approach. Depending on the type of access to the backend

database the appropriate ingress rules need to be enabled to allow database-to-database communication.

SOACS/MFTCS DR Deployment

In this document we assume, as a starting point, that the primary site, consisting of a single instance DB System,

SOACS Cluster and Oracle Traffic Director (OTD) is live. A secondary DR configuration, residing in a geographically

remote site, will be created for the existing primary system. Refer to the SOACS Documentation for details about

how to provision the initial set up. If the primary system has not been created yet, refer to Appendix B for details on

how to use an instance names that will allow creating the DR configuration under a single tenancy. If the primary

system exists already and is using a SOACS instance name longer than 8 chars, Appendix B provides also

indications to create the standby under that same tenancy. If the above do not apply (i.e. the primary system exists

and is not using a service name longer than 8 chars) follow these steps.

Since the primary SOACS may already be running production workloads, the DR configuration process is designed

to cause minimum downtime (in fact, only the modification of the front end address requires SOA server restarts). .

These are the summary steps for the set up process:

1. Assign a resolvable virtual hostname as front end for the primary system (or alternatively reuse any

friendly/vanity url or DNS name already used by the existing SOACS system)

2. Update front end address in the primary SOACS cluster

3. Update t3/rmi urls (if used)

4. Provision DB System in secondary location

5. Set up and verify DataGuard

6. Convert the physical standby database to snapshot standby

7. Provision SOACS in secondary location pointing to snapshot DB in a different tenancy

8. Run Disaster Recovery Setup tool (DRS) to configure the SOACS DR.

9. Verify SOACS in secondary site

10. Convert snapshot database to physical standby

11. Optionally, switchover to standby to verify a complete DR scenario

8 SOA servers will try to process SOA instances and complete pending work (callbacks, async invocation etc.) that may have been processed already

on the primary site. If the snapshot database has not been drained, only the administration server should be brought up in the standby domain for

systems already in production.

http://www.oracle.com/pls/topic/lookup?ctx=en/cloud/paas/soa-cloud&id=GUID-1095E09E-147C-4AC6-B094-918174D0CDA6


The following sections describe these steps in detail.


Set up Details

Before creating the SOACS system in standby, it is recommended that the middle tier and the database in the

primary system contain the latest patches and PSUs. More precisely a DB System Data Guard configuration across

different DB domains requires a fix for bug 22611167.Make sure the pertaining patch for your precise database

version is applied in the system. Additionally, make sure that the SOACS version and patch level to be provisioned

in the secondary location matches the one running in the primary site. Once the DB System and the SOACS

instance in the primary location are patched and functioning, follow these steps:

1. Assign virtual hostname for the frontend OTD/LBaaS/OCILBS and update frontend address in

primary site

By default SOACS’s provisioning sets the value of the SOA cluster’s front end address to the IP of

OTD’s/LBaaS/OCILBS listen address. This IP needs to be replaced with a virtual hostname and this hostname

needs to be resolvable by both the external clients and by the WLS servers that OTD/LBaaS/OCILBS is routing

requests to. To resolve the virtual hostname externally, local hosts file or any formal DNS resolution may be

used. To resolve the virtual hostname in the scope of the SOACS instance, either local hosts file resolution or

Oracle DNS Cloud Service can be used. In this last case the resolv.conf files for the SOA nodes needs to be

updated with the new DNS server hosting the record for the virtual hostname.

A. To determine the IP matching the virtual hostname for the SOACS Cluster, follow these steps:

Login to the WebLogic Server Administration Console for your SOACS instance.

In the left pane, choose Environment in the Domain Structure window and then choose

Clusters. The Summary of Clusters page appears. Select the SOA_Cluster cluster.

Select HTTP

Note down the IP used in the Frontend Host filed.

NOTE: Alternatively you can get this IP also from the SOACS instance screen information.

This IP would be listed as the PUBLIC IP for the Load Balancer:


B. Add the IP that maps to the virtual hostname to the DNS resolution in the nodes. In a command

shell prompt for your SOACS nodes, sudo to the root user and edit /etc/hosts to map this IP to a

virtual hostname (if a DNS is going to be used for the external resolution of this IP, this hostname

will have to be used here. If you are going to use file-based (/etc/hosts) hostname resolutions,

this would be your own choice). For example:

[oracle@soacsdroci-wls-1 ~]$ cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

10.0.0.11 drDb2a.sub10171440110.soacsvcnash.oraclevcn.com drDb2a

129.213.81.253 soacsdroci.paasmaaoracle.com soacsdroci

Repeat this step in ALL the other SOACS nodes that are hosting members of the

SOACS Cluster

To make the virtual hostname available to the external clients either update your DNS

server or modify the hosts file for those clients as above. For example in your on-

premises windows client, edit your C:\Windows\System32\drivers\etc\hosts file as

above

NOTE: This configuration for external clients will work if direct connections from the internet are used. Connections

from custom intranets will need to account for this hostname to be added to the required proxy server used by the

browsers or http clients

As a plausible alternative, Oracle DNS Zone management can be used to maintain the front end

hostname mapping. In this example, the front end address hostname would be a record more in the

appropriate zone. Client would need to update their /etc/resolv.conf nameserver entry to use the

appropriate DNS server.

C. Once the appropriate hostname resolution is in place per the steps above, modify the front end

address in the SOACS Cluster:

Login to the WebLogic Server Administration Console for your SOACS instance.


In the left pane, choose Environment in the Domain Structure window and then choose

Clusters. The Summary of Clusters page appears. Select the SOA_Cluster cluster.

Select HTTP

Set the value for the Frontend Host=virtual_hostname_created, in the example above

this would be soacsdroci.paasmaaoracle.com

Click Save.

To activate the changes, click Activate Changes in the Change Center section of the

Administration Console.

Restart the SOACS cluster using the WebLogic Administration Console for the front

end change to be effective.

2. Update t3/RMI urls (if used)

The urls used for RMI invocations in the SOACS cluster need to be agnostic to the IPs or hostnames used by each

site in the SOACS/MFTCS DR configuration. Instead of using the typical host:port,host:port JNDI urls, change them

to use the cluster syntax9. The cluster syntax is as follows: cluster:t3://cluster_name. For example, to modify the

JMS adapter factory properties to use this syntax, follow these steps:

A. Log into your Oracle WebLogic Server Administration Console for your SOACS instance

B. Click Deployments in the left pane for Domain Structure.

C. Click JmsAdapter under Summary of Deployments on the right pane.

D. Click the Configuration tab.

E. Click the Outbound Connection Pools tab and expand

oracle.tip.adapter.jms.IJmsConnectionFactory to see the configured connection factories.

F. Click the specific instance you are using (for example, eis/wls/Queue). The Outbound

Connection Properties for the connection factory opens.

G. Click Lock & Edit.

H. In the FactoryProperties field (click on the corresponding cell under Property value), alter the

java.naming.provider.url filed to use the cluster syntax (leave the rest of the fields as they were):

java.naming.provider.url= cluster:t3://cluster_name;

I. Click Save after you update the properties. The Save Deployment Plan page appears.

J. Enter a location for the deployment plan.

K. Copy the deployment plan from your SOACS node1 to your SOACS node2 in the exact same

directory/location or use the default DBFS mount point present in SOACS system as the location

to host these deployment plans (all nodes in the SOACS cluster can access

/u01/soacs/dbfs/share)

L. Click Save and Activate.

9 Obviously this is feasible only for inter-domain invocations. T3/rmi clients that are external to the SOACS domain will not be able to use this approach

and will have to either use a tcp load balancer for JNDi resolution or use the appropriate DNS mapping of the host:port list in the secondary site


M. Click Deployments.

N. Click Lock & Edit

O. Select the JMS Adapter.

P. Click Update.

Q. Select Update this application in place with new deployment plan changes (A deployment plan

must be specified for this option.) and select the deployment plan saved in a shared storage

location; all servers in the cluster must be able to access the plan.

R. Click Finish.

S. Activate the changes.

Similarly, any other custom JNDI urls used in the SOACS system should also be updated so that when a

switchover/failover occurs in the SOACSDR system, the urls are valid also in the secondary site.


3. Provision the SOACS System in secondary location pointing to snapshot database

The secondary SOACS system will be provisioned using the normal procedure. However, as explained in

previous sections, it needs to use the same instance name as primary, hence it cannot be created in the same

tenancy (unless the primary system is also created from scratch with flexibility in the instance name that it will

use. In this case refer to Appendix B) The following are key considerations for this set up:

1. The SOACS Service is created pointing to a database that will be configured as the Data Guard

standby of the database that is being used for the primary system.In order to use the standby

database as the container for the secondary SOACS provisioning, this physical standby needs to be

converted to a snapshot standby.

2. Oracle SOACS provisions SOA schemas using a prefix that is specific to each cloud

services/instance. This means that in the initial provisioning the secondary location SOACS servers

will point to the same Database but will use different schemas. This is critical for systems that are

already running because this will prevent the execution of composites/flows by the initial SOACS

domain in the secondary location. It is needed that only one site has active SOA servers pointing to

an available database at any point in time. Otherwise message and callback duplications could occur

leading the SOA system to inconsistencies.

Follow these steps to provision the secondary SOACS system:

IMPORTANT: Once the secondary location JDBC strings are updated to point to the same schemas as

production, the SOA servers in the secondary location will see the same data that the production ones were

seeing when the snapshot conversion occurred. If any SOA flows, callbacks etc. are pending, the servers in the

secondary location will try to complete those. Thus, it is important that instances are drained and completed on

the primary site before converting the standby database to snapshot or duplications could occur. Alternatively

the SOA servers in the primary location may be stopped and the database can be switched over to the

secondary location (incurs in higher downtime).

A. Provision a DB System instance in the secondary site

Make sure that the Database Name, Release, Patch level and PDB Names used in the secondary DB instance are

the same ones that were used when creating the primary one. This may require patching the primary system

(especially if it has been running for a long time) before creating the standby. Oracle recommends using the same

Compute Shape and Storage Size that were used for primary also. Follow the steps in the SOA CS documentation

to provision the required Database for the standby datacenter. The following table provides examples and

requirements for the properties that need to be used in the standby DB System creation process

NOTE: the SOACS DR Setup procedure described in this paper has not been certified with RAC database

Cloud Account

Configuration Property

Existing Primary DB

System/example

Secondary Db System being

created/example

Oracle Cloud Tenancy XXXX/paasmaa YYYY/paasmaa2 (should be different

from primary)

Oracle Cloud Account User XXXX/[email protected] YYYY/[email protected] (user name can

be different, tenancy must be different)

mailto:[email protected]


Oracle Cloud Account Password XXXX/Acme1234# XXXX/ Acme1234# (can be different

from primary)

DB System Configuration

Property

Existing Primary DB

System/example

Secondary Db System being

created/example

Display Name XXXX/soacsdrDBa YYYY/soacsdrDBb (can be different

from primary)

Availability Domain XXXX/efEXT:US-ASBURN-AD1 YYYY/ efXT:PHX-AD-3 (should be

different from primary)/

Shape Type VM, BM or Exa10 VM, BM or Exa10

Shape XXXX/VM.Standard2.1 XXXX VM.Standard2.1 (same as

primary)

Total Node Count N10/1 N10/1(same as primary)

DB System Software Edition Enterprise Edition High Performance or

Enterprise Edition Extreme Performance

/Enterprise Edition Extreme Performance

Enterprise Edition Extreme

Performance (same as primary)

Available Storage Size XXXX/256 XXXX/256 (same as primary)

License Type LI, BYOL/BYOL LI, BYOL/BYOL(can be different from

primary)

SSH Public Key XXXX YYYY (can be different from primary)

Virtual Cloud Network XXXX/soacsdrASH YYYY/soacsdrPH (systems are

expected to be remote, hence different

from primary. Needs to include a public

Internet Gateway)11

Client Subnet XXXX/Public Subnet efXT:US-ASBURN-

AD1

YYYY/ Public Subnet efXT:PHX-AD3

(systems are expected to be remote,

hence different from primary)

Hostname Prefix XXXX/soacsdrASH YYYY/soacsdrPH (can be different from

primary)

Database Name XXXX/ORCL XXXX/ORCL(same as primary)

Database Version XXXX10/12.2.0.1 XXXX/12.2.0.1 (same as primary)

Database Admin Password XXXX/Acme1234# XXXX/Acme1234# (same as primary)

Automatic Backup X/Checked X/Unchecked (disable backup in

standby)

NOTE: The default database instance created on the secondary site will be deleted lateras it cannot be used as

a Data Guard standby database. It is created with the same name as primary to get the required lifecycle

scripts seeded in the system with the same configuration as the primary DB

Make sure to apply the required patches to the DB in both locations (primary and secondary) so that to both are at

the same patch level. More precisely a Data Guard configuration across different DB domains requires a fix 10 Subjected to SOACS supportability of the shape type in non DR configuration

11 Remote peering can also be used for DG traffic across regions


for bug 22611167. Make sure the pertaining patch for your precise database version is applied in both the primary

and standby systems.

B. Configure Oracle Data Guard manually between the primary and the secondary database as

described in Appendix A (services using the exact same SOACS instance name in primary and

standby, the automated Data Guard Configuration provided in Cloud cannot be used).

C. Convert the physical standby database to snapshot standby. As oracle user in the primary

Database node, execute the following (replace your_sys_password and

secondary_db_unqname with the appropriate values for your case)

[oracle@soacsdrDBa ~]$ dgmgrl sys/your_sys_password@primary_db_unqname

DGMGRL> CONVERT DATABASE secondary_db_unqname to SNAPSHOT STANDBY;

Converting database "secondary_db_unqname " to a Snapshot Standby database, please

wait...

Database "secondary_db_unqname" converted successfully

D. Provision the secondary SOACS instance using the SOACS provisioning wizard:

Follow the steps in the SOACS documentation to create the secondary site SOACS system pointing to the

secondary DB System converted to snapshot in the previous step. Use the EXACT same name for the SOACS

service that you are using in your primary location. The following table summarizes the provisioning wizard

options for the set up:

Cloud Account

Configuration Property

Existing Primary SOACS

System/example

Secondary SOACS System

being created/example

Oracle Cloud Tenancy XXXX/paasmaa YYYY/paasmaa2 (should be different

from primary)

Oracle Cloud Account User XXXX/[email protected] YYYY/[email protected] (user name can

be different, tenancy must be different)

Oracle Cloud Account Password XXXX/Acme1234# XXXX/ Acme1234# (can be different

from primary)

SOACS Configuration Existing Primary SOACS

System/example

Secondary SOACS System being

created/example

SOACS Service Name XXXX/soacsdroci XXXX/soacsdroci (same as in primary)

Region XXXX/us-ashburn-1 YYYY/us-phoenix-1 (systems are expected

to be remote, hence different site from

primary

Availability Domain XXXX/efXT-ASHBURN-AD-1 YYYY/ efXT:PHX-AD-3 (should be different

from primary and the same as the

secondary Db System)

Subnet XXXX/Public Subnet efXT:US-ASBURN-

AD1

YYYY/ Public Subnet efXT:PHX-AD3

(systems are expected to be remote, hence

different from primary)



primary)

http://www.oracle.com/pls/topic/lookup?ctx=en/cloud/paas/soa-cloud&id=GUID-1095E09E-147C-4AC6-B094-918174D0CDA6

mailto:[email protected]


Software Release XXXX/12.2.1.3 XXXX/12.2.1.3(same as primary)

Service Type SOA with SB & B2B Cluster or

MFT/SOA with SB & B2B Cluster

SOA with SB & B2B Cluster(same as

primary)

Compute Shape XXXX/VM.Standard2.1 XXXX VM.Standard2.1 (same as primary)

Cluster Size N/2 N/2(same as primary)

WebLogic User Name XXXX/weblogic XXXX/weblogic(same as primary)

WebLogic Password XXXX/Acme1234# XXXX/Acme1234#(same as primary)

Database Type Oracle Cloud Infrastructure Oracle Cloud Infrastructure

Database Compartment Name XXXX/soacsdr XXXX/soacsdrB (can be different from

primary)


Database PDB Name XXXX/PDB1 XXXX/PDB1(same as primary)

Database Administrator User name XXXX/SYS XXXX/SYS(same as primary)

Database Admin Password XXXX/Acme1234# XXXX/Acme1234# (same as primary)

Load Balancer Provisioning Yes Yes

Load Balancer Policy Least Connection Count Least Connection Count

Load Balancer Compute Shape XXXX/VM.Standard2.1 XXXX VM.Standard2.1 (same as primary)

Backup and recovery Configuration XXXX YYYY ( different from primary)

Oracle recommends that the exact same capacity and compute configuration is used on both primary and standby

locations for the ideal failover/switchover behavior (otherwise, the required request-throttling in OTD/LBaaS/OCILBS

and sizing of SOACS nodes needs to be done on the secondary location). Once the provisioning process completes

the SOA servers can be sanity verified.

4. Assign virtual hostname for the frontend OTD/LBaaS/OCILBS in the secondary site

Follow the same steps as in section “Assign virtual hostname for frontend OTD/LBaaS/OCILBS and update frontend

address in primary site” above using the public IP of the OTD/LBaaS/OCILBS instance in the secondary site to

map the virtual host. You don’t need to update the front end address for the SOACS cluster as that information will

be copied from the primary WebLogic Domain configuration. Only the update of the /etc/hosts or DNS entry is

required in standby

NOTE: Despite the hostname alias being the same in both sites, the required trust stores/certificates will have to be

recreated in the standby locations and the standby’s OTD/LBaaS/OCILBS certificate will have to be imported in the

appropriate trust store if any SSL invocations are executed from the Cloud servers to the front end LBR. Refer to the

Enterprise Deployment Guide for Oracle SOA Suite for the required steps

5. Download and run the Disaster Recovery Setup (DRS) tool

The Disaster Recovery Setup tool (DRS) is a framework that orchestrates and runs the configuration steps for the

SOACS disaster recovery setup. The tool can be run from any host (with operating system OEL 7) that has ssh

connectivity to the SOACS and DB hosts where the DR setup is being done.

Steps to run the DRS tool:

Choose a host to run the DRS tool.

http://www.oracle.com/pls/topic/lookup?ctx=en/middleware/lifecycle/12.2.1.3/soedg&id=GUID-7499718B-A1A1-4403-8051-1995885B3B5C


TIP: create a compute instance (OEL 7) in your cloud tenancy to run the tool. This compute instance can

be removed later, once the DR configuration is done and DRS is not needed anymore.

Download the DRS tool from here and upload it to the host where the tool will run.

Extract the contents of this file using the command ‘tar -xzf drs.tar.gz’ and navigate to the ‘drs’ directory it

creates.

Open README.md and follow instructions. Please make sure you review IN DETAIL the steps and

recommendations in DRS's README.md file. It is critical that you check the config file required to run it

and meet the requirements in it for the appropriate behavior in the set up.

The DRS tool will automatically perform the required steps to configure secondary SOACS as standby SOACS DR

site. These steps include:

Initial checks to verify that the environment is prepared for the DR setup.

A backup of the secondary domain configuration before the DR setup (i.e. /u01/data/domains/

soacsdrs_domain_backup_09:29:05-16-12-19)

Required host aliases configuration in the /etc/hosts files, in primary and secondary SOA servers.

DBFS keystore recreation for the SOA DBFS mounts, and TNS alias configuration to remote database (will

be used later, for future domain configuration sync).

Primary domain configuration copy to secondary site: it copies the domain config from primary to the dbfs

mount, and then from the dbfs mount to domain folder in secondary hosts.

Verification that the secondary domain is correctly configured for DR by starting the secondary managed

servers in a rolling manner after the DR configuration, using the database in snapshot mode.

During the process, the tool performs some database role conversions in the secondary database (conversion to

snapshot standby and back to physical standby). Once it finishes, it leaves the secondary database in physical

standby role and the secondary admin and managed servers stopped.

IMPORTANT: Up to this point, the SOA servers in the secondary location have been pointing to “empty” SOAINFRA

schemas with no composites deployed, no policies and no flows pending of execution. Once the secondary location

JDBC strings have been updated to point to the same schemas as production per the above steps, the SOA servers

in the secondary location will see the same data that the production ones were seeing. If any flows, callbacks, etc.

were pending to be executed; the secondary location servers will try to complete those at this point if started. Thus, it

is important that instances are drained and completed on the primary site before converting to snapshot the standby

database as already indicated above.

https://www.oracle.com/a/tech/docs/drs.tar.gz


6. OPTIONAL: Verify full switchover process

The system should now be ready for a switchover. It is a best practice to perform a complete and full

switchover test (if primary is already “production” a maintenance window should be scheduled) to confirm

the proper behavior of database and servers. To perform an initial switchover test, follow these steps

1. Switchover Database

Execute these steps as oracle user in the primary Database host:


DGMGRL> switchover to secondary_db_unqname

Performing switchover NOW, please wait...

Operation requires a connection to instance "ORCL" on database "primary_db_unqname"

Connecting to instance "ORCL"...

Connected as SYSDBA.

New primary database "secondary_db_unqname" is opening...

Oracle Clusterware is restarting database "orcl" ...

Switchover succeeded, new primary is "secondary_db_unqname "

2. Verify SOACS in secondary site

At this point the SOA servers in both sites are pointing to the same schemas and the database is open in the

secondary location. Start the servers and verify composite deployments and composite instances (if any) in the new

active site

a. Start the Administration and Managed servers in the secondary site

a. Logon to the Enterprise Manager FMW Control Console for the secondary SOACS instance and

verify the soa-infra systems in both servers in the cluster. It is a good practice to perform an

endpoint test to confirm the correct behavior of the system

3. Switchback to original site.

To revert the system back to its original state, switchback the database and restart SOA servers in the primary

location.

http://www.oracle.com/pls/topic/lookup?ctx=en/cloud/paas/java-cloud/jscug&id=GUID-BD5190E0-5DE4-454E-BAD3-E889C575D432

http://www.oracle.com/webfolder/technetwork/tutorials/obe/cloud/sscs/DeployingComposite_Cloud/soa-comp-cloud-deployment.html


SOACS/MFTCS DR Lifecycle Procedures

Switchover

In a switchover operation an administrator reverts the roles of the two sites in a planned operation. The roles change

from the primary to the standby as well as from standby to primary. Oracle Site Guard can be used to automate the

task required in a switchover and reduce the RTO in such operation. Refer to Appendix C for details. To perform a

manual switchover in a SOACS/MFTCS DR configuration follow these steps:

a. Stop the managed servers in the SOACS primary site.

Use the SOACS documentation to stop the Administration and Managed servers in the primary site

b. If configuration changes have been applied recently to the primary SOA/WebLogic domain,

make sure to propagate those to the standby location (Refer to section “Applying domain

configuration changes to the system” bellow)

c. Perform the required DNS changes to point consumers to the new primary OTD

Perform the required DNS push in the DNS server hosting the names used by the system or alter the file host

resolution to point the front end address of the system to the public IP used by OTD/LBaaS/OCILBS in site2

d. Perform a database switchover using Data Guard Broker

From the primary site’s DB System node and as oracle user:


DGMGRL> switchover to secondary_db_unqname

Performing switchover NOW, please wait...

Operation requires a connection to instance "ORCL" on database "secondary_db_unqname"

Connecting to instance "ORCL"...

Connected as SYSDBA.

New primary database "secondary_db_unqname" is opening...

Oracle Clusterware is restarting database "primary_db_unqname " ...

Switchover succeeded, new primary is "secondary_db_unqname"

e. Start Managed servers in the new SOACS primary site

Use the SOACS documentation to start the Managed servers in the secondary (now new primary) site. (the

Administration server and Node Manager can be kept up on standby)

Failover

In a failover operation, the primary site becomes unavailable and an administrator fails over the Database and starts

managed servers in the secondary site. You can role-transition a standby database to a primary database when the

original primary database fails and there is no possibility of recovering the primary database in a timely manner. This

is known as a manual failover. There may or may not be data loss depending upon whether your primary and target

standby databases were transactionally consistent at the time of the primary database failure. Oracle Site Guard can

be used to automate the task required in a failover and reduce the RTO in such operation. Refer to Appendix C for

details to perform a switchover in a SOACS/MFTCS DR configuration follow these steps:

a. Perform the required DNS changes to point consumers to the new primary OTD




Perform the required DNS push in the DNS server hosting the names used by the system or alter the file host

resolution to point the front end address of the system to the public IP used by OTD/LBaaS/OCILBS in Site2

b. Perform a database failover using Data Guard Broker. Start Data Guard broker on the secondary

DB System node. As oracle user execute these steps:

[oracle@soacsdrDBb ~]$ dgmgrl sys/your_sys_password@primary_db_unqname

DGMGRL> failover to secondary_db_unqname

Performing failover NOW, please wait...

Failover succeeded, new primary is "secondary_db_unqname"

c. Start Managed servers in the new SOACS primary site

Use the SOACS documentation to start the Managed servers in the secondary (now new primary) site

Applying domain configuration changes to the system

The fact that direct file system synchronization across Cloud datacenter virtual machines is not allowed, precludes

WLS domain config changes from being automatically propagated to the secondary site (as it would occur in

standard on-premise Active Passive DR deployments). Two main approaches can be used to maintain the same

configuration (ear deployments, WLS domain configuration, deployment plans etc.) in both locations. The

applicability of each depends on how frequently this “file-system-resident” configuration is modified.

a) For those SOACS cases where the domain configuration is infrequently altered (notice that composite

deployments, domain and WSM policies and MDS updates do not fall into this category as they are

stored in the Database) it is recommended to simply apply the configuration changes manually twice,

once in production and once in standby.

b) For those SOACS cases where the file system configuration is modified regularly, Oracle Database

File System (DBFS) can be used to synchronize the configuration using Data Guard (DBFS provides

a file system view of data that is stored in the Database). Using DBFS for configuration replication has

implications from the setup, disk space allocation and lifecycle perspective and oracle recommends

using it when it is necessary to replicate configuration changes frequently. There are other

alternatives to DBFS such as direct use of rsync across sites, but those present other risks including

lack of transactional control in the copy and possible corruption of the domain structure in the event of

a failure during the copy operations.

Both approaches are described below.

a) Repeating domain configuration changes in both sites

To maintain the file system configuration synchronized by repeating the config change in both sites, follow these

steps:

A. Apply the configuration change normally in the primary site

Use the WLS Administration Console in the primary location to apply the configuration change. Activate the

change, restart the required SOACS servers if needed and verify that the change is working as expected.

B. Convert the standby database to a snapshot standby

Execute these steps as oracle user in the primary Database host:

[oracle@soacsdrDBa ~]$dgmgrl sys/your_sys_password@primary_db_unqname




Converting database " secondary_db_unqname" to a Snapshot Standby database, please

wait...

Database " secondary_db_unqname" converted successfully

C. Start (if it wasn’t started) the Administration Server on the secondary site

Follow the steps in the Oracle Cloud documentation to start the administration server. It is important that ONLY

the administration server and not the managed servers is started on the secondary location13.

D. Repeat the configuration change in the secondary site

Use the WLS Administration Console in the primary location to apply the configuration change. Activate the

change and verify that the change is working as expected. This change should not alter any of the

configurations in the database, only WLS domain configuration or application deployments. Any modifications

in the database will be overwritten by the primary when the Db is reverted to physical standby in the next step.

E. Revert the database to physical standby Execute this steps as oracle user in the primary Database host:


DGMGRL> CONVERT DATABASE secondary_db_unqname to PHYSICAL STANDBY;

Converting database " secondary_db_unqname" to a Physical Standby database, please

wait...

Oracle Clusterware is restarting database "orclb" ...

Continuing to convert database " secondary_db_unqname" ...


b) Using DBFS for configuration propagation

Database File System (DBFS) uses database features to store files and manage relational data and expose them as

a standard file system that can be accessed by any operating system. The Oracle Database File System (DBFS)

creates a standard file system interface on top of files and directories that are stored in database tables. DBFS is

similar to NFS in that it provides a shared network file system that looks like a local file system. Like NFS, both a

server component and a client component are required to run DBFS.

Given the restrictions in file replication across Oracle Cloud data centers, DBFS can be used with Oracle Data

Guard to copy files from a primary site to a secondary site. When the system’s lifecycle involves frequents updates

to the domain file system, DBFS can be used to replicate the WLS domain configuration across sites on a regular

basis. However, the SOACS domain Configuration cannot reside directly on the DBFS mount because that would

make the middle tier dependent on the DBFS infrastructure in order to come up (the dependency is not only on the

database but also on FUSE libraries, mount points etc.). Notice also that the SOACS WebLogic domain

configuration cannot be copied “as is” in this paper’s design since each site in the SOACS/MFTCS DR solution

contains references to the Cloud identity domain and the local DB service in the JDBC connect strings. The

configuration has to be modified after it is copied to each site. In this approach, a directory with a copy of the primary

site’s domain configuration is kept up to date with Data Guard in the standby location. This directory is not available

to the standby site unless the DB is open in read-only mode (when Active Data Guard is used), a conversion to a

snapshot standby is performed or a switchover or failover is executed. The conversion to snapshot will allow

mounting the DBFS file system for writes also, but has the caveat that it will not preclude SOA servers in the

13 Changes to a reduced number of configuration artifacts in SOA and OSB may require the servers to be up in order to be applied, in these cases a

start of the managed servers will be needed. Refer to the specific product documentation to identify these artifacts. In this case and if there are pending

messages in the database those could be re-executed in the standby location. In such scenarios, Oracle recommend draining/truncating the SOA

database schemas in the snapshot database following the SOA standard procedures BEFORE starting the SOA WLS servers


secondary location from starting (either accidentally, or by a reboot) and may cause duplications and re-executions

of work already processed by the primary Location. Oracle recommends, however that a complete start and test of

the servers in the secondary location is performed on a regular basis to verify that the configuration and

deployments are being replicated properly and that the Cloud identity Domain and JDBC connect strings are being

replaced correctly. This can be done by doing a switchover or by converting the secondary database to snapshot

standby. In this last case, Oracle recommends to “drain” first the Primary Database from any SOA work pending

(this includes pending async invocations and JMS messages) to avoid duplicated executions. Draining may imply

blocking new incoming requests and waiting for all pending messages to be processed/completed. The following

diagram describes the model used to replicate WebLogic Domain configuration changes from the primary to the

secondary location.

For application deployment operations, Oracle recommended using the WebLogic deployment “Upload your files”

option in the WebLogic Administration Console so that the deployed files are placed under the upload directory of

the Administration Server (under domain directory/servers/admin_server_name/upload). That way these files will be

synced to standby by the DBFS copy scripts

In summary the use of DBFS for domain configuration synchronization requires the following steps:

1. Verify the DBFS configuration already created by SOACS instance and the appropriate mount points for

DR.

2. Create domain copy and configuration replacement scripts.

3. Enable domain copy (either by manual execution or by configuring cron jobs)14

The following subsections describe these steps in detail:

14 Changes applied to files in different nodes of the SOACS Cluster that do not reside under the domain/config directory, need to be manually synced to

those nodes. The DBFS approach described in this paper replicate the domain configuration to the Administration Server node only.


Use the DBFS configuration already created by the SOACS instance and verify appropriate mount points for DR

DBFS is already configured out of the box for SOACS systems. A client command-line interface named

dbfs_client runs on each SOACS computer with pre-configured access to the DBFS mount points. dbfs_client

allows users to copy files in and out of the database from any host on the network. DBFS is a part of Oracle

database installation and requires fuse (FileSytem in Userspace) in the nodes that access the mount point. It

implements simple file system commands such as list and copy in a manner that is similar to shell utilities.

dbfs_client and fuse are already installed in SOACS starting 12.2.1.2 VMs. The DBFS mount points and client

configuration created by default in SOACS VMs are used for MFT, B2B and File Adapter cases (to share a common

mount point between the members in the SOA cluster). For performance and administration-isolation purposes the

dbfs mount point used for replicating the domain across sites is configured and mounted separately from the default

DBFS mount points under /u01/soacs). This guarantees that configuration operations will not interfere with the

system’s runtime behavior (for example, thanks to this separation, it will be possible to unmount the DBFS

configuration directory without affecting the processing of files by the file adapter in a cluster). There will be a

“configuration” DBFS mount point under /u01/soacs/dbfs (configured in the steps bellow) and a “runtime” DBFS

mount point under /u01/soacs/dbfs_directio. Both are automatically created/configured with the SOACS instance.

Check the filesystem in DBFS.

As the oracle user in each one of the primary and each one of the standby middle tier nodes, check the existence of

the mount point and the /u01/soacs/dbfs/share/ directory.

[oracle@soacsdroci-wls-1~]$ mount | grep dbfs

dbfs-@ORCLB:/ on /u01/soacs/dbfs type fuse

(rw,nosuid,nodev,max_read=1048576,default_permissions,user=oracle)

dbfs-@ORCLB:/ on /u01/soacs/dbfs_directio type fuse

(rw,nosuid,nodev,max_read=1048576,default_permissions,user=oracle)

[oracle@soacsdroci-wls-1~]$ ls -lart /u01/soacs/dbfs/share/

total 0

drwxr-x---. 4 oracle oracle 0 Nov 21 18:09 ..

drwxr-x---. 2 oracle oracle 0 Nov 23 15:45 .

Create a test file in primary and check that it is readable in standby. In any of the middle tier primary nodes run

[oracle@soacsdroci-wls-1~]$ echo "test" > /u01/soacs/dbfs/share/test.txt

If the database edition is enabled for Active Data Guard (the Extreme Performance Edition required) the standby

database will automatically be open for reads, so changes applied in the dbfs mounts in primary can immediately be

read in standby. If the Database edition has not enabled Active Data Guard, the database in standby will have to be

converted to snapshot in order to be able to read the file created in primary. If your system is not using Active Data

Guard, convert the standby database to snapshot (when using Active Data Guard, the standby file system should

have access to the file created without any steps):



Converting database " secondary_db_unqname" to a Snapshot Standby database, please

wait...


Check the file on the secondary’s WebLogic Administration node


[oracle@soacsdroci-wls-1~]$ cat /u01/soacs/dbfs/share/test.txt

test

Once the basic DBFS file system propagation is verified, you can proceed with the customization of the dbfscopy.sh

script that will synchronize the domain directory from primary to standby. Here are some considerations for using

DBFS to replicate configuration:

It is critical that deployments are pushed to the WebLogic domain directory before doing manual

switchovers (or by using the appropriate cron job frequency). Otherwise, there may be windows between

the last update to the standby domain directory and the role switch that could cause data loss. The data

loss window will be determined by the frequency of the cron jobs in primary and standby.

Any data or files that are created OUTSIDE the domain directory in the WebLogic Administration Server

node, are not taken care of by the dbfscopy.sh script and need to be synchronized separately.

Download, customize and run the dbfscopy.sh script. Optionally enable cron jobs

The domain configuration copy and replacement script is intended to copy (using rsync) the WebLogic domain

configuration to the DBFS stage directory in the primary location and to copy (using rsync) the stage directory

contents to the WebLogic domain directory in the secondary/standby system. It also “maps” (replaces) the

Datasources’ connect strings and Cloud identity domain names in each site. The script is used on both sites so that

either one can assume primary role and replicate configuration to the other. It can be executed manually in a

controlled fashion or automatically at regular intervals using cron jobs. Whether the manual or “croned” approach

applies better to each case depends on the frequency and size of the configuration changes applied. The frequency

recommended will vary depending on each system’s lifecycle (how frequently new deployments and configuration

changes are applied). When the configuration copy is “croned” and the changes do not involve large files, then rsync

and Data Guard will provide a quick propagation that will transfer everything in a small lapse of time from the primary

to the secondary location. However if the rsync copy takes a long time (because some files are very large or

because many files are involved) it could occur that only a part of the configuration changes get “applied” to the

primary DBFS stage directory at the point in time where the standby cron moves things to the secondary domain

location. If the primary node crashed at that point in time, the standby domain config could be invalid/inconsistent.

This scenario can be avoided with different approaches:

1)Using the --delay-updates will make the rsync copy operation more “atomic” (this is achieved by creating a

temporary file for each updated file into a holding directory until the end of the transfer, at which time all the files are

renamed into place in rapid succession)

2)Using rsync to copy to another intermediate folder from which a tar file is created, copying this tar file to the DBFS

location and then extracting it in standby. Both approaches incur in additional space requirement.

The applicability of these options will depend on the type of configuration changes and deployment frequency. For

typical change frequencies (a few new deployments per day) and average composite and ear file sizes (less than

500KB) Oracle has found this rsync options unnecessary. The example dbfscopy.sh script provided, however, can

be extended to use also the tar approach.

Additionally, the script requires access from each middle tier site to the remote Database to retrieve status

information. By default the OCI DB System instances used by SOACS need to reside in in public networks and

access to their 1521 port can be enabled from the external world with the appropriate Internet Gateway. If the

access to the DB was configured through a VPN tunnel or a Dynamic Routing Gateway, the primary and standby


middle tiers will have to be enabled to access these accordingly. Refer to the SOACS and OCI documentation for

details on the network requirements for SOACS on OCI.

To use the DBFS copy scripts follow these steps:

A. Download/copy the script from OTN to the primary WebLogic Administration Server node. The

script is available here.

B. Open the dbfscopy.sh script, read its instructions. Latest version of this script do not need any

variable customization, it just prompts for the sys password.

C. Execute the script manually first in the primary WebLogic Administration Server node before

creating a cron in primary to execute it at regular intervals. Monitor the execution and watch for

any errors. The script will verify the DG status and will copy the domain configuration from the

primary WebLogic domain to the DBFS mount point.

D. Once it completes, download/copy the dbfscopy.sh script to the secondary WebLogic

Administration Server node.

E. Open the script, read its instructions. Latest version of this script do not need any variable

customization, it just prompts for the sys password.

F. Execute the script manually in the secondary middle tier WebLogic Administration Server node

before creating a cron in standby to execute at regular intervals. Monitor the execution and watch

for any errors. The script will verify the DG status, will copy the domain configuration from the

DBFS mount point to the secondary WebLogic domain and will make the required replacement

in the configuration that are required in the standby.

G. Restart the administration server in the secondary location for the changes to be effective (needs

to have the secondary DB in snapshot standby mode or use Active Data Guard)

IMPORTANT: The configuration under domain_home/config is automatically copied over to all other nodes that are

part of the WebLogic domain when the managed servers are restarted and connect to the Administration Server .

Any other configuration residing out of the domain_home/config directory will be copied ONLY to the first node and

will have to be manually replicated to each of the managed servers nodes. This includes any customizations to start

scripts under domain_home/bin domain_home/security etc.

H. Once this initial execution in primary and secondary is complete, the scripts can be added to the

cron list in the system so that they get executed regularly.

IMPORTANT: Notice that “croning” the copy script automates synchronization but also has the following

implications:

1) Synchronization may incur in latency as high as the frequency of the cron jobs in both locations added up. i.e. if

the cron jobs are set to execute every 30 minutes each, the changes may take 60 minutes to be available if the

window in primary overlaps with the one on the secondary location. Before performing a switchover, make sure that

this amount of time has passed by after the last configuration change. Otherwise, you could switchover before the

change is present on standby and overwrite the changes originally applied with the role switch

https://cloud.oracle.com/opc/soa/documentation

https://docs.cloud.oracle.com/iaas/Content/Network/Tasks/managingDRGs.htm

https://www.oracle.com/technetwork/database/availability/downloads/dbfscopy-5592965.zip


2) The cron frequency should be set at minimum to the largest amount of time a deployment or configuration change

may take to be copied from the domain directory to the dbfs stage directory. Otherwise, copy jobs may overlap

Scaling the SOACSDR system

Scale out operations may not be performed in a SOACSDR service in the current Cloud version.


Conclusion

Disaster recovery in a SOA Cloud Service configuration consists of a production database and a standby database

synchronized by Oracle Data Guard. Two separate middle tier configurations, each pointing to their local database,

are created to minimize the file synchronization needs across data centers. With this Disaster Recovery solution,

Oracle Cloud eliminates the costs and complexity of owning and managing a standby hardware, software, and

remote data center – while achieving the best Recovery Time Objective and Recovery Point Objective.

The use of Oracle Data Guard for disaster recovery provides better RTO and RPO than restoring a remote backup;

production is quickly failed over to an already running and synchronized copy of your production database on the

Oracle Cloud. The standby database in the cloud not only provides disaster recovery, it can also be used to seed

clone databases for development and test.

The use of middle tiers with a streamlined configuration replication facilitates maintenance and reduces the

overhead caused by continuous configuration approaches. However, an appropriate methodology and regular

standby verifications are need to guarantee a consistent recovery. Depending on each system’s lifecycle, different

synchronization approaches may be used for optimum behavior.


Appendix A - Setting up Data Guard manually

It is required to configure Data Guard manually when it is not possible to use a single tenancy for primary and

standby or when the automated Data Guard option provided by OCI is not enabled for the DB flavor and/or locations

involved in the DR configuration. The manual process described in this section assumes that the primary’s site

Database is already provisioned. This paper provides tools and examples to create a Data Guard configuration for a

single instance DB.

NOTE: the SOACS DR Setup procedure described in this paper has not been certified with RAC database

For single instance DB systems, proceed with the following steps.

1. The secondary database needs to be created with specific database name, shape and version to

match primary. To obtain the required data for your OCI Database click on your Database System in

the OICI Console

Note down the following information:

SHAPE: for example VM.Standard2.1

SOFTWARE EDITION: For example Enterprise Edition Extreme Performance

AVAILBALE DATA STORAGE: For example 256GB

DB SYSTEM VERSION: For example 12.2.0.1.180417

DATABASE NAME: For example ORCLTEST

PORT: 1521

Verify also the pdb name that is used in primary. This can be determined with the following SQL query in the

primary DB node


SQL> SELECT NAME, CON_ID FROM V$CONTAINERS;

NAME CON_ID

-------------------- ----------

CDB$ROOT 1

PDB$SEED 2

PDBTEST 3

2. Change the region in the OCI- Console to the appropriate location where the standby will reside. Click

the launch DB System button to create the secondary DB. The following table describes the

parameters that should be used in the secondary system’s provisioning screen using the examples

above:

DB Sytem Configuration Existing Primary Db System

System/example

Secondary DB System System

being created/example

Display Name XXXX/soadb XXXX/soadb (same as in primary)

Availability Domain XXXX/us-ashburn-1 YYYY/us-phoenix-1 (systems are expected

to be remote, hence different from primary

but needs to be the same where the

secondary SOACS service will be created)

Shape XXXX/Virtual Machine VM Standard 2.1 XXXX/Virtual Machine VM Standard 2.1

(same as primary)

Oracle Database Edition XXXX/ Enterprise Edition Extreme

Performance

XXXX/ Enterprise Edition Extreme

Performance (same as primary)

Available Storage Size XXX/256 XXX/256 (same as primary)



primary)

Virtual Cloud Network XXXX/soacsDRVCN XXXX/soacsDRVCN (different region will

use a different subnet from primary)15

Hostname prefix XXXX/soacsDR YYYY/soacsDR (can be different from

primary)


Database Version XXX/12.2.0.1.180417 XXX/12.2.0.1.180417 (check the “Display

All available versions” and make sure you

select the same one as in primary)

PDB Name XXXX/PDBTEST XXXX/PDBTEST (same as primary)

Database Admin Password XXXX/Acme1234# XXXX/Acme1234#(same as primary)

Database Workload ON-LINE TRANSACTION

PROCESSING

ON-LINE TRANSACTION PROCESSING

Once the secondary DB is created make sure to apply the required patches to the DB in both locations (primary

and secondary) so that to both are at the same patch level. More precisely a Data Guard configuration

15 SQLNet connectivity is required between the primary and standby DB. This connectivity can be achieved using the public internet since the DG traffic

is encrypted. Other options like Dynamic Routing Gateways are possible as long as the SOACS system using the DB system supports them. By

default the DB System instances used by SOACS need to reside in in public networks and access to their 1521 port can be enabled from the external

world with the appropriate Internet Gateway. If the access to the DB was configured through a VPN tunnel or a Dynamic Routing Gateway, the primary

and standby middle tiers will have to be enabled to access these accordingly. Refer to the SOACS and OCI documentation for details on the network

requirements for SOACS on OCI

https://cloud.oracle.com/opc/soa/documentation



across different DB domains requires a fix for bug 22611167. Make sure the pertaining patch for your

precise database version is applied in both the primary and standby systems.

Before running the following steps, verify that the appropriate ingress rules are defined in the OCI console to

allow connections between the primary and secondary databases. It is also needed that each database can

reach its own “external” IP16 on the appropriate listener port.

3. Download/copy the dataguardit_primary.sh script from OTN to the primary Database node. The

script is available here.

4. Open the script as oracle user, read its instructions and customize the variables explained in its initial

section.

5. Once customized, execute the script as oracle user. As explained in the script, it’s execution will

create a tar file that needs to be copied from primary to standby. The location where the tar is created

can be customized with a script variable.

6. Copy the tar file to the standby database node. As the opc user in the primary db node execute the

following:

[oracle@soacsdrDBa ~]$ scp -i cloud_private_sshkey.ppk /tmp/ORCL.GZ opc@standby_DB

System_public_ip:/tmp/

For example

[oracle@soacsdrDBa ~]$ scp -i /u02/install/fca-toshare.ssh.ppk /tmp/ORCL.GZ

[email protected]:/tmp/

7. As opc user in standby, change the rights on the file to make it readable by the oracle user

[opc@soacsdrDB2 ~]$chmod o+r /tmp/ORCL.GZ

8. Download/copy the dataguardit_standby_root.sh script from OTN to the secondary Database node.

The script is available here.

9. Open the script as root user, read its instructions and customize the variables explained in its initial

section.

NOTE: As documented in the Oracle Public Cloud documentation, you can gain root access to a

provisioned cloud instance by connecting to the opc user and then using the sudo command:

[opc@soacsdb ~]$ sudo su -

10. Once customized, execute the script as root user. The script will delete the existing database instance

and will create a new one duplicating the primary. It will also set up de database listener and

configuration required for Oracle Data Guard broker. Monitor de execution of the script and check for

any errors. The script will create a log file (/tmp/dataguardit_date.log ) Check this log for

troubleshooting information. The script can be executed again in case of failure (it will do the cleanup

of any previous failed executions).

16 This IP is the one that Will be used for redo transport. It can be the public one when using an internet gateway or the internal one when using a

dynamic routing gateway.

https://www.oracle.com/technetwork/database/availability/downloads/dataguardit-primary-5592937.zip

https://www.oracle.com/technetwork/database/availability/downloads/dataguardit-standby-root-sh-5592964.zip


11. After the script completes, enter the Data Guard Broker CLI from the primary system to check the

configuration (redo apply may take some time to catch up)

DGMGRL> show configuration verbose

Configuration - ORCL_ORCLB_12:55:19-22-11-18

Protection Mode: MaxPerformance

Members:

orcl - Primary database

orclb - Physical standby database

Properties:

FastStartFailoverThreshold = '30'

OperationTimeout = '30'

TraceLevel = 'USER'

FastStartFailoverLagLimit = '30'

CommunicationTimeout = '180'

ObserverReconnect = '0'

FastStartFailoverAutoReinstate = 'TRUE'

FastStartFailoverPmyShutdown = 'TRUE'

BystandersFollowRoleChange = 'ALL'

ObserverOverride = 'FALSE'

ExternalDestination1 = ''

ExternalDestination2 = ''

PrimaryLostWriteAction = 'CONTINUE'

ConfigurationWideServiceName = 'ORCL_CFG'

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS

12. OPTIONAL: Set TCP Socket Maximum Sizes recommended for DG in primary and secondary

[root@soacsdb ~]$ sysctl -w net.core.rmem_max=10485760

[root@soacsdb ~]$ sysctl -w net.core.wmem_max=10485760

The /etc/sysctl.conf file should also be edited to reflect the changes so that they would be preserved during an

instance reboot.


NOTE: the dataguardit_standby_root.sh script performs a default sizing of the Online Redo Logs The redo logs

created by default in the Database Cloud Service were 1GB in size and these were sufficient for our testing. The

online redo logs should be sized by this formula, but should not be less than 1GB in size:

peak redo rate per minute x 20

Redo rates can be extracted from AWR reports during peak workload periods such as batch processing, quarter or

year end processing. It is very important to use peak workload and not averages (averages can obscure peak redo

rates and lead to provisioning redo logs that are too small).

The MAA best practice for Standby Redo Logs (SRLs) is to create the same number as there are groups of Online

Redo Logs (ORLs) plus 1. On RAC this must be done for each thread. Use the following query to discover how

many redo log groups you have in each thread.

select thread#, count(group#) from v$log group by thread#;

SRLs should be created the same size as the largest of the ORLs. The group numbers are shared with the ORLs

and so SRLs are created with higher group numbers. To gather the size of the largest ORLs and the current highest

group number execute the following query:

select max(bytes), max(group#) from v$log;

The MAA best practice is that SRLs are not duplexed.


Appendix B – Cloud backups in DB Systems

Backing up the DB System is a key aspect of any Oracle database environment. Oracle Cloud offer various approaches: you can store backups in local or cloud storage; the backup can be automatic, custom using rman, or dbcli. In a DR scenario, there are some special considerations because the databases are configured with Data Guard. When the Data Guard is configured manually (with steps explained in the previous Appendix A), the backup needs to be configured manually in order to get the optimal configuration in a Data Guard environment. You can perform the backups in one of the databases (primary or standby) and control the archivelog growth in the other one. To configure manual backups in the primary DB System:

If the automatic backup was enabled in OCI Console for this system, the backup module should be already configured by the automatic backups. In that case, disable automatic backup so you can customize it. If automatic backup have never been enabled before, you can follow the steps described in Backing Up a Database to Object Storage Using RMAN to install and configure the backup module in the Primary DB.

Configure rman settings as recommended in the link. In addition to that, ensure that you also include the archivelog deletion policy recommended for Data Guards:

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE'

APPLIED ON ALL STANDBY;

Create your rman backup scripts as per your backup requirements and include it in the crontab. This is just an example to run a full backup:

# Run RMAN

export ORACLE_HOME=/u01/app/oracle/product/12.2.0.1/dbhome_1

export ORACLE_SID=ORCL

$ORACLE_HOME/bin/rman <<RMAN

connect target /

SET ENCRYPTION ON;

BACKUP DATABASE PLUS ARCHIVELOG TAG "FULL_BACKUP";

exit;

RMAN

echo "Completed full backup for" $ORACLE_SID

To control the archivelog growth in the standby:

Disable automatic backup if it was enabled for this system, and configure the proper archivelog deletion policy so archivelog are not deleted if they are not yet applied to standby (archivelog deletion policy to “applied on all standby”).

Although setting the correct archivelog deletion policy should be enough to control the archivelog growth in the FRA, you can also create a cleanup script to delete old archive logs. This is an example to clean old archive logs that uses a archivelog deletion policy to prevent undesired archivelog deletion:

######################################################

# Use this script to clean old archive logs from disk

# when the database is in STANDBY role and no backups are performed

# Run RMAN

export ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_1

export ORACLE_SID=ORCL

$ORACLE_HOME/bin/rman <<RMAN

connect target /

https://docs.cloud.oracle.com/iaas/Content/Database/Tasks/backingupOSrman.htm


# To prevent undesired archivelog deletion if this DB takes primary role

CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY;

# Delete archivelog older than 20 days

delete noprompt archivelog all completed before 'SYSDATE-20';

exit;

RMAN

echo "deleted applied old archivelogs on $ORACLE_SID"

######################################################

When the Data Guard is configured using Cloud Console UI, you can enable automatic backups in the primary database and this is a good approach. The default rman configuration in those cases uses the recommended archivelog deletion policy for the Data Guard scenario. However, you will have to control the archivelog growth in the secondary database as well as explained before.


Appendix C – Using a Single Tenancy to create a DR system

As explained in in the Configuration Requirements section in this paper, it is possible to create a SOACS/MFTCS

DR configuration using a single tenancy. The WebLogic domain name and WebLogic server names used in a

SOACS/MFTCS system are based on the SOACS/MFTCS cloud instance name provided during the provisioning

process. This way, if soacsdroci (for example) is used as the SOACS cloud instance service name, the provisioning

tools will create a WebLogic domain named soacsdro_domain, with a cluster named soacsdro_cluster and two

servers named socsdro_server_1 and soacsdro_server_2. To keep these WebLogic names consistent in primary

and standby under a unique tenancy, it is required that the primary cloud service instance uses a name with at least

eight characters (which is the limit used for the WebLogic domain, cluster and server names). That way we can

create a secondary instance name (that is called soacsdrocistandby for example) in that same tenancy and get the

exact same WebLogic artifacts’ names (soacsdro_domain, soacsdro_cluster, soacsdro_server_1 and

soacsdro_server_2). With this, since there are no conflicts at the Cloud Service instance name, a single tenancy can

be used to create primary and standby. In summary, primary and standby SOACS/MFTCS instance will have to

share the first 8 characters in their name (for example soacsdroci/soacsdrocistandby,

soacsdmycompany/socsdrmycompanystandby, mycompanyprimary/mycompanysecondary etc.)

Once a single tenancy has been used use the same steps and procedures defined in the “SOACS/MFTCS DR

Deployment” section in this white paper with the following considerations:

1. When using a single tenancy, the configuration option available in the OCI Console for DB Systems

can be used to configure Data Guard for the primary Database (refer to the Database Cloud Service

Documentation to set up Data Guard) i.e. you do not need to follow the process to configure Data

Guard manually as described in Appendix A. Keep in mind, however, the following considerations

a. SOACS and MFTCS need support such a database “flavor” (Database version, RAC,

deployment platform etc.)

b. The DG configuration option needs to be available in the precise two locations that you want

to use for your SOACS DR system.

c. It will be needed to access the Database Nodes to manage the Data Guard configuration as

described in different set up sections in this white paper (converting the physical standby to

physical snapshot, reverting to physical standby etc.)

d. Even when the Data Guard configuration is created automatically, make sure that the latest

patches are applied to the Db software (Data Guard configuration across different DB

domains requires a fix for bug 22611167. Make sure the pertaining patch for your precise

database version is applied in both the primary and standby system)

2. For the SOACS/MFTCS set up process using the automated Data Guard configuration, follow the

same steps described in the “Set up Details” section in this document. Notice however that the

process to create a database service and configure Data Guard for it are replaced directly with the

automated Data Guard process in the Cloud Console. Refer to the Database Cloud service

documentation for details on this.

https://docs.cloud.oracle.com/iaas/Content/Database/Tasks/usingdataguard.htm

https://docs.cloud.oracle.com/iaas/Content/Database/Tasks/usingdataguard.htm


2. Once the Data Guard configuration is ready and the appropriate patches have been applied,

convert the physical standby database to snapshot standby logging to the secondary Database

node just like described in previous sections for the tow tenancies case.

3. Per the above, use the appropriate nine characters (at least) service name when creating the

SOACS secondary service where the 8 first characters are common with the primary service

name.

4. The execution of the DRS tool is required as described in previous sections for the two tenancies

case.

5. The lifecycle procedures are the same whether the DG system was created manually or using

the Cloud Console.


Appendix D – Using Enterprise Manager Site Guard to manage the SOACS DR switchovers

Oracle Site Guard can be used to coordinate the SOACS Disaster Recovery switchover as described in this

whitepaper. Oracle Site Guard is a disaster-recovery solution that enables administrators to automate complete site

switchover or failover. It orchestrates the coordinated failover of Oracle Fusion Middleware, Oracle Fusion

Applications, and Oracle Databases. It is also extensible to include other data center software components. Oracle

Site Guards offers the following benefits:

Minimizes disaster-recovery time

Reduces human errors

Flexible and customizable

Eliminates the need for special skills

Oracle Site Guard is included in Enterprise Manager Cloud Control Fusion Middleware Plugin. Enterprise Manager

Cloud Control Management Server and Agent deployment is required to use Oracle Site Guard in a SOACS DR

environment.

A sample “SOACSDR with Site Guard” topology is described in the next picture:

Figure 2 SOACS DR topology and Enterprise Manager Cloud Control Site Guard

Notice that a single EM installation like the one described in this appendix can be used to orchestrate and

manage multiple Disaster Protection systems


The following steps are required to accomplish this setup:

1. Enterprise Manager Cloud Control Setup

Install and configure the EM Cloud Control Oracle Management Server in Cloud.

2. Network Setup

Create the required network rules to allow the communications between targets and Oracle management

Server.

3. Agent Installation

Install Enterprise Manager Cloud Control Agent in the SOACS DR hosts.

4. Target Discovery

Discover the targets that will be managed by the Site Guard (WebLogic Domains, Databases, etc.)

5. Site Guard Configuration

Configure Site Guard (sites, credentials, scripts, plans, etc.) to orchestrate the switchover and failover in

the SOACS DR environment.

It is expected that the required Enterprise Manager Cloud Control licenses with Oracle Site Guard are used. Basic

technical background on Enterprise Manager Cloud Control concepts and administration is assumed for completing

the setup. Refer to the next sections for details on each one of the steps.

ENTERPRISE MANAGER CLOUD CONTROL SETUP

If you already have an Enterprise Manager Cloud Control installed and configured, you can skip this step and

continue with the rest of the sections (network setup, agent installation, Site Guard configuration).

If you do not have an Enterprise Manager Cloud Control, you can follow the steps in Setting Up Oracle Enterprise

Manager 13.3 on Oracle Cloud Infrastructure in order to create and configure an EM in your cloud tenancy based on

a Market Place EM image. This Marketplace Image contains pre-configured Oracle Enterprise Manager (13.3

PG) with co-located Oracle Database(19.3). As a result, an EM host will be created with the name

“emcc.marketplace.com”. This hostname is used in next points to refer to the OMS hostname.

NETWORK SETUP

1. Hostnames setup

Enterprise Manager hostname and its monitored hosts must be mutually resolvable.

Primary and Standby sites are typically located in different cloud datacenters, and the OMS can be located in one of

them or in another different datacenter.

When there is no internal communication between the datacenters, Enterprise Manager Cloud Control OMS and

SOACS DR targets will communicate each other via their public IPs. Oracle recommends using hostnames

associated to the public IPs of the cloud hosts. This can be done by registering them in a DNS server, or by

configuring the name resolution in the /etc/hosts file of the OMS and target hosts.

Private names for the hosts are given by cloud infrastructure, but the public names must be defined/customized in

each case. For example:

Site Private Name Public Name (defined by customer) Public IP

SiteN emcc.marketplace.com emcc.marketplace.com 111.111.111.100

Site1 soacsdroci-wls-

1.site1cloudinternaldomain.com

soacsdroci-wls-1-

public.site1.example.com

111.111.111.11

https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.3.1/emoci/index.html

https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.3.1/emoci/index.html


soacsdroci-wls-


soacsdroci-wls-2-


111.111.111.12

soacsdba.site1cloudinternaldomain.com soacsdba-public.site1.example.com 111.111.111.13

Site2 soacsdrocistby-wls-


soacsdrocistby-wls-1-


222.222.222.11

soacsdrocistby-wls-


soacsdrocistby-wls-2-


222.222.222.12

soacsdbb.site2cloudinternaldomain.com soacsdbb-public.site2.example.com 222.222.222.13

These is an example of the /etc/hosts file entries that would be set in OMS host and in each monitored host when

they are communicated using public IPs:

###############################################################################

# OMS host public IP

111.111.111.100 emcc.marketplace.com

###############################################################################

# Public names for cloud monitored hosts

###############################################################################

# SITE 1

111.111.111.11 soacsdroci-wls-1-public.site1.example.com

111.111.111.12 soacsdroci-wls-2-public.site1.example.com

111.111.111.13 soacsdba-public.site1.example.com

# SITE 2

222.222.222.11 soacsdrocistby-wls-1-public.site2.example.com

222.222.222.12 soacsdrocistby-wls-2-public.site2.example.com

222.222.222.13 soacsdbb-public.site2.example.com

In scenarios when the communication between OMS and monitored hosts is possible using private networks

(through a VPN tunnel or a Dynamic Routing Gateway17), the OMS and monitored hosts can use the appropriate

internal IPs/names instead of the public IP/names to communicate each other. In that case, the /etc/hosts files of the

OMS should contain the internal IPs and internal names of the hosts.

2. Security Rules

Oracle Management Servers needs to communicate with the agents in the monitored hosts, and the agents connect

to OMS server to upload the monitoring data. For discovering FMW targets, the OMS needs to be able to

communicate with the Administration Server of the WebLogic domains, and for database monitoring, OMS needs to

be able to connect to the target database. All this traffic is encrypted given that secured protocols are used (HTTPS,

t3s and SQL*NET with network encryption18). The following communications need to be allowed between the OMS

and the monitored targets:

SOURCE DESTINATION PROTOCOL

Any monitored host OMS upload port (4903) HTTPS

OMS host Any monitored host Agent port (3872) HTTPS

OMS host Any target Database listener port (1521) SQL (with network encryption)

17 Refer to the Dynamic Routing Gateways for details on the network configuration 18 Administering Oracle Database Cloud Service > Using Network Encryption and Integrity


https://docs.oracle.com/en/cloud/paas/database-dbaas-cloud/csdbi/use-network-encryption-and-integrity.html


OMS host Monitored WebLogic domain Admin Server port

(7002)

T3S

OMS host Any monitored host ssh port (22) SSH (to copy the agent software)

Internet (19) OMS console port (7803) HTTPS

However, some of these communications are not be allowed by default. To enable them, security rules are required.

Follow the steps in the next points to create them.

A. Security Rules in OMS’ side

The following network rules need to be defined to allow communication to OMS:

1. Login to the OCI Console

2. Navigate to Networking > Virtual Cloud Networks.

3. Click on the Virtual Cloud Network of the OMS.

4. Click in the security list under Security Lists.

5. Add Ingress Rule (stateful) to allow traffic from Any monitored host to OMS upload port:

Source Type: CIDR

Source CIDR: Monitored hosts network CIDR20. Example: 111.111.111.0/24

IP Protocol: TCP

Source Port Range: All

Destination Port Range: 4903

Repeat for each monitored hosts network.

6. Add Ingress Rule (stateful) to allow traffic from Internet (or from specific network) to OMS

console port:

Source Type: CIDR

Source CIDR: Use 0.0.0.0/0 to allow access to all or specify a network CIDR.

IP Protocol: TCP



B. Security Rules in targets’ side

The following network rules need to be defined to allow communication from OMS to host targets:

1. Login to the OCI Console

2. Navigate to Networking > Virtual Cloud Networks.

3. Click on the Virtual Cloud Network of the targets in Site1.

4. Click in the security list under Security Lists.

5. Add Ingress Rule (stateful) to allow traffic from Any monitored host to OMS upload port:

Source Type: CIDR

Source CIDR: OMS IP in CIDR format21. Example: 111.111.111.100/32

IP Protocol: TCP

19 This access is required for Enterprise Manager/Site Guard Administrators. If possible, open the OMS console port only for their specific

IPs instead to opening to internet. 20 Use public network CIDR when OMS and targets communicate via internet, use private network CIDR when the communication via

private IPs is possible. 21 Use OMS public IP CIDR when OMS and targets communicate via internet, use OMS private IP CIDR when the communication via

private IPs is possible




6. Repeat for the Virtual Cloud Network of the targets in Site2.

NOTE: access to the listener port (1521), admin server console (7002) and ssh (22) are expected to be already

open. If not, add the pertinent rules to allow traffic from OMS public IP to those ports in the same way as above.

In addition to network rules the following iptables rules need to be added in the target DB systems:

7. Connect via ssh to the DB system host.

8. Log in as opc and then sudo to the root user.

9. Save a copy of iptables as a backup.

[root@soacsdrDBa ~]# iptables-save > /tmp/iptables.orig

If necessary, you can restore the original file by using the command iptables-restore <

/tmp/iptables.orig.

10. Dynamically add a rule to iptables to allow inbound traffic on the OMS upload port, as shown in

the following sample.

[root@soacsdrDBa ~]# iptables -I INPUT 8 -p tcp -m state --state NEW -m

tcp --dport 3872 -j ACCEPT -m comment --comment "Required for EM agent

port"

11. Make sure the rules were added.

[root@soacsdrDBa ~]# service iptables status

12. Save the updated file to /etc/sysconfig/iptables.

[root@soacsdrDBa ~]# service iptables save

13. The change takes effect immediately and will remain in effect when the node is rebooted.

Using the previous example, this is a summary of the security rules created:

Cloud Site for

the Rule

Rule

Type Source

TYPE

Source

CIDR

IP

PROTOCOL

Source Port

Range Destination Port Range

SiteN (OMS) Ingress

Stateful

CIDR Site1 Network CIDR

Example: 111.111.111.0/24

TCP All 4903

(upload port)

SiteN (OMS) Ingress

Stateful

CIDR Site2 Network CIDR

Example: 222.222.222.0/24

TCP All 4903

(upload port)

SiteN (OMS) Ingress

Stateful

CIDR Administrator’s Network CIDR

(0.0.0.0/o to allow Access to Internet)

TCP All 7803

(OMS Console Port)

Site1 Ingress

Stateful

CIDR OMS IP in CIDR format.

Example: 111.111.111.100/32

TCP All 3872

(agent port)

Site2 Ingress

Stateful

CIDR OMS IP in CIDR format.

Example: 111.111.111.100/32

TCP All 3872

(agent port)


AGENT INSTALLATION

Enterprise Manager Cloud Control Agent must be installed in all the hosts of the SOACS DR environment.

1. Target Host preparation

Perform the following steps to prepare the target host for the agent installation.

A. Create user and group

A dedicated user can be used to install and run the agent software, in order to isolate processes,

environment variables, etc. from the monitored software. Create the user (for example: emcadm) in the host

where the agent will be installed, and add it to the group of the user that is running the software in the cloud

machine (oracle group in SOACS hosts, oinstall group in DB hosts).

For SOACS hosts, software group is oracle:

[root@host]# useradd -g oracle emcadm

For DB hosts, software group is oinstall (oracle group does not exist):

[root@host]# useradd -g oinstall emcadm

NOTE: specific user for the agent is not mandatory. User “oracle” can also be used for installing and running

the agent. The steps in this documentation use a specific agent user (emcadm), in order to identify any step

requirement/action needed when the user running the agent is different than the user running the monitored

software.

B. Verify communications with OMS

Verify that the host is able to resolve the Enterprise Manager OMS hostname and its own public name (it

public names are used). This should be already configured as described in the previous section NETWORK

SETUP.

Verify that Enterprise Manager OMS upload port is reachable. Use the appropriate OMS IP (public or

private) for your environment:

$ nc -v -w 5 -z <oms_ip> 4903

Connection to <oms_ip> 4903 port [tcp/*] succeeded!

C. Create Agent Home base folder

The agent home base folder has some requirements, especially important when Privilege Delegation

Provider is used (which will be used for the credentials). See Agent Base Directory Requirements in the

“Installing Oracle Management Agent in Silent Mode" chapter of the Enterprise Manager Cloud Control

documentation.

The following folders are suggested for the agent home base in the storage volumes that are already

mounted in the hosts:

For DB hosts: /u01/agent13c (under /u01 volume)

For SOACS hosts: /u01/agent13c (note this is under / volume)

AGENT_BASE_DIR will be used to refer to the agent base folder.

1. Create the folders, with user root:

[root@soacsdroci-wls-1~]# mkdir -p AGENT_BASE_DIR

2. Change the ownership of that folder to the user and group that will run the agent

In SOACS hosts:

[root@soacsdroci-wls-1~]# chown emcadm:oracle AGENT_BASE_DIR

In DB hosts:

http://www.oracle.com/pls/topic/lookup?ctx=E93167-01&id=GUID-EE16062E-D583-4A0A-8786-03C2895BD1DB


[root@soacsdrDBa]# chown emcadm:oinstall $AGENT_BASE_DIR

3. Add read and execute permissions to the group and others to the folder and to the parent folders

also. This is also required for Privilege Delegation:

[root@soacsdroci-wls-1]# chmod go+rx /u01/agent13c

[root@soacsdroci-wls-1]# chmod go+rx /u01

D. Review other requirements

The complete list of the requirements for the agent is in Table 6-1 Prerequisites for Installing Oracle

Management Agent in Silent Mode in the “Installing Oracle Management Agent in Silent Mode" chapter of

the Enterprise Manager Cloud Control documentation. It is expected that SOACS and DB hosts meet those

requirements, and no additional actions are required.

2. Get the agent software

AgentDeploy method will be used. In this method, you must use EM CLI to download the Management Agent

software onto the remote destination host before executing the script to install the Management Agent. You can

either choose to use EM CLI from the OMS host, or from the remote destination host. If you choose to use EM CLI

from the OMS host, you must transfer the downloaded Management Agent software to the remote destination host

before executing the script to install the Management Agent. This method supports many additional parameters, and

is ideal for a customized Management Agent install.

Use emcli in the OMS host to download the agent software and then copy it to the target host:

A. Login to emcli in the OMS host (MIDDLEWARE_HOME is /u01/app/em13c/middleware):

[oracle@emcc bin]$ cd $MIDDLEWARE_HOME/bin

[oracle@emcc bin]$ ./emcli login -username=sysman

Enter password :

Login successful

[oracle@emcc bin]$ ./emcli sync

Synchronized successfully

B. Verify the available software and get the agent image for the Linux x86-64 platform:

[oracle@emcc bin]$ ./emcli get_supported_platforms

-----------------------------------------------

Version = 13.3.0.0.0

Platform = Linux x86-64

-----------------------------------------------

Platforms list displayed successfully.

[oracle@emcc bin]$ ./emcli get_agentimage -destination=/tmp/agent_image -

platform="Linux x86-64" -version=13.3.0.0.0

=== Partition Detail ===

Space free : 8 GB

Space required : 1 GB

..

Downloading /tmp/agent_image/13.3.0.0.0_AgentCore_226.zip

Agent Image Download completed successfully.

C. Copy it to each remote host where the agent will be install using scp, for example:

[oracle@emcc ~]$ scp -i <public_ssh_key>.ppk

/tmp/agent_image/13.3.0.0.0_AgentCore_226.zip opc@<monitored-host-

name>:/tmp/agent_image

http://www.oracle.com/pls/topic/lookup?ctx=E93167-01&id=GUID-EE16062E-D583-4A0A-8786-03C2895BD1DB

mailto:[email protected]:/tmp/agent_image/

mailto:[email protected]:/tmp/agent_image/


3. Install the agent

In the host where the agent is going to be installed:

A. Verify that the agent user (example emcadm) can read the agent software zip.

B. Unzip the software with the agent user (emcadm)

[emcadm@soacsdba agent_image]$ unzip 13.3.0.0.0_AgentCore_226.zip

C. In the folder where the agent is unzipped, install it using agenDeploy.sh with the agent user

(emcadm). Example:

./agentDeploy.sh AGENT_BASE_DIR=/u01/agent13c OMS_HOST=emcc.marketplace.com

EM_UPLOAD_PORT=4903 AGENT_REGISTRATION_PASSWORD=welcome1

LOCALHOST=soacsdba.site1cloudinternaldomain.com LOCALPORT=3872 AGENT_PORT=3872

ORACLE_HOSTNAME=soacsdba-public-site1.example.com ALLOW_IPADDRESS=TRUE

START_AGENT=true

Where:

AGENT_BASE_DIR Directory where agent will be installed.

OMS_HOST The name to connect to the OMS. In this example: emcc.marketplace.com

EM_UPLOAD_PORT The upload port of the OMS. Typically 4903

AGENT_REGISTRATION_PASSWORD The agent registration password

LOCALHOST FQDN of the hostname where the agent is being installed (the private name).

Example:

[root@soacsdroci-wls-1 tmp]# hostname --fqdn

soacsdroci-wls-1.site1cloudinternaldomain.com

LOCALPORT/AGENT_PORT The port where the agent will listen. Example: 3872

ORACLE_HOSTNAME In case that the communication between OMS and targets is via internet, this

is the PUBLIC name of the monitored host that OMS will use to

communicate with this agent. Must either the public IP of the host, or a public

name resolvable to that public IP.

In case that communication between OMS and targets is done via private

networks (i.e.: DRG is used), this is the PRIVATE name of the monitored

host.

TIP: Use hostnames instead of IPs is recommended.

IMPORTANT!! Write this name in lowercase. Using uppercase in the public

name can cause errors validating the agent certificate.

ALLOW_IPADDRESS Enter TRUE if you want to specify an IP address for ORACLE_HOSTNAME.

If ALLOW_IPADDRESS is set to FALSE, a prerequisite check fails when you

specify an IP address for ORACLE_HOSTNAME while installing a

Management Agent. Not required is using hostnames.

TIP: Using names instead of IPs is recommended.

START_AGENT When set to true, agent will be started after the installation.

D. Once the installation has finished successfully, execute the root script as root user:

<AGENT_BASE_DIR>/agent_13.3.0.0.0/root.sh

E. Verify the status with <AGENT_BASE_DIR>/agent_13.3.0.0.0/bin/emctl status agent


Agent URL should show the public hostname of the host (when OMS and target are communicated

via public IPs), or private hostname (when OMS and target are communicated via private networks).

Verify that the Local Agent URL is using the private hostname

Verify that the Repository URL points to the OMS name

[emcadm@soacsdroci-wls-1 bin]$ cd <AGENT_BASE_DIR>/agent_13.3.0.0.0/bin

[emcadm@soacsdroci-wls-1 bin]$ ./emctl status agent

Oracle Enterprise Manager Cloud Control 13c Release 3

Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.

---------------------------------------------------------------

Agent Version : 13.3.0.0.0

OMS Version : 13.3.0.0.0

Protocol Version : 12.1.0.1.0

Agent Home : <AGENT_BASE_DIR>/agent_inst

Agent Log Directory : <AGENT_BASE_DIR>/agent_inst/sysman/log

Agent Binaries : <AGENT_BASE_DIR>/agent_13.3.0.0.0

Core JAR Location : <AGENT_BASE_DIR>/agent_13.3.0.0.0/jlib

Agent Process ID : 9778

Parent Process ID : 9673

Agent URL : https://soacsdroci-wls-1-

public.site1.example.com:3872/emd/main/

Local Agent URL in NAT : https://soacsdroci-wls-

1.site1cloudinternaldomain.com:3872/emd/main/

Repository URL : https://emcc.marketplace.com:4903/empbs/upload

Started at : 2018-10-31 11:33:27

Started by user : emcadm

…

Last successful heartbeat to OMS : 2018-10-31 11:36:32

Next scheduled heartbeat to OMS : 2018-10-31 11:37:32

---------------------------------------------------------------

Agent is Running and Ready

F. Verify that the agent upload runs ok:

[emcadm@soacsdroci-wls-1 bin]$ cd <AGENT_BASE_DIR>/agent_inst/bin

[emcadm@soacsdroci-wls-1 bin]$ ./emctl upload agent



---------------------------------------------------------------

EMD upload completed successfully

G. Restart agent to verify that everything is properly configured:

[emcadm@maa4-wls-1 bin]$ ./emctl stop agent



Stopping agent ... stopped.

[emcadm@maa4-wls-1 bin]$ ./emctl start agent

Oracle Enterprise Manager C

loud Control 13c Release 3


Starting agent ............. started.


H. Login in the OMS Console https://<oms_public_ip>:7803/em , go to Targets > Hosts and verify that

the host is registered.

NOTE: If there is any error and the agent needs to be reinstalled, it can be de-installed using this (execute it in

a folder outside the agent base dir). Write it in a single line:

<AGENT_BASE_DIR>/agent_13.3.0.0.0/perl/bin/perl

<AGENT_BASE_DIR>/agent_13.3.0.0.0/sysman/install/AgentDeinstall.pl –agentHome

<AGENT_BASE_DIR>/agent_13.3.0.0.0

If any target of the agent was registered in the OMS, it must be deleted also by decommissioning the agent.

See: “ EM 13C: How to Deinstall the Enterprise Manager 13c Cloud Control Agent (Doc ID 2095678.1)”


TARGET DISCOVERY

Primary and standby WebLogic domains and databases must be discovered in EM. The procedure to discover and

promote the targets running on an Oracle Cloud host is the same as the procedure to discover and promote targets

running on any normal host in the on-premises environment.

1. Promoting automatically discovered targets

Some targets are automatically discovered, and just required to be promoted. This is typically the process for Oracle

Database home, Oracle Grid Infrastructure home, Oracle High Availability Service and Cluster. To promote

automatically discovered these targets:

A. Login in OMS Console

B. Go to Setup > Add Target > Configure Auto Discovery

C. See “Target on Hosts”. You can select the host and click in “Discover now”

D. Then go to Setup > Add Target > Auto Discovery Results

E. See “Targets on Hosts”

F. Review the auto discovered targets, click in one of them and click in Promote.

Databases and ASM instances and cluster can be also automatically discovered. To promote them, see the

following points.

2. Discover ASM targets22

ASM instances are usually automatically discovered. Follow these steps to complete the discovery:

A. ASSNMP user is typically used to monitor the ASM databases. The customer needs to change the

password of the ASSNMP user23 . Do the following in any target DB host using ASM:

Login in the DB host as opc user and sudo to user grid.

Connect to the ASM instance:

sqlplus " / as sysasm"

Reset the password for the user ASMSNMP:

sql> alter user asmsnmp identified by <new password>;

B. In the Auto Discovery Results in OMS Console, select the discovered “Cluster ASM” target in and

click “Promote”.

C. In the Results screen, select the Cluster ASM target and click on “Configure”:

In General tab, set the Monitor username to ASMSNMP and set the password.

In Instances tab, ensure it is set to the hostname that the OMS uses to connect to this host

target. OMS needs to connect to the database. Use the proper machine host name (public

or private) depending on your network topology.

Click “Test Connection” to verify that the connection is successful and then click “Save”.

D. Back in the Results screen, verify that the ASM listener is also selected. Leave default values for the

listener. ASM listener target is monitored by the local agent and the private machine name can be

used.

E. Click Next and then Save

3. Discover Database targets

22 Docs SOP: Discover a Cluster ASM in EM 12c (Doc ID 1557390.1) 23 The ASMSNMP default password in OCI (Doc ID 2455524.1)


To promote or discover a database target:

The user DBSNMP is typically used to monitor the database. This user account is locked by default.

Connect to the primary database as sysdba to unlock if necessary and set a password:

sql> ALTER USER DBSNMP ACCOUNT UNLOCK;

Sql> ALTER USER DBSNMP IDENTIFIED BY password;

SOACS DR databases use Data Guard. If the standby database is open (Active Data Guard), the

DBSNMP user will be able to login with normal role. If standby database is mounted (not Active Data

Guard), the user DBSNMP needs SYSDBA privilege to login to the database24.

Run below SQL to check if DBSNMP user having SYSDBA privilege:

SQL> select username, sysdba from v$pwfile_users where

username='DBSNMP';

If above SQL does not return any rows, this means SYSDBA privilege not granted to

DBSNMP user.

Login in primary DB system and connect to primary database as sysdba

sqlplus / as sysdba

Grant sysdba to dbnsmp user:

SQL> grant sysdba to dbsnmp container=all

For database versions previous to 12.2, copy the password file from primary host to standby

host. The default location of the password file is $ORACLE_HOME/dbs/orapw$ORACLE_SID

Discover the database if it was not automatically discovered:

Login in OMS Console https://<oms_public_ip>:7803/em

Go to Setup > Add Targets Manually

In “Add Non-Host Target Using Guided Process” click in “Add Using Guided Process”

Select “Oracle Database, Listener and Automatic Storage Manage” and click “Add..”

In the “Database Discover: Search Criteria” screen, select the public name of the host

running the database

If the database is not found, verify that there is and entry for it in /etc/oratab and retry.

Once the Database has been discovered (either automatically or manually), select it in the Results

screen and click in “Configure”. Update with the following:

“Listener Machine Name”: ensure it is set to the hostname that the OMS uses to connect to

this host target. OMS needs to connect to the database. Use the proper machine host name

(public or private) depending on your network topology.

“Monitor Username”: enter the name of the database user that will be used to monitor it

from OMS. Typically dbsnmp.

“Role”: select “SYSDBA”.

“Monitor password”: enter the password for the dbsnmp.

Click the “Test Connection” and verify that it is successful.

Click Save.

F. Back in the “Database Discovery: Results” page, verify that the listener is also selected. Leave default

values for the listener. Listener is monitored by the local agent and the private machine name is used.

24 SOP: How to Discover a Standby Database in EM12c (Doc ID 1529428.1)


NOTE: if two listener target are discovered, LISTENER and LISTENER0, select the first one named as

LISTENER_<target_hostname> and ignore the duplicated.

Click Next

In the “Database Discovery: Review”, click Save

Once finished, you can go to Targets > Databases to verify that the database has been discovered.

4. Discover WebLogic Domains targets

WebLogic domains and its associated targets need to be manually discovered.

NOTE: the admin server and the agents of the target domain needs to be up and running for discovering the

domain.

A. Login in OMS Console

B. Go to Setup > Add Targets Manually

C. In “Add Non-Host Target Using Guided Process” click in “Add Using Guided Process”

D. Select “Oracle Fusion Middleware/WebLogic Domain” and click “Add..”

E. Provide the following details:

Administrator Server Host Set it to the public IP of the SOACS admin server host. This is used only for the discovery.

Port Admin server public port. In SOACS, typically 7002

Username/Password weblogic/<weblogic_password>

Node Manager Username/Password weblogic/<weblogic_password>

Unique Domain Identified Identifier used to differentiate WebLogic domains with the same names that are discovered in

the same Enterprise Manager. This is very important in SOACS Disaster Recovery

environments because the WebLogic domain name is the same for the primary and standby.

Use Site1 when discovering the WebLogic domain in Site1

Use Site2 when discovering the WebLogic domain in Site2

Agent Select the agent of the host where admin server is running

Discover Application versions Not relevant for Site Guard, can be checked or not

F. Click in Advance and:

JMX Protocol Use t3s.

Discover Down Servers Check

Enable Automatic Refresh Check

G. Leave the rest of the properties by default (empty)

H. Click Next

I. In the “Assign Agents” screen, review carefully that the agent and host assigned to each target is the

correct (specially for the second managed server). By default, each target should be assigned to the

local agent that is on the same host as the target. In case they are not assigned to the proper agent

or host, use “Change Hostname” or “Assign Agent” to correct any mismatch.


J. Once finished, go to Targets > Middleware to verify that the WebLogic domain has been discovered.

K. Repeat to discover the standby WebLogic domain. At least, admin server needs to be up.

NOTE: some managed server properties may be not be discovered when the managed server is stopped. It is

recommended to discover the standby WebLogic domain (or refresh it) when it is up, that is, when the site 2 is

the primary or when the standby database is in snapshot mode and the secondary WebLogic domain is started.

5. Target Discovery post-steps

Some post-steps must be performed to verify the sanity of the discovered targets.

A. Verify that the PDB is detected in the Database target

Both CDBs and PDBs are discovered in Enterprise Manager. They must be shown when clicking in the

Targets > Databases and expand the discovered database. If PDBs are not shown for a database:

Go to the database Target > Target Setup > Monitoring Configuration

At the bottom, click on “Sync Pluggable Databases”

B. Verify the Listener target status

Login in the Enterprise Manager OMS Console and:

Go to Targets > All Targets

Click on Listener

Verify that both ASM and database listener discovered shows the correct status.

If there is any mismatch, go to the listener target page in the EM and

Click in Oracle Listener > Target Setup > Monitoring Configuration and verify that the values

are correct: private machine name is used for monitor the listener, Oracle Home is correct,

listener.ora location is correct, etc.

C. Verify Node Managers discovery and status

Node Manager is automatically discovered by EM when its domain is added. In disaster recovery topologies,

it can happen that, instead of 4 node managers (2 in Site1, 2 in Site2), only 2 node managers are

discovered. Enterprise Manager uses the listen address set in the WebLogic domain machines to identify the

node manager target and to find its associated servers. If the listen address set for the machines is the same

in primary and standby WebLogic domains, the EM is not able to differentiate between the primary and

standby node managers. Refer to bug 2892181025 for details.

This problem, however, has no impact on the Site Guard operations because the EM node manager targets

are not directly used in Site Guard operations.

Additionally, Node Manager targets cane be shown as “down” while being properly running. This can be

caused by bug 2551312026. Agent uses the WebLogic default trust keystore to check the Node Manager

status instead of using its own trust keystore. This forces to have read permissions for the user running the

agent in the WebLogic default keystore. But this workaround is not recommended for SOACS environments,

because the file permissions of the WebLogic keystore can be reset to its original value when the machine is

restarted. The recommended workaround for SOACS is to configure the agent to use its own keystore when

checking the nodemanager status. To do this edit the file <agent_home>/.../scripts/weblogic/nmresponse.pl

and add the agent keystore to the WLST_PROPERTIES variable:

Original:

$ENV{'WLST_PROPERTIES'} = '-Dwlst.offline.log=disable';

Updated:

25 See Bug 28921810 - NodeManager discovery not working ok for FMW/SOACS Disaster Recovery domains 26 Bug 25513120 - NodeManager status only uses DemoTrust.jks instead of custom trust or AgentTrust


$ENV{'WLST_PROPERTIES'} = '-Dwlst.offline.log=disable -

Dweblogic.security.SSL.trustedCAKeyStore=<agent_home>/agent_inst/sysman/config/montr

ust/AgentTrust.jks';

SITE GUARD CONFIGURATION

The steps described in this section are based on the Site Guard Administrator's Guide for Enterprise Manager Cloud

Control version 13.3. Refer to the documentation for more details.

1. Creating Primary and Standby Site Systems

A disaster recovery site managed by Oracle Site Guard is modeled as a Generic System target type in Oracle

Enterprise Manager. Follow these steps to create a Generic System for the Site 1:

A. Login in EM OMS Console

B. Go to Targets > Systems

C. Click Add > Generic System

D. Generic System: General Screen

E. Enter a Name for the System. For example: SITE1_soacsdr.

F. You can optionally add system properties (Department, Line of Business, Location, etc.)

G. Add members to the system. For the primary site add:

The primary WebLogic Domain target

The primary Database Instance target

It is not required to add explicitly any other components like hosts, node managers, etc.

NOTE: do NOT add the database system itself. The Data Guard system that is part of will be added and it will

include primary and standby databases.

H. Click Next

I. Generic System: Define Associations. You can add the db as a key member. Click Next.

J. Generic System: Availability Criteria. Leave defaults and click Next.

K. Generic System: Charts Screen. You can leave defaults and click Finish.

Repeat same steps to create the standby site system (example: SITE2_soacsdr).

2. Creating named credentials

You must create named credentials for the targets associated with Oracle Site Guard: hosts, Oracle Node

Managers, Oracle WebLogic Servers and Oracle Databases. This table summarizes the minimum named

credentials required for managing SOACS DR with Oracle Site Guard:

CREDENTIAL NAME AUTHENTICATION

TARGET TYPE

CREDENTIAL

TYPE

TARGET

TYPE TARGET

TARGET

USERNAME

SOACSDR_SITE1_SOACSDBA_HOST_ORACLE Host SSH Key Cred. Host Site1 db

host

opc (with sudo

to oracle)

SOACSDR_SITE2_SOACSDBB_HOST_ORACLE Host SSH Key Cred. Host Site2 db

host

opc (with sudo

to oracle)

SOACSDR_SITE1_WLS_HOSTS_ORACLE Host SSH Key Cred. n/a (global) n/a (global) opc (with sudo

to oracle)

https://docs.oracle.com/cd/cloud-control-13.3/GUARD/toc.htm


SOACSDR_SITE2_WLS_HOSTS_ORACLE Host SSH Key Cred. n/a (global) n/a (global) opc (with sudo

to oracle)

SOACSDR_DOMAIN_NODEM Oracle WebLogic

Domain

Node Manager

Cred.

n/a (global) n/a (global) weblogic

SOACSDR_DOMAIN_WEBLOGIC Oracle WebLogic

Domain

WebLogic

Admin Cred.


SOACSDR_ADMIN_WEBLOGIC Oracle WebLogic

Server

Oracle

WebLogic

Cred.


SOACSDR_SITE1_SOACSDBA_SYS Database Instance Database Cred. Database

Instance

Primary

Database

sys as sysdba

SOACSDR_SITE2_SOACSDBB_SYS Database Instance Database Cred. Database

Instance

Standby

Database

sys as sysdba

(OPTIONAL) ANY_AUX_HOST_USER

Samples:

OMS_HOST_ROOT

OMS_HOST_ORACLE

Host SSH Key Cred. Host Any required

user for

executing

scripts in an

auxiliary host.

Sample: opc

(with sudo to

root) for OMS

host

Table 1 List of required Named Credentials

NOTE: Node Manager Credentials, WebLogic Administrator Credentials and Oracle WebLogic credentials are the

same for the primary and standby WebLogic domain (they must be the same in a SOACS DR environment), so

using global credentials instead of targeted credentials simplifies the configuration.

NOTE: In Cloud, the SSH authentication is done using SSH keys. SSH key with passphrases are NOT currently

supported in Enterprise Manager. In case that the SSH keys used by your cloud instances use passphrase, you

need to add to them a SSH key that doesn’t use a passphrase. Refer to the Cloud Documentation to add a new

SSH27. More than one SSH key can be configured for a cloud instance.

A. Configure Privilege Delegation

Only the opc user can login directly in a SOACS and DB hosts. It has sudo privileges so it can sudo to any

user (oracle, root). To allow Enterprise Manager to login as oracle user in the SOACS hosts, privilege

delegation feature must be configured for the hosts.

1. Login in Enterprise Manager OMS Console

2. Go to Setup > Security > Privilege Delegation Setting

3. Select the SOACS host you want to configure and click “Edit”

4. Select Sudo

5. Enter sudo command “/usr/bin/sudo -u %RUNAS% %COMMAND%”

27 For Paas Using Oracle Cloud Infrastructure Compute Classic > Adding an SSH Public Key

For DB Systems, go to OCI Console, navigate to DB System and click on “Add SSH Keys” Oracle Cloud Infrastructure Documentation > Managing Key Pairs on Linux Instances

https://docs.oracle.com/en/cloud/iaas/compute-iaas-cloud/stcsg/adding-ssh-public-key.html

https://docs.cloud.oracle.com/iaas/Content/Compute/Tasks/managingkeypairs.htm


6. Click Save

7. Repeat for all the SOACS and DB hosts in primary and standby sites, and for the OMS host (it

will be configured later as an auxiliary host for executing some post scripts).

B. Create host credentials for SOACS and DB hosts

1. In OMS console, Setup > Security > Named Credentials.

2. Click Create, and complete with the following:

Credential Name Set the Credential Name. See table Table 1 List of required Named

Credentials

Authenticating Target Type Host

Credential Type SSH Key Credentials

Scope: When creating a credential for SOACS use Global.

When creating a credential for DB host, you can target to the specific DB host.

UserName Use opc. It is the only user tan can login directly

SSH Private Key Upload or paste the private private SSH key for the SOACS instance

SSH Public Key Upload or paste the public private SSH key for the SOACS instance

Run Privilege Select “Sudo” privilege and set the user oracle in “Run as”

3. Test and Save. Select a SOACS host to test the credential before saving.28

4. Repeat the same to create the credential for the standby hosts. (Example:

SOACSDR_SITE2_WLS_HOSTS_ORACLE)

C. Create host credentials for Node Manager targets


2. Click Create and complete with the following, then click Save:

Credential Name Set the Credential Name. This credential will be valid for all the node

managers in the primary and standby WebLogic domains. Example:

SOACSDR_DOMAIN_NODEM

Authenticating Target Type Oracle WebLogic Domain

Credential Type Node Manager Credentials

Scope: Global

UserName Use weblogic (this is the default node manager’s user)

NodeManager Password Enter Node Manager’s password

D. Create credentials for WebLogic Domain targets



28 If you receive the error “PDP execution may have failed :bash: <AGENT_BASE>/agent_13.3.0.0.0/sbin/nmgsshe: Permission denied” when testing

the credential, verify that agent home and parent folders have read and execute permissions for others.


Credential Name Set the Credential Name. This credential will be valid for all the node

managers in the primary and standby WebLogic domains. Example:

SOACSDR_DOMAIN_WEBLOGIC

Authenticating Target Type Oracle WebLogic Domain

Credential Type WebLogic Administrator Credentials

Scope: Global. The same will be used for primary and standby WebLogic domains.

UserName Use weblogic

NodeManager Password Enter weblogic’s password

E. Create credentials for WebLogic Server targets



Credential Name Set the Credential Name. This credential will be valid for primary and standby

admin

servers. Example: SOACSDR_ADMIN_WEBLOGIC

Authenticating Target Type Oracle WebLogic Server

Credential Type Oracle WebLogic Credentials

Scope Global. The same will be used for primary and standby admin servers

UserName Use weblogic

NodeManager Password Enter weblogic’s password

F. Create credentials for Database Instance targets


2. Click Create and complete with the following:

Credential Name Set the Credential Name. This credential will be used for the database or standby

Example: SOACSDR_SITE1_SOACSDBA_SYS

Authenticating Target Type Database Instance

Credential Type Database Credentials

Scope Target

Target Name Select the primary database target

UserName Use sys

Password Enter sys password

Role SYSDBA

3. Test and Save

4. Repeat the same to create the credential for the standby database target.

G. Create host credentials for auxiliary hosts

Auxiliary hosts are host that are not part of the primary and standby systems but that are used to run

additional scripts during switchover/failover procedures. As an example in this approach, the OMS

host will be used as auxiliary host to run some post-scripts. Create credentials for it:



2. Click Create, and complete with the following:

Credential Name Set the Credential Name. See table Table 1 List of required Named

Credentials. Example: OMS_HOST_ORACLE for oms host with oracle user,

OMS_HOST_ROOT for oms host with root user.

Authenticating Target Type Host

Credential Type SSH Key Credentials

Scope: You can target to the specific aux host. (Sample: oms host)

UserName Use opc. It is the only user tan can login directly.

SSH Private Key Upload or paste the private private SSH key for the auxiliary host (OMS host)

SSH Public Key Upload or paste the public private SSH key for the auxiliary host (OMS host)

Run Privilege Select “sudo” privilege and set the required user (root, or oracle) in “Run as”

3. Test and Save.

4. Repeat the same to create additional credentials for the host or for any other auxiliary host.

3. Configuring preferred credentials

Once the named credentials have been created, they can be assigned to the targets as the preferred credentials.

This approach is recommended to simplify the Site Guard configuration. Follow these steps to configure the

preferred credentials for a target:

A. Login in EM, go to Setup > Security > Preferred Credentials

B. Select a target type, and click “Manage Preferred Credentials”.

C. Set the preferred credentials for each target credential, by clicking each row and selecting the

appropriate named credential created in the previous step.

D. Do this for the following targets:

Primary and Standby Database Instance (at minimum, sysdba credentials, database hosts

credentials)

Primary and Standby WebLogic Domain (weblogic administrator credentials, hosts credentials)

Primary and Standby Admin Servers (Oracle WebLogic Administration Credentials, Host

Credentials)

Primary and Standby hosts. Only “Normal Host credentials” are required for SOACS DR.

Any other aux host. OMS host will be configured later as an example auxiliary host for running

some post scripts. Set normal and privilege preferred credentials for this host.

4. Defining Site Roles

Once a disaster recovery site managed by Oracle Site Guard has been modeled as a Generic System target type in

Oracle Enterprise Manager, then you designate it as a primary sites or a standby site. Once the sites have been

created, it is needed to assign roles to the sites:

A. Login in EM, go to Targets > Systems

B. Click on the name of the primary site system

C. On the system's home page, from the Generic System menu, select Site Guard > Configure.

D. On the General tab, click in Create

E. On the General tab, in the Standby System(s) section, click Add.


F. Choose the standby system, and click Select.

G. Click Save and OK to confirm the action. Site Guard saves the standby system configuration.

H. Verify that the roles have been assigned.

In the primary Site system, in Oracle Site Guard Configuration, General Tab, you must see:

Current Role Primary

In the secondary Site system, in Oracle Site Guard Configuration, General Tab, you must see:

Current Role Standby

5. Configuring auxiliary hosts

You can configure one or more hosts managed by Oracle Enterprise Manager as an auxiliary host to a site. These

hosts are not part of the system but are used to run Pre Scripts, Post Scripts, or Storage Scripts on a site. To add an

auxiliary host to a system:

emcli add_siteguard_aux_hosts -system_name="system_name" -host_name="host_name"

The following auxiliary hosts are required for a SOACS DR Site Guard configuration:

Auxiliary hosts for running dbfscopy.sh sync script in the other site

This is required when you are propagating configuration changes between primary and standby using the

dbfscopy.sh script explained in section b) Using DBFS for configuration propagation.

This script runs first in primary soahost1, to synchronize changes from primary WebLogic domain configuration

to the dbfs filesystem. Then it runs in standby soahost1, and synchronizes changes from dbfs filesystem to

standby WebLogic domain conf.

It is a good practice to run the script before any switchover operation when possible, to ensure than

configuration in the new standby is up-to-date.

To modeling this in Site Guard, it is required to define SOACS Site2 host1 as an auxiliary host for Site1 and

define SOACS Site1 host1 as an auxiliary host for Site2. Perform the following steps:

A. Login in Enterprise Manager host and login in emcli:

[oracle@emcc bin]$ cd $EM_HOME/middleware/bin


Enter password :

Login successful



B. Use emcli to add the SOACS Site 2 host1 as an auxiliary host for Site1. Use the hostname as it is

registered in the EM. Example:

[oracle@emcc bin]$ ./emcli add_siteguard_aux_hosts -

system_name="SITE1_soacsdr" -host_name="soacsdroci-wls-1-

public.site2.example.com"

Auxiliary host(s) added to system SITE1_soacsdr

C. Use emcli to add the SOACS Site 1 host1 as an auxiliary host for Site2. Use the hostname as it is

registered in the EM. Example:

[oracle@emcc bin]$ ./emcli add_siteguard_aux_hosts -

system_name="SITE2_soacsdr" -host_name="soacsdroci-wls-1-

public.site1.example.com"


Auxiliary host for running DNS or /etc/hosts change


The public name used by SOA consumers (defined as the frontend in the WebLogic domains) must always

point to the public IP used by OTD in the site that has the primary role.

So, at the end of each switchover, it is required to perform a DNS push or alter the file host resolution to point

that address to the public IP used by OTD in the new primary site.

NOTE: The name resolution does not change for SOACS hosts: they always point to its own OTD. The change must

be effective for soa consumers.

You can use the Site Guard to run a custom script that performs a change in the /etc/hosts file or push a

change in the DNS server. The host or hosts where the script will run must be discovered in the Enterprise

Manager and added as auxiliary hosts to the systems.

As an example in this document, the EM host (i.e: emcc.marketplace.com) will run scripts that perform frontend

name IP changes, so it is added as auxiliary host to the sites:

[oracle@maaemsg bin]$ ./emcli add_siteguard_aux_hosts -system_name="SITE1_soacsdr" -

host_name="emcc.marketplace.com"


[oracle@maaemsg bin]$ ./emcli add_siteguard_aux_hosts -system_name="SITE2_soacsdr" -

host_name="emcc.marketplace.com"


6. Credential associations

Credentials are associated with targets and used by Oracle Site Guard operation plans when they are executed.

These associations must be configured for primary and standby systems:

A. Login to Enterprise Manager

B. From the Targets menu, click Systems

C. On the Systems page, click the name of the system for which you want to configure credential

associations.

D. On the system's home page, from the Generic System menu, select Site Guard > Configure.

E. Click the Credentials tab. Now associate the different types of credentials.

F. Normal Host Credentials section, click Add, select All and check Preferred, Normal Host

Credentials. Click Save.

G. Privileged Host Credentials are not required for the SOACS and DB hosts, because Site Guard

scripts are executed with oracle user. But privileged credential may be required for auxiliary hosts that

run scripts with root user (example, to update the /etc/hosts file). If this is the case, add the privileged

host credential to that auxiliary host (EM host is used as an example auxiliary host in this approach).

H. Oracle Node Manager Credentials section, click Add, select All and Named. Select the

NodeManager credential created before. Example: SOACSDR_DOMAIN_NODEM. Click Save

I. WebLogic Administration Credentials section, click Add, select All and Preferred. Click Save

J. SYSDBA Database Credentials section, click Add, select All and Preferred, “SYSDBA Database

Credentials”. Click Save

7. Configuring required scripts

Oracle Site Guard provides a mechanism for you to configure scripts for managing disaster recovery operations.

Different kind of scripts can be defined for Site Guard:


To configure these scripts for SOACS DR:

A. Login in EM, and go to Targets > Systems

B. Click in the System where you want to configure the scripts

C. Click in Site Guard > Configure > Pre/Post scripts tab

D. Add Pre Scripts to Site 1 system:

Insert the Script Path to the dbfscopy.sh script. Example: /u01/install/dbfscopy.sh

Select Script Type: Global-PreScript

Select Operation: Switchover

Select target host soahost1 in Site1 and soahost1 in Site2 (which is an aux host for this site)

Select the normal preferred credentials for the hosts

NOTE: By default the dbfscopy.sh prompts for the sysdba password. Customer can customize it to take the

password as an argument in order to be executed from EM Site Guard.

E. Add Pre Scripts to Site2 system:

Insert the path to the dbfscopy.sh script. Example: /u01/install/dbfscopy.sh

Select Script Type: Global-PreScript


Select Role: Primary

29 Sample script change_frontend_ip_from_SITE1_to_SITE2.sh:

# Script to change /etc/hosts file entry soacsdrfrontend.example.com ip (entry must exist in the /etc/host file)

# from SITE1 OTD ip: 111.111.111.10

# to SITE2 OTD ip: 222.222.222.20

sed -i 's/111.111.111.10/222.222.222.20/g' /etc/hosts

SITEGUARD SCRIPTS TYPES

PreCheck, Mount and unmount or Storage Not required for SOACS DR

Pre Scripts The scripts that perform the config synchronization can be run as pre scripts. Example:

/u01/install/dbfscopy.sh in soa host 1 of Site1, run as oracle.

/u01/install/dbfscopy.sh in soa host 1 of Site2, run as oracle.

Post Scripts Following scripts will be run as post scripts in this example:

Post script that update the frontend virtual hostname IP in the /etc/hosts after a

switchover/failover. Example:

/root/scripts/change_frontend_ip_from_SITE1_to_SITE2.sh29

/root/scripts/change_frontend_ip_from_SITE2_to_SITE1.sh

Post script that update the frontend virtual hostname IP in DNS after a

switchover/failover. For scenarios where DNS is used for the external frontend

resolution (Oracle Cloud DNS, commercial DNS, etc.) appropriate API can be used

to push the change. An example that push this change in an Oracle Cloud DNS

can be found here.

Post script to check the soa-infra url after the complete switchover/failover, to verify

that the switchover has been successful. Sample script can be found here.

https://www.oracle.com/technetwork/database/availability/downloads/soacsdr-siteguard-script-updatedns-5592969.zip

https://www.oracle.com/technetwork/database/availability/downloads/check-soainfra-5592914.zip


Select target host soahost1 in Site2 and soahost1 in Site1 (which is an aux host for this site)

Select the normal preferred credentials for the hosts

F. Add Post Scripts to Site1 system, to be executed post the switchover from Site2 to Site1:

Insert the path to the script that changes the dns name to the Site 1 OTD public IP.

Select Script Type: Post-Script (Global-PostScript is also valid)


Select Role: Standby. We want this to be executed at the end of the operation

plan when switching to Site1.

Select the aux host where host file is updated, OMS host in this case.

Select the privileged preferred credential for the host (only root can modify hosts file).

G. To create the same post script for the failover operation from Site2 to Site1.

Select the post script created in previous step and click on “Add Like”

Change operation to “Failover”

Click Save

H. Add Post Scripts to Site2 system, to be executed post the switchover from Site1 to Site2:

Insert the path to the script that changes the name to the Site2 OTD public IP.

Select Script Type: Post-Script (Global-PostScript is also valid)

Select Operation : Switchover

Select Role: Standby. We want this to be executed at the end of the operation

plan when switching to Site2.

Select the aux host where host file is updated, OMS host in this case.

Select the privileged preferred credential for the host (only root can modify hosts file)

I. To create the same post script for the failover from Site1 to Site2.

Select the post script created in previous step and click on “Add Like”

Change operation to “Failover”

Click Save

J. Repeat to add any other post-script (for example, the soa-infra url verification script).

8. Configuring apply and transport lag thresholds

Site Guard verifies the apply and transport lag of the Data Guard during the prechecks and the switchover. By

default, if the value is different than cero, the precheck fails and the switchover is not performed. You can define a

threshold value to allow a few seconds so the check is more permissive. Example to set the thresholds to 10

seconds:

A. Connect via SSH to the OMS host

B. Login to emcli:



Enter password :

Login successful



C. Set the threshold to 10 seconds in both sites:


[oracle@emcc]$ ./emcli configure_siteguard_lag -system_name=SITE1_soacsdr -

property_name=apply_lag -value=10


property_name=apply_lag -value=10


property_name=transport_lag -value=10


property_name=transport_lag -value=10

9. Creating switchover and failover operation plans

An operation plan describes the flow of execution that Oracle Site Guard performs in a disaster recovery operation.

It consists of (ordered) actions that can be executed in series or in parallel. Oracle Site Guard creates a default

version of the operation plan based on the site topology and the Oracle Site Guard configuration. You can use this

default operation plan or customize it depending on your configuration.

These operations plans are recommended for SOACS DR:

PLAN DESCRIPTION

SWITCHOVER_SITE1_TO_SITE2_WITH_SYNC Switchover from Site1 to Site2 that performs a config synchronization based on

dbfscopy.sh script

SWITCHOVER_SITE2_TO_SITE1_WITH_SYNC Switchover from Site2 to Site1 that performs a config synchronization based on

dbfscopy.sh script

SWITCHOVER_SITE1_TO_SITE2 Switchover from Site1 to Site2, without config sync (RTO is reduced)

SWITCHOVER_SITE2_TO_SITE1 Switchover from Site1 to Site2, without config sync (RTO is reduced)

FAILOVER_SITE1_TO_SITE2 Failover from Site1 to Site2. No dbfscopy.sh synchronization is performed, as a

failover is an unplanned event when the primary site becomes unavailable.

FAILOVER_SITE2_TO_SITE1 Failover from Site2 to Site1. No dbfscopy.sh synchronization is performed, as a

failover is an unplanned event when the primary site becomes unavailable.

A. Create SWITCHOVER_SITE1_TO_SITE2_WITH_SYNC

Operation plan for performing a switchover from SITE to SITE 2, that performs a config synchronization

based on dbfscopy.sh script before the switchover.

1. Login in EM, go to Target > Systems

2. Click in Site1 System

3. Go to Site Guard > Operations.

4. Click Create. The Create New Operation Plan dialog is displayed.

5. Enter a name for the plan. Example: SWITCHOVER_SITE1_TO_SITE2_WITH_SYNC

6. Select Operation Type: Switchover

7. Select the other site (Site2) as the standby system

8. Click Save

9. The default created plan needs to be customized. Edit the plan and:

10. Verify that sync pre-scripts dbfscopy.sh are run in Serial mode (by default they run in parallel)

11. Verify that the first execution of the dbfscopy.sh in the Site that is the primary in this plan. For

example, when the plan is defined to switchover from Site1 to Site2, the first execution of the

dbfscopy.sh must run in Site1 soahost1.


12. Disable node manager stop/start steps. They are not required and skipping them reduces the

RTO.

NOTE: It is recommended to disable non-required start/stop steps related with components instead of deleting them.

Topology Prechecks can fail when running the plan and warning “This operation plan is out of sync with current

topology of the system.” when the start/stop steps are deleted for some components in the system. However, there

are no warnings when the start/stop steps are disabled.30

13. Verify that the post-script that checks the soa-infra url test is done after the dns or hosts file

update.

14. Save the changes of the operation plan

B. Create SWITCHOVER_SITE2_TO_SITE1_WITH_SYNC

Follow the same steps than in the previous but in Site2 system.

Select the Site1 as the standby and verify that the pre-script dbfscopy.sh runs first in Site2 soacs host1.

C. Create SWITCHOVER_SITE1_TO_SITE2

Select the plan SWITCHOVER_SITE1_TO_SITE2_WITH_SYNC and click in “Create like”

Edit it and delete the Global Pre-scripts steps to skip dbfscopy.sh script executions.

D. Create SWITCHOVER_SITE2_TO_SITE1

Select SWITCHOVER_SITE2_TO_SITE1_WITH_SYNC and click in “Create like”

Edit it and delete the Global Pre-scripts steps to skip dbfscopy.sh script executions.

E. Create FAILOVER_SITE1_TO_SITE2

Operation plan for performing a failover from SITE1 to SITE 2. No dbfscopy.sh synchronization is performed,

as a failover is an unplanned event when the primary site becomes unavailable.


2. Click in Site1 System

3. Go to Site Guard > Operations.

4. Click Create. The Create New Operation Plan dialog is displayed.

5. Enter a name for the plan. Example: FAILOVER_SITE1_TO_SITE2

6. Select Operation Type: Failover

7. Select the other site (Site2) as the standby system

8. Click Save

9. The created plan needs to be customized. Select the plan an click on “Edit”

10. Verify that pre scripts for dbfscopy.sh have not been included in the plan

11. Disable node manager start steps. They are not required and skipping them reduces the RTO.

12. Verify that the post-scripts are included in the correct order and reorder if needed (soa-infra

check must be the last one)

13. Save the changes.

30 Bug 29005772 - TOPOLOGY PRECHECKS FAIL IN SG RUN PRECHECKS WHEN SOME COMPONENT START/STOP STEPS ARE DELETED


F. Create FAILOVER_SITE2_TO_SITE1

Follow the same steps than in the previous but in Site2 system. Select the Site1 as the standby.

NOTE: These operation plans are valid also for cases where the AdminServer is already started in the standby. The

status of the Admin is checked before trying to start it and skipped the start if it is already RUNNING.


PERFORMING A SWITCHOVER WITH SITE GUARD

Once the operations plans are created you can perform the switchover of the complete SOACS DR Site with

Enterprise Manager Site Guard. To execute a switchover operation using OMS Console:


2. Click in the current primary Site System

3. Go to Site Guard > Operations

4. Select the operation plan you want to execute.

5. Click “Execute Operation”

Alternatively, the operation plans can be submitted using EMCLI:

1. SSH to OMS host with user oracle.

2. Login to emcli



Enter password :

Login successful



3. Submit the operation plan

emcli submit_operation_plan –name="name_of_operation_plan"

Site Guard will orchestrate all the steps defined in the switchover plan per the following:

Oracle Site Guard runs the precheck utility to perform some validations: it checks the agent status in the

involved hosts, checks if any targets have been added or deleted from the generic systems, runs Oracle

Data Guard Broker prechecks to ascertain whether the Database is ready for role reversal.

It executes the pre-scripts: if the plan includes the dbfscopy.sh as defined in the previous section, it will run

dbfscopy.sh in the SOACS host1 of the primary site and then it will run dbfscopy.sh in the SOACS host1 of

the standby site to synchronize the WebLogic domain configuration.

The SOACS WebLogic domain in the primary Site will be shut down: first, the WebLogic managed servers

(in parallel) and then the WebLogic Administration Server.

Oracle Site Guard performs the database switchover from primary database to standby database using

Data Guard broker.

Once the database switchover is done, the SOACS WebLogic domain in the standby Site is started: first

the WebLogic Administration Server and then the WebLogic managed servers (in parallel).

Any defined post-scripts will be run: to change the frontend resolution or to verify the soa-infra status.

If everything is successful, the roles of the sites are updated in the Site Guard metadata schema.

The progress of the operation plan can be monitored in the OMS Console, in System > Site Guard > Operations >

Operation Activities

Details of each steps are provided (timing, actions, result, etc.) and any failed step can be retried.

Once finished, the sites’ role change can be verified using the System > Site Guard > Configure > General screen


PERFORMING A FAILOVER WITH SITE GUARD

You can perform a failover using Site Guard as explained in the previous section, by executing the failover operation

plan with OMS Console or EMCLI. Site Guard will orchestrate the steps for the SOACS DR failover:

Oracle Site Guard runs the precheck utility to perform validations before executing the failover (it checks

the agent status in the involved hosts and runs Oracle Data Guard Broker prechecks)

No pre-scripts are executed. The script dbfscopy.sh is not defined for failover operations. A failover is an

unplanned event that happens when the primary site becomes unavailable so configuration

synchronization is not expected.

During failover shutdown of the SOACS WebLogic domain in the primary Site is skipped by default

although it can be enabled manually if required).

Oracle Site Guard performs the database failover from primary database to standby database using Data

Guard broker.

Once the database failover is completed, the SOACS WebLogic domain in the standby Site is started: first

the WebLogic Administration Server and then the WebLogic managed servers in parallel.

The post-scripts are executed to change the frontend resolution or to verify the soa-infra status.

If everything is successful, the roles of the sites are updated in the Site Guard metadata schema.

After a failover operation, the pertinent actions need to be performed to bring the original primary site back to a

healthy status: solve the problem that forced the failover, reinstantiate the database, etc. Then a switchback can be

performed with Site Guard.


Oracle Corporation, World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone: +1.650.506.7000

Redwood Shores, CA 94065, USA Fax: +1.650.506.7200

Copyright © 2017 Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the

contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0615 White Paper Title: SOA Cloud Service Disaster Recovery - Production and DR in the Cloud April 2020

C O N N E C T W I T H U S

blogs.oracle.com/oracle

facebook.com/oracle

twitter.com/oracle

oracle.com


Documents

SOA Cloud Services Disaster Recovery · Oracle Service-Oriented Architecture Cloud Service (SOACS) is a Platform as a Service (PaaS) computing platform solution for running Oracle