23
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12 http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 1/23 Troubleshooting RAC RapidClone issues with Oracle Applications R12 [ID 1303962.1]  Modified 15-SEP-2011 Type TROUBLESHOOTING Status PUBLISHED In this Document  Purpose  This note only covers the Database Tier(s).  Last Review Date  Instructions for the Reader  Troubleshooting Details  Section 1: Cloning log files required by Support  Section 2: Preparation - How it works  Section 3: Configuration - How it works  Section 4: Known Issues  a) Final DB Startup fails if Flash Recovery is enabled  b) On Secondary nodes, adclonectx.pl fails with Instance Number used by host  c) RAC to RAC : Inventory Registration still references the source cluster nodes  d) RAC to non-RAC : Inventory Registration still contains the source cluster nodes  References Applies to: Oracle Applications Manager - Version: 12.0.0 to 12.1.3 - Release: 12 to 12.1 Information in this document applies to any platform. Purpose This troubleshooting note is intended as a guide to help determine the cause of some problems that may arise during the Database Tier phase only of cloning E-Business Suite Release 12 with Real Application Clusters. This Note is NOT a replacement for Document 559518.1 - Cloning Oracle E-Business Suite Release 12 RAC-Enabled Systems with Rapid Clone. That document must be followed when performing a clone of Oracle Applications R12 running on RAC. The idea is to provide a detailed explanation into what each step is trying to achieve making it easier to identify which part of the clone has failed and why. Locations of log files are listed in Section 1, and are explained in each section. These should be uploaded when reporting a cloning issue to Oracle Support. Note: This Note does not yet cover 11.2.0 Grid specific functionality such as the SCAN listener, but this document will be updated regularly and will be included in a future update. This note only covers the Database Tier(s). For the Application Tier, refer to Document 603104.1 - Troubleshooting RapidClone issues with Oracle Applications R12. Last Review Date March 15, 2011 Instructions for the Reader A Troubleshooting Guide is provided to assist in debugging a specific issue. When possible, diagnostic tools are included in the document to assist in troubleshooting. Rate this document https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT... 1 of 23 12/18/2012 6:00 PM

Troubleshooting RAC RapidClone issues with Oracle Application R12

Embed Size (px)

Citation preview

Page 1: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 1/23

Troubleshooting RAC RapidClone issues with Oracle Applications R12 [ID 1303962.1]

  Modified 15-SEP-2011 Type TROUBLESHOOTING Status PUBLISHED

In this Document  Purpose  This note only covers the Database Tier(s).  Last Review Date  Instructions for the Reader   Troubleshooting Details  Section 1: Cloning log files required by Support  Section 2: Preparation - How it works  Section 3: Configuration - How it works  Section 4: Known Issues  a) Final DB Startup fails i f Flash Recovery is enabled

  b) On Secondary nodes, adclonectx.pl fails with Instance Number n used by host  c) RAC to RAC : Inventory Registration still references the source cluster nodes  d) RAC to non-RAC : Inventory Registration still contains the source cluster nodes  References

Applies to:

Oracle Applications Manager - Version: 12.0.0 to 12.1.3 - Release: 12 to 12.1

Information in this document applies to any platform.

PurposeThis troubleshooting note is intended as a guide to help determine the cause of some problems that may arise during the Database Tier phase only of cloning

E-Business Suite Release 12 with Real Application Clusters.

This Note is NOT a replacement for Document 559518.1 - Cloning Oracle E-Business Suite Release 12 RAC-Enabled Systems with Rapid Clone.

That document must be followed when performing a clone of Oracle Applications R12 running on RAC.

The idea is to provide a detailed explanation into what each step is trying to achieve making it easier to identify which part of the clone has failed and why.

Locations of log files are listed in Section 1, and are explained in each section. These should be uploaded when reporting a cloning issue to Oracle Support.

Note: This Note does not yet cover 11.2.0 Grid specific functionality such as the SCAN listener, but this document will be updated regularly and will be included in a future update.

This note only covers the Database Tier(s).

For the Application Tier, refer to Document 603104.1 - Troubleshooting RapidClone issues with Oracle Applications R12.

Last Review Date

March 15, 2011

Instructions for the Reader 

A Troubleshooting Guide is provided to assist in debugging a specific issue. When possible, diagnostic tools are included in the document to assist in troubleshooting.

Rate this document

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

1 of 23 12/18/2012 6:00 PM

Page 2: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 2/23

Troubleshooting Details

Section 1: Cloning log files required by Support

If the problem could not be diagnosed or resolved with the details in this document, please raise a Service Request to Oracle Support for assistance and

upload the following log files from each Database node:

Source System log files 

adpreclone.pl logfile:

$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDBTier_xxxxxx.log

adclone.pl [mode=stage] logfile:

$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDatabase_xxxxxx.log

RMAN backup script and logfile:

<clone stage>/data/template/adcldbstage-rac.rman

<clone stage>/data/out/adcldbstage-rac.log

Target System log files 

Create Context file logfile:

Generated in the working directory when adclonectx.pl was run: 

$ORACLE_HOME/appsutil/clone/bin/CloneContext_xxxxxx.log

Port Pool logfile:

$ORACLE_HOME/appsutil/out/portpool.lst

Oracle Home configuration logfiles:

Primary node: 

$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_xxxxxx.log

Secondary node(s): 

$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTechStack_xxxxxx.log

Relink logfile:

$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/make_<timestamp>.log

Inventory Registration and configuration logfiles:

$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/log/ohclone.log

<Global Inventory>/logs/silentInstall<timestamp>..log

<Global Inventory>/logs/oraInstall<timestamp>..out

<Global Inventory>/logs/cloneActions<timestamp>..log

RMAN Restore scripts and logfiles:

$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac1.rman

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

2 of 23 12/18/2012 6:00 PM

Page 3: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 3/23

Page 4: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 4/23

Page 5: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 5/23

https://support oracle com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT

Page 6: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 6/23

SQL>select 'configure device type disk parallelism 5 backup type to backupset;' \

from dual;

SQL>select 'configure maxsetsize to '||round((max(bytes)/1024/1024) + 200)||'m;' \

from dba_data_files;

SQL>select 'backup as backupset tag ''RapidClone_RAC'' database format \''<your location>/data/stage/%U'';' from dual;

SQL>select 'backup as backupset tag ''RapidClone_RAC'' archivelog all format \

''<your location>/data/stage/%U'';' from dual;

SQL>select 'backup tag ''RapidClone_RAC'' as copy current controlfile format \

''<your location>/data/stage/backup_controlfile.ctl'';' from dual;

SQL>select 'exit' from dual;

SQL>spool off;

SQL>exit;

The adcldbstage-rac.rman script generates the logfile called :

[Stage Directory]/data/out/adcldbstage-rac.log

Create Oracle Home Archive 

The third and final preparation phase is to create an archive of the ORACLE_HOME:

$ cd $ORACLE_HOME/..

$ tar -cvzf [Stage Directory]/rac_db_oh.tgz [DATABASE TOP LEVEL DIRECTORY]

As this is a manual step there is no log generated unless specifically requested when running the tar command.

Oracle does recommended using data integrity utilities such asmd5sum, sha1sum, or cksum to validate the file sum both before and after transfer to the target system.

Section 3: Configuration - How it works

About Non-RAC clone steps 

On the target host system the context file creation should be done as in the case of Single Instance cloning.

All other primary target restore tasks remain the same from the RAC-to-RAC clone in the case of Single Instance Restore.

Target Instance is RAC (y/n) [y] : [Enter n]

As the target non-RAC is cloned from a RAC enabled system, the interview question above pre-selects a default value of being a RAC Instance.

Be certain to select "N" for the above question.

By creating a context file with without RAC attributes present, Rapid Clone will configure and convert the RDBMS Technology Stack and its binaries on the target system. This to ensure a Single Instance restore can

be performed.

The Rapid Clone commands to restore the Oracle Home and the database on the target system remain the same whether the target is to be RAC or Single Instance.

Extract Oracle Home Archive 

The first phase on the Target is to extract the Oracle Home from the archive created from the Source:

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

6 of 23 12/18/2012 6:00 PM

https://support oracle com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT

Page 7: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 7/23

$ tar -xvzf rac_db_oh.tgz

As mentioned in the previous section, there is no log generated unless specifically requested when running the tar command,

but data integrity utilities should be used, such as md5sum , sha1sum , or cksum to validate the file sum both before and after transfer to the target system.

Create pairsfile.txt File for Primary Node 

The pairsfile needs to be the same across all nodes with the only exception being the value for the variable “ s_undo_tablespace ”

s_undo_tablespace=[UNDOTBS1 for Initial Node]

s_dbClusterInst=[Total number of Instances in a cluster e.g. 2]

s_db_oh=[Location of new ORACLE_HOME]

Create Primary Node context file 

This step is achieved using the adclonectx.pl command. This will clone an existing context file and update it with the configuration details entered via prompts.

It will give the option to validate the inputs, such as check port availability.

Several prompts requesting information about the target Primary node will be displayed at this point:

...

...

Provide information for the initial RAC node:

Host name [<source host>] :

Virtual Host name [null] :

Instance number [1] : 1

Private interconnect name [<source host>] :

...

...

A full list of these prompts are shown in Section 5 of the main cloning Document 559518.1 - Cloning Oracle E-Business Suite Release 12 RAC-Enabled Systems with Rapid Clone 

There is a new variable introduced with the latest AutoConfig patches s_db_listener which needs to be updated manually, and is also mentioned in the main clone note as follows:

Note: If the most current AutoConfig Template patch as listed in OracleMetalink Document 387859.1 has already been applied to the source system,

it is necessary to edit the value of the target system context variable "s_db_listener" to reflect the desired name for the tns listener.

The traditionally accepted value is "LISTENER_[HOSTNAME]" but may be any valid value unique to the target host.

The main log file is generated in the working directory when adclonectx.pl was run:

$ORACLE_HOME/appsutil/clone/bin/CloneContext_xxxxxx.log

If the input validation option is selected, the port availability check will generate the following log:

$ORACLE_HOME/appsutil/out/portpool.lst

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

7 of 23 12/18/2012 6:00 PM

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

Page 8: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 8/23

Note: When cloning RAC to non-RAC, ensure the correct response of "n" is entered for the prompt:

Target Instance is RAC (y/n) [y] : n

Configure the Target Primary Oracle Home 

On the Primary Target node (RAC or non-RAC), the adclone.pl command is used to configure the Target Database home and the database. The command is:

perl adclone.pl \

java=[JDK 1.5 Location] \

component=dbTier \

mode=apply \

stage=[ORACLE_HOME]/appsutil/clone \

method=CUSTOM \

dbctxtg=[Full Path to the Target Context File] \

rmanstage=[Location of the Source RMAN dump files... i.e. RMAN_STAGE/data/stage] \

rmantgtloc=[Shared storage loc for datafiles...ASM diskgroup / NetApps NFS mount / OCFS mount point]srcdbname=[Source RAC system GLOBAL name] \

pwd=[APPS Password] \

showProgressode

Note one of the values requested is the [Source RAC system GLOBAL name] . The cloning process used this value when restoring the database files.

The restored database uses the same global name db_name as the source, but a different db_unique_name.

The backup files are restored and renamed on the target, and then the database is renamed using DBNEWID (NID) at the final stages.

Failure to provide the correct source srcdbname in the above command, will generate the following error:

RMAN-00571: ===========================================================RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of alter db command at 09/14/2011 09:44:17

ORA-01103: database name '<Source>' in control file is not '<Target>'

This will be explained later in this document.

The adclone.pl step which generates the log file:

$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDatabase_xxxxxx.log

The adclone.pl script will start the process, which involves the following:

Registers and configures the Oracle Home in the inventory using the script:

$ORACLE_HOME/appsutil/clone/ouicli.pl

This generates the log files :

$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/log/ohclone.log

<Global Inventory>/logs/silentInstall<timestamp>..log

<Global Inventory>/logs/oraInstall<timestamp>..out

<Global Inventory>/logs/cloneActions<timestamp>..log

Relinks the Oracle Home using script :

$ORACLE_HOME/appsutil/clone/adlnkoh.sh

p pp w yp yp

8 of 23 12/18/2012 6:00 PM

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

Page 9: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 9/23

This generates the log file:

$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/make_<timestamp>.log

Creates the Target environment and configuration files

Generates and executes a series of RMAN restore scripts

The restoration is done in stages and is explained in this section.

The scripts are generated under :

$ORACLE_HOME/appsutil/out/$CONTEXT_NAME

The scripts are:

restore-rac1.rman

restore-rac2.sql

restore-rac2.rman

restore-rac21.sql

restore-rac-logfile.sql

restore-rac3.rman

Creates a temporary initSID1.ora.tmp file used to start the Database during the restoration from the backup files.

The temporary initSID1.ora.tmp is generated via ApplyRmanDatabase.java 

This file contains some hardcoded values, and is built as follows:

db_name=<source Global DB name - srcdbname value entered during adclone.pl command>

db_unique_name=<Target Global DB name - %s_dbGlnam% value from context file>

db_files = 4096

db_create_file_dest=<RMAN Target “rmantgtloc” entered during adclone.pl command>

db_recovery_file_dest=<RMAN Target “rmantgtloc” entered during adclone.pl command>db_recovery_file_dest_size=300g

control_files=<rmantgtloc>/%s_dbGlnam%/controlfile_<%s_dbGlnam%>.ctl

sga_target=1g

processes=200

sga_max_size=2g

open_cursors=1000

compatible=11.1.0 (if %s_database" starts with "db11")

Restore the Database (Primary node) 

The restore is launched by adclone.pl on the Primary target node (RAC or non-RAC) and is done in phases using scripts located under:

$ORACLE_HOME/appsutil/out/$CONTEXT_NAME

1. First phase of RMAN restore 

The scripts involved in this phase are as follows:

restore-rac1.rman

This generates the log:

$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac1.log

p pp yp yp

9 of 23 12/18/2012 6:00 PM

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

Page 10: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 10/23

This restore-rac1.rman script provides the required directives to rman to restore the control file and mount the target database.

Note the DB is started using the initSID1.ora.tmp parameter file

Executed via ApplyRmanDatabase.java as :

rman target / nocatalog @$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac1.rman \

> $ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac1.log

An example of the contents of the restore-rac1.rman script:

startup nomount pfile=[ORACLE_HOME]/dbs/init<SID>1.ora.tmp

restore controlfile from '[RMAN clone stage]/data/stage/backup_controlfile.ctl';

alter database mount;

exit

2. Second phase of RMAN restore 

There are several scripts involved in this phase, and will be explained individually.

The first script is:

restore-rac2.sql

This generates:

$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac2.rman

and log file :

$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac2.log

Executed via ApplyRmanDatabase.java as :

sqlplus /nolog @$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac2.sql

An example of the contents of the restore-rac2.sql script:

connect / as sysdba

set echo off pages 0 feed off sqlp #

spool [ORACLE_HOME]/appsutil/out/<CONTEXT_NAME>/restore-rac2.rman

select 'catalog start with ''[RMAN clone stage]/data/stage'' NOPROMPT;' from dual;

select 'RUN { ' from dual;

select 'set newname for datafile '||file#||' to NEW;' from v$datafile;

select 'set newname for tempfile '||file#||' to NEW;' from v$tempfile;

select 'restore database;' from dual;

select 'switch datafile all;' from dual;

select 'switch tempfile all;' from dual;

select 'recover database;' from dual;

select '} ' from dual;

select 'exit' from dual;

spool off

exit

10 of 23 12/18/2012 6:00 PM

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

Page 11: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 11/23

At this point, a series of files are moved from the clone stage to a temp location.

The following is an entry in the log:

$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDatabase_xxxxxx.log

...

...

Attempting to move clone specific files to temporary directory..Src location: <clone stage>/data/stage

Target location: <ORACLE_HOME>/appsutil/temp/stageMoving files:

adcrdb.zip

addbhomsrc.xml

addbhomtgt.xml

Once the files are moved, the second rman script is run:

restore-rac2.rman

This was generated by the first script, restore-rac2.sql 

This second rman script sets the target location and names of the datafiles to be restored, restores the database, then does a recovery.

Executed via ApplyRmanDatabase.java as :

rman target / nocatalog @$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac2.rman \

> $ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac2.log

An example of the contents of the restore-rac2.rman script as as follows:

catalog start with '[RMAN Clone stage]/data/stage' NOPROMPT;

RUN {

set newname for datafile 1 to NEW;

set newname for datafile 2 to NEW;

set newname for datafile 3 to NEW;

set newname for datafile 4 to NEW;

set newname for datafile 5 to NEW;

set newname for datafile 6 to NEW;

set newname for datafile 7 to NEW;

set newname for datafile 8 to NEW;

set newname for datafile 9 to NEW;

set newname for datafile 10 to NEW;

set newname for datafile 11 to NEW;

set newname for datafile 12 to NEW;set newname for datafile 13 to NEW;

set newname for datafile 14 to NEW;

set newname for datafile 15 to NEW;

set newname for datafile 16 to NEW;

set newname for datafile 17 to NEW;

set newname for datafile 18 to NEW;

set newname for datafile 19 to NEW;

set newname for datafile 20 to NEW;

set newname for datafile 21 to NEW;

set newname for tempfile 1 to NEW;

set newname for tempfile 2 to NEW;

restore database;

switch datafile all;switch tempfile all;

recover database;

}

11 of 23 12/18/2012 6:00 PM

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

Page 12: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 12/23

exit

3. Third phase of RMAN restore 

There are several scripts involved in this phase, and will be explained individually.

The first script is:

restore-rac21.sql

This script generates :

$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac21.sql

Executed via ApplyRmanDatabase.java as :

sqlplus /nolog @$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac21.sql

An example of the contents of the restore-rac21.sql script is as follows:

connect / as sysdba

set echo off pages 0 feed off linesize 5000 TRIM on TRIMS on sqlp #

spool [ORACLE_HOME]/appsutil/out/<CONTEXT_NAME>/restore-rac-logfile.sql

select 'connect /as sysdba' from dual;

select 'alter database rename file '''||member||''' to

'||'''+DATA/<SID>/datafile/'|| REGEXP_REPLACE(''||member||'','.*/') ||''';' from v$logfile;

select 'exit' from dual;

spool off

exit

The next script is:

restore-rac-logfile.sql

This script, as shown above, was generated above by restore-rac21.sql and is run to rename the redo log files to their new file locations

Executed via ApplyRmanDatabase.java as :

sqlplus /nolog @$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac-logfile.sql

An example of the contents of the restore-rac-logfile.sql script is shown below. This script renames the database files to their new locations.

This is why the [Source RAC system GLOBAL name] value is entered as one of the parameters to the adclone.pl command :

connect /as sysdba

alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_1.308.709218019'

to '+DATA/<Target SID>/datafile/group_1.308.709218019';

alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_1.309.709218111'

to '+DATA/<Target SID>/datafile/group_1.309.709218111';

alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_2.310.709218201'to '+DATA/<Target SID>/datafile/group_2.310.709218201';

alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_2.311.709218293'

to '+DATA/<Target SID>/datafile/group_2.311.709218293';

alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_3.305.709218395'

12 of 23 12/18/2012 6:00 PM

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

Page 13: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 13/23

to '+DATA/<Target SID>/datafile/group_3.305.709218395';

alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_3.304.709218485'

to '+DATA/<Target SID>/datafile/group_3.304.709218485';

alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_4.307.709218577'

to '+DATA/<Target SID>/datafile/group_4.307.709218577';

alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_4.306.709218667'

to '+DATA/<Target SID>/datafile/group_4.306.709218667';

exit

The next script is:

restore-rac3.rman

This generates the log file:

$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac3.log

This third rman script opens the new database with "open resetlogs " parameter, shuts down the database and then mounts it.

Within the same rman session, the db_name parameter within the initSID1.ora.tmp parameter file is updated to reflect the target value.

This is in preparation for the DBNEWID changes and a database restart in the next stage.

Executed via ApplyRmanDatabase.java as :

rman target / nocatalog @$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac3.rman \

> $ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac3.log

An example of the contents of the restore-rac-logfile.sql script is shown below.

host 'echo "In the above output, if RMAN-03002 and RMAN-06054 errors are present,

they can be safely ignored." \

>> [ORACLE_HOME]/appsutil/out/<CONTEXT_NAME>/restore-rac2.log';

alter database open resetlogs;

shutdown normal

startup mount pfile=[ORACLE_HOME]/dbs/init<SID>1.ora.tmp

host 'sed "s:db_name=.*:db_name=<Target SID>:g" [ORACLE_HOME]/dbs/init<SID>1.ora.tmp >

[ORACLE_HOME]/dbs/init<SID>1.ora.tmp.delete';

host 'cp [ORACLE_HOME]/dbs/init<SID>1.ora.tmp.delete [ORACLE_HOME]/dbs/init<SID>1.ora.tmp';

exit

>>

4. Final DB Startup 

The final stages are not executed via individual scripts, but directly by ApplyRmanDatabase.java 

DBNEWID (NID)

13 of 23 12/18/2012 6:00 PM

Page 14: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 14/23

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

Page 15: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 15/23

Tip: If any of the Primary node Database steps fail, it may be possible to execute them manually in the same sequence to complete the clone.

Create pairsfile.txt File for Secondary Node(s) 

The pairsfile needs to be the same across all nodes with the only exception being the value for the variable “ s_undo_tablespace” 

s_undo_tablespace=[UNDOTBS(+1) for additional Nodes]

s_dbClusterInst=[Total number of Instances in a cluster e.g. 2]

s_db_oh=[Location of new ORACLE_HOME]

Create Secondary Node context file 

This step is achieved using the adclonectx.pl command. This will clone an existing context file and update it with the configuration details entered via prompts.

It will give the option to validate the inputs, such as check port availability.

Several prompts requesting information about the target Primary node will be displayed at this point:

...

...

Target Instance is RAC (y/n) [y] : Y

Please provide the details to connect to one of live RAC nodes

Host name of the live RAC node :

Domain name of the live RAC node :

Database SID of the live RAC node :

Listener port number of the live RAC node :

Provide information for the new Node:

Host name :

Virtual Host name :

Instance number : 2

Private interconnect name :

...

...

A full list of these prompts are shown in Section 6 of the main cloning Document 559518.1 - Cloning Oracle E-Business Suite Release 12 RAC-Enabled Systems with Rapid Clone 

As with the Primary node, ensure to edit the value of the target system context variable "s_db_listener " to reflect the desired name for the tns listener.

The traditionally accepted value is "LISTENER_[HOSTNAME] " but may be any valid value unique to the target host.

The main log file is generated in the working directory when adclonectx.pl was run:

$ORACLE_HOME/appsutil/clone/bin/CloneContext_xxxxxx.log

If the input validation option is selected, the port availability check will generate the following log:

$ORACLE_HOME/appsutil/out/portpool.lst

Configure the Target Secondary Oracle Home(s) 

On the Secondary node(s) the perl adcfgclone.pl dbTechStack command is used during this step which generates the log file:

15 of 23 12/18/2012 6:00 PM

Page 16: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 16/23

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

Page 17: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 17/23

about registering the current node.

Section 4: Known Issues

a) Final DB Startup fails if Flash Recovery is enabled

Screen output would be as follows:

...

...

Completed first phase of RMAN restore

Entering second phase of RMAN restore

\ 50% completed

Completed second phase of RMAN restore

Entering third phase of RMAN restore

- 50% completed

Completed third phase of RMAN restore

- 50% completed

ERROR while running Apply..

To find out why this failed, the information will be recorded in the log file created for this third phase restore-rac3.log 

$ cat $ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac3.log

...

...

...

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of startup command at 03/07/2011 16:33:43

ORA-38760: This database instance failed to turn on flashback database

Cause: 

This was expected as the Restrictions section of Document 559518.1 - Cloning Oracle E-Business Suite Release 12 RAC-Enabled Systems with Rapid Clone states:

“During the cloning process, no allowance is made for the use of a Flash Recovery Area (FRA). If an FRA needs to be configured on the target system, it must be done manually. ”

Potential Solution/Workaround: 

Complete the cloning steps manually. At the point the clone failed above, the only remaining steps are:

1. Enter the flash recovery parameters and check the control_file parameter

Add the flash recovery parameters to the $ORACLE_HOME/dbs/<SID>1_<hostname>_ifile.ora.

17 of 23 12/18/2012 6:00 PM

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

Page 18: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 18/23

For example:

db_recovery_file_dest='+FLASH'

db_recovery_file_dest_size='120G'

Check the <ORACLE_HOME>/dbs/<sid>_APPS_BASE.ora and ensure it contains the correct control file entry:

control_files = +DATA/<SID>/controlfile_<SID>.ctl

2. Start the Database and run the FND_NODES cleanup process .

$ sqlplus / as sysdba

SQL> startup

SQL> exec fnd_conc_clone.setup_clean;

SQL> conn apps/<pass>

Tip: To avoid the clone terminating with this failure, it’s possible to create and update the <SID>_<host>_ifile.ora manually with the correct flash recovery parameters before running the adclone.pl command.

This file will not be referenced until the “Final DB startup ” phase where the Database is renamed and started with the init<SID>.ora parameter file.

b) On Secondary nodes, adclonectx.pl fails with Instance Number n used by host

Screen output would be as follows:

...

...

Please provide the details to connect to one of live RAC nodes

Host name of the live RAC node : ractrgt01

Domain name of the live RAC node : uk.oracle.com

Database SID of the live RAC node : R12RAC1

Listener port number of the live RAC node : 1521

Provide information for the new Node:

Host name : ractrgt02

Virtual Host name : vip-ractrgt02

Instance number : 2

Instance number '2' is used by host 'racnode02'

Instance number : ERROR: context creation not completed successfully

Cause: 

If the clone does not completed successfully, it may not run the FND_NODES Cleanup process was not run. I did not run this manually either, which has caused this problem.

Solution: 

Run FND_NODES Cleanup process manually

18 of 23 12/18/2012 6:00 PM

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

Page 19: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 19/23

On Primary node

1. Run the following

SQL> exec FND_CONC_CLONE.SETUP_CLEAN

2.Run AutoConfig

$ cd $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME

$ ./adautocfg.sh

On Secondary node

3. Retry adclonectx.pl

c) RAC to RAC : Inventory Registration still references the source cluster nodes

This has been raised as Bug 11705106 - R12 RAPIDCLONE RAC CLONING DOES NOT UPDATE INVENTORY CORRECTLY 

Problem: 

After successful inventory registration, the inventory.xml in the Central Inventory still contains the source Cluster nodes, when they should now reflect the new cluster nodes.

For example:

Source nodes: racnode01 & racnode02

Target nodes: ractrgt01 & ractrgt02

Both target nodes still reference racnode01 & racnode02 :

ractrgt01: /etc/oraInventory/ContentsXML/inventory.xml contains:

<HOME NAME="R12RAC1_DB__oracle_R12RAC_db_tech_st_11_1_0" LOC="/oracle/R12RAC/db/tech_st/11.1.0" TYPE="O" IDX="3">

<NODE_LIST>

<NODE NAME="racnode01"/>

<NODE NAME="racnode02"/>

</NODE_LIST>

</HOME>

ractrgt02: /etc/oraInventory/ContentsXML/inventory.xml contains:

<HOME NAME="R12RAC2_DB__oracle_R12RAC_db_tech_st_11_1_0" LOC="oracle/R12RAC/db/tech_st/11.1.0" TYPE="O" IDX="3">

<NODE_LIST>

<NODE NAME="racnode01"/>

<NODE NAME="racnode02"/>

</NODE_LIST>

</HOME>

Cause: 

The script called by the cloning process, ouicli.pl which launches runInstaller, does not pass the CLUSTER_NODES parameter

The RAC Cloning process runs the following, which was extracted from ohclone.log :

19 of 23 12/18/2012 6:00 PM

Page 20: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 20/23

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

Page 21: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 21/23

$ $ORACLE_HOME/oui/bin/detachHome.sh

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.

Actual 4909 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /etc/oraInventory

'DetachHome' was successful.

3. The detachHome.sh script will overwrite any existing oraclehomeproperties.xml file, so now restore the backup file taken in step 1.

Restore: oraclehomeproperties.xml.source 

$ cd $ORACLE_HOME/inventory/ContentsXML

$ cp oraclehomeproperties.xml.source oraclehomeproperties.xml

4. Run the runInstaller command with the correct CLUSTER_NODES parameter.

Ensure you use the correct HOME_NAME for each node, as it’s based on the instance name R12RAC1, R12RAC2, not the service name R12RAC 

Node 1: (HOME_NAME=R12RAC1_DB__oracle_R12RAC_db_tech_st_11_1_0 )

/oracle/R12RAC/db/tech_st/11.1.0/oui/bin/runInstaller -clone \

-silent -force -nolink -waitForCompletion \

-invPtrLoc /etc/oraInst.loc \

ORACLE_HOME=/oracle/R12RAC/db/tech_st/11.1.0 \

ORACLE_BASE=/oracle/R12RAC/db/tech_st/11.1.0 \ORACLE_HOME_NAME=R12RAC1_DB__oracle_R12RAC_db_tech_st_11_1_0 \

"CLUSTER_NODES={ractrgt01,ractrgt02}" \

-J-Doracle.installer.noLink=true

The cloning of R12RAC1_DB__oracle_R12RAC_db_tech_st_11_1_0 was successful.

Please check '/etc/oraInventory/logs/cloneActionsxxxx-xx-xx_xx-xx-xxXX.log' for more details.

f) Re-check /etc/oraInventory/ContentsXML/inventory.xml 

<HOME NAME="R12RAC1_DB__oracle_R12RAC_db_tech_st_11_1_0" LOC="/oracle/R12RAC/db/tech_st/11.1.0" TYPE="O" IDX="3"><NODE_LIST>

<NODE NAME="ractrgt01"/>

<NODE NAME="ractrgt02"/>

</NODE_LIST>

</HOME>

g) Repeat the above on all Target RAC nodes

h) Finally, check the inventory metadata on all nodes:

$ORACLE_HOME/OPatch/opatch lsinventory -all_nodes

21 of 23 12/18/2012 6:00 PM

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

Page 22: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 22/23

d) RAC to non-RAC : Inventory Registration still contains the source cluster nodes

Problem: RAC to non-RAC 

After successful inventory registration, the inventory.xml in the Central Inventory still contains the source Cluster nodes, when they should now reflect a single non-cluster.

  Source nodes: racnode01 & racnode02

  Target nodes: mynode01

On mynode01 /etc/oraInventory/ContentsXML/inventory.xml contains:

<HOME NAME="R12NORAC_DB__oracle_R12NORAC_db_tech_st_11_1_0"

LOC="/oracle/R12NORAC/db/tech_st/11.1.0" TYPE="O" IDX="30">

<NODE_LIST>

<NODE NAME="racnode01"/>

<NODE NAME="racnode02"/>

</NODE_LIST>

</HOME>

Cause: 

The cause is the same as the RAC-to-RAC clone, and raised in Bug 11705106. The runInstaller should include "CLUSTER_NODES={}" for RAC to Single clones.

The "CLUSTER_NODES " parameter will ensure the oraclehomeproperties.xm l inside the Local Home inventory ($ORACLE_HOME/inventory/ContentsXML) will be updated when run on each "LOCAL_NODE ".

Solution: 

On the target database node, de-register the ORACLE_HOME, and re-run the correct runInstaller command manually.

a) Run detachHome.sh

$ $ORACLE_HOME/oui/bin/detachHome.sh

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.

Actual 4909 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /etc/oraInventory

'DetachHome' was successful.

b) Run the runInstaller command with the CLUSTER_NODES={}” parameter

/oracle/R12NORAC/db/tech_st/11.1.0/oui/bin/runInstaller \

-clone -silent -force -nolink -waitForCompletion \

-invPtrLoc /etc/oraInst.loc ORACLE_HOME=/oracle/R12NORAC/db/tech_st/11.1.0 \

ORACLE_BASE=/oracle/R12NORAC/db/tech_st/11.1.0 \

ORACLE_HOME_NAME=R12NORAC_DB__oracle_R12NORAC_db_tech_st_11_1_0 \

"CLUSTER_NODES={}" \-J-Doracle.installer.noLink=true

22 of 23 12/18/2012 6:00 PM

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...

Page 23: Troubleshooting RAC RapidClone issues with Oracle Application R12

7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12

http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 23/23

The cloning of R12NORAC_DB__vm3_R12NORAC_db_tech_st_11_1_0 was successful.

Please check '/etc/oraInventory/logs/cloneActionsxxxx-xx-xx_xx-xx-xxXX.log' for more details.

c) Re-check /etc/oraInventory/ContentsXML/inventory.xml

<HOME NAME="R12NORAC_DB__oracle_R12NORAC_db_tech_st_11_1_0"

LOC="/oracle/R12NORAC/db/tech_st/11.1.0" TYPE="O" IDX="30"/></HOME>

References

NOTE:406982.1 - Cloning Oracle Applications Release 12 with Rapid Clone

NOTE:559518.1 - Cloning Oracle E-Business Suite Release 12 RAC-Enabled Systems with Rapid Clone

NOTE:603104.1 - Troubleshooting RapidClone issues with Oracle Applications R12

 Related

Products

Oracle E-Business Suite > Applications Technology > Lifecycle Management > Oracle Applications Manager

Keywords

ADCFGCLONE.PL; TROUBLESHOOT; ADCLONE.PL; ADCLONECTX.PL; ADCONFIG.SH; ADPRECLONE.PL; E-BUSINESS; RAC

Errors

OUI-25023; ORA-38760; ORA-1103; RMAN-6054; RMAN-3002; RMAN-569; RMAN-571; 06054 ERROR

Back to top

Copyright (c) 2007, 2010, Oracle. All rights reserved. Legal Notices and Terms of Use | Privacy Statement

23 of 23 12/18/2012 6:00 PM