Upload
yusufcar
View
234
Download
0
Embed Size (px)
Citation preview
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 1/23
Troubleshooting RAC RapidClone issues with Oracle Applications R12 [ID 1303962.1]
Modified 15-SEP-2011 Type TROUBLESHOOTING Status PUBLISHED
In this Document Purpose This note only covers the Database Tier(s). Last Review Date Instructions for the Reader Troubleshooting Details Section 1: Cloning log files required by Support Section 2: Preparation - How it works Section 3: Configuration - How it works Section 4: Known Issues a) Final DB Startup fails i f Flash Recovery is enabled
b) On Secondary nodes, adclonectx.pl fails with Instance Number n used by host c) RAC to RAC : Inventory Registration still references the source cluster nodes d) RAC to non-RAC : Inventory Registration still contains the source cluster nodes References
Applies to:
Oracle Applications Manager - Version: 12.0.0 to 12.1.3 - Release: 12 to 12.1
Information in this document applies to any platform.
PurposeThis troubleshooting note is intended as a guide to help determine the cause of some problems that may arise during the Database Tier phase only of cloning
E-Business Suite Release 12 with Real Application Clusters.
This Note is NOT a replacement for Document 559518.1 - Cloning Oracle E-Business Suite Release 12 RAC-Enabled Systems with Rapid Clone.
That document must be followed when performing a clone of Oracle Applications R12 running on RAC.
The idea is to provide a detailed explanation into what each step is trying to achieve making it easier to identify which part of the clone has failed and why.
Locations of log files are listed in Section 1, and are explained in each section. These should be uploaded when reporting a cloning issue to Oracle Support.
Note: This Note does not yet cover 11.2.0 Grid specific functionality such as the SCAN listener, but this document will be updated regularly and will be included in a future update.
This note only covers the Database Tier(s).
For the Application Tier, refer to Document 603104.1 - Troubleshooting RapidClone issues with Oracle Applications R12.
Last Review Date
March 15, 2011
Instructions for the Reader
A Troubleshooting Guide is provided to assist in debugging a specific issue. When possible, diagnostic tools are included in the document to assist in troubleshooting.
Rate this document
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
1 of 23 12/18/2012 6:00 PM
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 2/23
Troubleshooting Details
Section 1: Cloning log files required by Support
If the problem could not be diagnosed or resolved with the details in this document, please raise a Service Request to Oracle Support for assistance and
upload the following log files from each Database node:
Source System log files
adpreclone.pl logfile:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDBTier_xxxxxx.log
adclone.pl [mode=stage] logfile:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDatabase_xxxxxx.log
RMAN backup script and logfile:
<clone stage>/data/template/adcldbstage-rac.rman
<clone stage>/data/out/adcldbstage-rac.log
Target System log files
Create Context file logfile:
Generated in the working directory when adclonectx.pl was run:
$ORACLE_HOME/appsutil/clone/bin/CloneContext_xxxxxx.log
Port Pool logfile:
$ORACLE_HOME/appsutil/out/portpool.lst
Oracle Home configuration logfiles:
Primary node:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_xxxxxx.log
Secondary node(s):
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTechStack_xxxxxx.log
Relink logfile:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/make_<timestamp>.log
Inventory Registration and configuration logfiles:
$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/log/ohclone.log
<Global Inventory>/logs/silentInstall<timestamp>..log
<Global Inventory>/logs/oraInstall<timestamp>..out
<Global Inventory>/logs/cloneActions<timestamp>..log
RMAN Restore scripts and logfiles:
$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac1.rman
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
2 of 23 12/18/2012 6:00 PM
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 3/23
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 4/23
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 5/23
https://support oracle com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 6/23
SQL>select 'configure device type disk parallelism 5 backup type to backupset;' \
from dual;
SQL>select 'configure maxsetsize to '||round((max(bytes)/1024/1024) + 200)||'m;' \
from dba_data_files;
SQL>select 'backup as backupset tag ''RapidClone_RAC'' database format \''<your location>/data/stage/%U'';' from dual;
SQL>select 'backup as backupset tag ''RapidClone_RAC'' archivelog all format \
''<your location>/data/stage/%U'';' from dual;
SQL>select 'backup tag ''RapidClone_RAC'' as copy current controlfile format \
''<your location>/data/stage/backup_controlfile.ctl'';' from dual;
SQL>select 'exit' from dual;
SQL>spool off;
SQL>exit;
The adcldbstage-rac.rman script generates the logfile called :
[Stage Directory]/data/out/adcldbstage-rac.log
Create Oracle Home Archive
The third and final preparation phase is to create an archive of the ORACLE_HOME:
$ cd $ORACLE_HOME/..
$ tar -cvzf [Stage Directory]/rac_db_oh.tgz [DATABASE TOP LEVEL DIRECTORY]
As this is a manual step there is no log generated unless specifically requested when running the tar command.
Oracle does recommended using data integrity utilities such asmd5sum, sha1sum, or cksum to validate the file sum both before and after transfer to the target system.
Section 3: Configuration - How it works
About Non-RAC clone steps
On the target host system the context file creation should be done as in the case of Single Instance cloning.
All other primary target restore tasks remain the same from the RAC-to-RAC clone in the case of Single Instance Restore.
Target Instance is RAC (y/n) [y] : [Enter n]
As the target non-RAC is cloned from a RAC enabled system, the interview question above pre-selects a default value of being a RAC Instance.
Be certain to select "N" for the above question.
By creating a context file with without RAC attributes present, Rapid Clone will configure and convert the RDBMS Technology Stack and its binaries on the target system. This to ensure a Single Instance restore can
be performed.
The Rapid Clone commands to restore the Oracle Home and the database on the target system remain the same whether the target is to be RAC or Single Instance.
Extract Oracle Home Archive
The first phase on the Target is to extract the Oracle Home from the archive created from the Source:
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
6 of 23 12/18/2012 6:00 PM
https://support oracle com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 7/23
$ tar -xvzf rac_db_oh.tgz
As mentioned in the previous section, there is no log generated unless specifically requested when running the tar command,
but data integrity utilities should be used, such as md5sum , sha1sum , or cksum to validate the file sum both before and after transfer to the target system.
Create pairsfile.txt File for Primary Node
The pairsfile needs to be the same across all nodes with the only exception being the value for the variable “ s_undo_tablespace ”
s_undo_tablespace=[UNDOTBS1 for Initial Node]
s_dbClusterInst=[Total number of Instances in a cluster e.g. 2]
s_db_oh=[Location of new ORACLE_HOME]
Create Primary Node context file
This step is achieved using the adclonectx.pl command. This will clone an existing context file and update it with the configuration details entered via prompts.
It will give the option to validate the inputs, such as check port availability.
Several prompts requesting information about the target Primary node will be displayed at this point:
...
...
Provide information for the initial RAC node:
Host name [<source host>] :
Virtual Host name [null] :
Instance number [1] : 1
Private interconnect name [<source host>] :
...
...
A full list of these prompts are shown in Section 5 of the main cloning Document 559518.1 - Cloning Oracle E-Business Suite Release 12 RAC-Enabled Systems with Rapid Clone
There is a new variable introduced with the latest AutoConfig patches s_db_listener which needs to be updated manually, and is also mentioned in the main clone note as follows:
Note: If the most current AutoConfig Template patch as listed in OracleMetalink Document 387859.1 has already been applied to the source system,
it is necessary to edit the value of the target system context variable "s_db_listener" to reflect the desired name for the tns listener.
The traditionally accepted value is "LISTENER_[HOSTNAME]" but may be any valid value unique to the target host.
The main log file is generated in the working directory when adclonectx.pl was run:
$ORACLE_HOME/appsutil/clone/bin/CloneContext_xxxxxx.log
If the input validation option is selected, the port availability check will generate the following log:
$ORACLE_HOME/appsutil/out/portpool.lst
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7 of 23 12/18/2012 6:00 PM
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 8/23
Note: When cloning RAC to non-RAC, ensure the correct response of "n" is entered for the prompt:
Target Instance is RAC (y/n) [y] : n
Configure the Target Primary Oracle Home
On the Primary Target node (RAC or non-RAC), the adclone.pl command is used to configure the Target Database home and the database. The command is:
perl adclone.pl \
java=[JDK 1.5 Location] \
component=dbTier \
mode=apply \
stage=[ORACLE_HOME]/appsutil/clone \
method=CUSTOM \
dbctxtg=[Full Path to the Target Context File] \
rmanstage=[Location of the Source RMAN dump files... i.e. RMAN_STAGE/data/stage] \
rmantgtloc=[Shared storage loc for datafiles...ASM diskgroup / NetApps NFS mount / OCFS mount point]srcdbname=[Source RAC system GLOBAL name] \
pwd=[APPS Password] \
showProgressode
Note one of the values requested is the [Source RAC system GLOBAL name] . The cloning process used this value when restoring the database files.
The restored database uses the same global name db_name as the source, but a different db_unique_name.
The backup files are restored and renamed on the target, and then the database is renamed using DBNEWID (NID) at the final stages.
Failure to provide the correct source srcdbname in the above command, will generate the following error:
RMAN-00571: ===========================================================RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of alter db command at 09/14/2011 09:44:17
ORA-01103: database name '<Source>' in control file is not '<Target>'
This will be explained later in this document.
The adclone.pl step which generates the log file:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDatabase_xxxxxx.log
The adclone.pl script will start the process, which involves the following:
Registers and configures the Oracle Home in the inventory using the script:
$ORACLE_HOME/appsutil/clone/ouicli.pl
This generates the log files :
$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/log/ohclone.log
<Global Inventory>/logs/silentInstall<timestamp>..log
<Global Inventory>/logs/oraInstall<timestamp>..out
<Global Inventory>/logs/cloneActions<timestamp>..log
Relinks the Oracle Home using script :
$ORACLE_HOME/appsutil/clone/adlnkoh.sh
p pp w yp yp
8 of 23 12/18/2012 6:00 PM
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 9/23
This generates the log file:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/make_<timestamp>.log
Creates the Target environment and configuration files
Generates and executes a series of RMAN restore scripts
The restoration is done in stages and is explained in this section.
The scripts are generated under :
$ORACLE_HOME/appsutil/out/$CONTEXT_NAME
The scripts are:
restore-rac1.rman
restore-rac2.sql
restore-rac2.rman
restore-rac21.sql
restore-rac-logfile.sql
restore-rac3.rman
Creates a temporary initSID1.ora.tmp file used to start the Database during the restoration from the backup files.
The temporary initSID1.ora.tmp is generated via ApplyRmanDatabase.java
This file contains some hardcoded values, and is built as follows:
db_name=<source Global DB name - srcdbname value entered during adclone.pl command>
db_unique_name=<Target Global DB name - %s_dbGlnam% value from context file>
db_files = 4096
db_create_file_dest=<RMAN Target “rmantgtloc” entered during adclone.pl command>
db_recovery_file_dest=<RMAN Target “rmantgtloc” entered during adclone.pl command>db_recovery_file_dest_size=300g
control_files=<rmantgtloc>/%s_dbGlnam%/controlfile_<%s_dbGlnam%>.ctl
sga_target=1g
processes=200
sga_max_size=2g
open_cursors=1000
compatible=11.1.0 (if %s_database" starts with "db11")
Restore the Database (Primary node)
The restore is launched by adclone.pl on the Primary target node (RAC or non-RAC) and is done in phases using scripts located under:
$ORACLE_HOME/appsutil/out/$CONTEXT_NAME
1. First phase of RMAN restore
The scripts involved in this phase are as follows:
restore-rac1.rman
This generates the log:
$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac1.log
p pp yp yp
9 of 23 12/18/2012 6:00 PM
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 10/23
This restore-rac1.rman script provides the required directives to rman to restore the control file and mount the target database.
Note the DB is started using the initSID1.ora.tmp parameter file
Executed via ApplyRmanDatabase.java as :
rman target / nocatalog @$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac1.rman \
> $ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac1.log
An example of the contents of the restore-rac1.rman script:
startup nomount pfile=[ORACLE_HOME]/dbs/init<SID>1.ora.tmp
restore controlfile from '[RMAN clone stage]/data/stage/backup_controlfile.ctl';
alter database mount;
exit
2. Second phase of RMAN restore
There are several scripts involved in this phase, and will be explained individually.
The first script is:
restore-rac2.sql
This generates:
$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac2.rman
and log file :
$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac2.log
Executed via ApplyRmanDatabase.java as :
sqlplus /nolog @$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac2.sql
An example of the contents of the restore-rac2.sql script:
connect / as sysdba
set echo off pages 0 feed off sqlp #
spool [ORACLE_HOME]/appsutil/out/<CONTEXT_NAME>/restore-rac2.rman
select 'catalog start with ''[RMAN clone stage]/data/stage'' NOPROMPT;' from dual;
select 'RUN { ' from dual;
select 'set newname for datafile '||file#||' to NEW;' from v$datafile;
select 'set newname for tempfile '||file#||' to NEW;' from v$tempfile;
select 'restore database;' from dual;
select 'switch datafile all;' from dual;
select 'switch tempfile all;' from dual;
select 'recover database;' from dual;
select '} ' from dual;
select 'exit' from dual;
spool off
exit
10 of 23 12/18/2012 6:00 PM
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 11/23
At this point, a series of files are moved from the clone stage to a temp location.
The following is an entry in the log:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDatabase_xxxxxx.log
...
...
Attempting to move clone specific files to temporary directory..Src location: <clone stage>/data/stage
Target location: <ORACLE_HOME>/appsutil/temp/stageMoving files:
adcrdb.zip
addbhomsrc.xml
addbhomtgt.xml
Once the files are moved, the second rman script is run:
restore-rac2.rman
This was generated by the first script, restore-rac2.sql
This second rman script sets the target location and names of the datafiles to be restored, restores the database, then does a recovery.
Executed via ApplyRmanDatabase.java as :
rman target / nocatalog @$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac2.rman \
> $ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac2.log
An example of the contents of the restore-rac2.rman script as as follows:
catalog start with '[RMAN Clone stage]/data/stage' NOPROMPT;
RUN {
set newname for datafile 1 to NEW;
set newname for datafile 2 to NEW;
set newname for datafile 3 to NEW;
set newname for datafile 4 to NEW;
set newname for datafile 5 to NEW;
set newname for datafile 6 to NEW;
set newname for datafile 7 to NEW;
set newname for datafile 8 to NEW;
set newname for datafile 9 to NEW;
set newname for datafile 10 to NEW;
set newname for datafile 11 to NEW;
set newname for datafile 12 to NEW;set newname for datafile 13 to NEW;
set newname for datafile 14 to NEW;
set newname for datafile 15 to NEW;
set newname for datafile 16 to NEW;
set newname for datafile 17 to NEW;
set newname for datafile 18 to NEW;
set newname for datafile 19 to NEW;
set newname for datafile 20 to NEW;
set newname for datafile 21 to NEW;
set newname for tempfile 1 to NEW;
set newname for tempfile 2 to NEW;
restore database;
switch datafile all;switch tempfile all;
recover database;
}
11 of 23 12/18/2012 6:00 PM
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 12/23
exit
3. Third phase of RMAN restore
There are several scripts involved in this phase, and will be explained individually.
The first script is:
restore-rac21.sql
This script generates :
$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac21.sql
Executed via ApplyRmanDatabase.java as :
sqlplus /nolog @$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac21.sql
An example of the contents of the restore-rac21.sql script is as follows:
connect / as sysdba
set echo off pages 0 feed off linesize 5000 TRIM on TRIMS on sqlp #
spool [ORACLE_HOME]/appsutil/out/<CONTEXT_NAME>/restore-rac-logfile.sql
select 'connect /as sysdba' from dual;
select 'alter database rename file '''||member||''' to
'||'''+DATA/<SID>/datafile/'|| REGEXP_REPLACE(''||member||'','.*/') ||''';' from v$logfile;
select 'exit' from dual;
spool off
exit
The next script is:
restore-rac-logfile.sql
This script, as shown above, was generated above by restore-rac21.sql and is run to rename the redo log files to their new file locations
Executed via ApplyRmanDatabase.java as :
sqlplus /nolog @$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac-logfile.sql
An example of the contents of the restore-rac-logfile.sql script is shown below. This script renames the database files to their new locations.
This is why the [Source RAC system GLOBAL name] value is entered as one of the parameters to the adclone.pl command :
connect /as sysdba
alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_1.308.709218019'
to '+DATA/<Target SID>/datafile/group_1.308.709218019';
alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_1.309.709218111'
to '+DATA/<Target SID>/datafile/group_1.309.709218111';
alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_2.310.709218201'to '+DATA/<Target SID>/datafile/group_2.310.709218201';
alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_2.311.709218293'
to '+DATA/<Target SID>/datafile/group_2.311.709218293';
alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_3.305.709218395'
12 of 23 12/18/2012 6:00 PM
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 13/23
to '+DATA/<Target SID>/datafile/group_3.305.709218395';
alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_3.304.709218485'
to '+DATA/<Target SID>/datafile/group_3.304.709218485';
alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_4.307.709218577'
to '+DATA/<Target SID>/datafile/group_4.307.709218577';
alter database rename file '+DATA/[Source RAC system GLOBAL name]/onlinelog/group_4.306.709218667'
to '+DATA/<Target SID>/datafile/group_4.306.709218667';
exit
The next script is:
restore-rac3.rman
This generates the log file:
$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac3.log
This third rman script opens the new database with "open resetlogs " parameter, shuts down the database and then mounts it.
Within the same rman session, the db_name parameter within the initSID1.ora.tmp parameter file is updated to reflect the target value.
This is in preparation for the DBNEWID changes and a database restart in the next stage.
Executed via ApplyRmanDatabase.java as :
rman target / nocatalog @$ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac3.rman \
> $ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac3.log
An example of the contents of the restore-rac-logfile.sql script is shown below.
host 'echo "In the above output, if RMAN-03002 and RMAN-06054 errors are present,
they can be safely ignored." \
>> [ORACLE_HOME]/appsutil/out/<CONTEXT_NAME>/restore-rac2.log';
alter database open resetlogs;
shutdown normal
startup mount pfile=[ORACLE_HOME]/dbs/init<SID>1.ora.tmp
host 'sed "s:db_name=.*:db_name=<Target SID>:g" [ORACLE_HOME]/dbs/init<SID>1.ora.tmp >
[ORACLE_HOME]/dbs/init<SID>1.ora.tmp.delete';
host 'cp [ORACLE_HOME]/dbs/init<SID>1.ora.tmp.delete [ORACLE_HOME]/dbs/init<SID>1.ora.tmp';
exit
>>
4. Final DB Startup
The final stages are not executed via individual scripts, but directly by ApplyRmanDatabase.java
DBNEWID (NID)
13 of 23 12/18/2012 6:00 PM
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 14/23
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 15/23
Tip: If any of the Primary node Database steps fail, it may be possible to execute them manually in the same sequence to complete the clone.
Create pairsfile.txt File for Secondary Node(s)
The pairsfile needs to be the same across all nodes with the only exception being the value for the variable “ s_undo_tablespace”
s_undo_tablespace=[UNDOTBS(+1) for additional Nodes]
s_dbClusterInst=[Total number of Instances in a cluster e.g. 2]
s_db_oh=[Location of new ORACLE_HOME]
Create Secondary Node context file
This step is achieved using the adclonectx.pl command. This will clone an existing context file and update it with the configuration details entered via prompts.
It will give the option to validate the inputs, such as check port availability.
Several prompts requesting information about the target Primary node will be displayed at this point:
...
...
Target Instance is RAC (y/n) [y] : Y
Please provide the details to connect to one of live RAC nodes
Host name of the live RAC node :
Domain name of the live RAC node :
Database SID of the live RAC node :
Listener port number of the live RAC node :
Provide information for the new Node:
Host name :
Virtual Host name :
Instance number : 2
Private interconnect name :
...
...
A full list of these prompts are shown in Section 6 of the main cloning Document 559518.1 - Cloning Oracle E-Business Suite Release 12 RAC-Enabled Systems with Rapid Clone
As with the Primary node, ensure to edit the value of the target system context variable "s_db_listener " to reflect the desired name for the tns listener.
The traditionally accepted value is "LISTENER_[HOSTNAME] " but may be any valid value unique to the target host.
The main log file is generated in the working directory when adclonectx.pl was run:
$ORACLE_HOME/appsutil/clone/bin/CloneContext_xxxxxx.log
If the input validation option is selected, the port availability check will generate the following log:
$ORACLE_HOME/appsutil/out/portpool.lst
Configure the Target Secondary Oracle Home(s)
On the Secondary node(s) the perl adcfgclone.pl dbTechStack command is used during this step which generates the log file:
15 of 23 12/18/2012 6:00 PM
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 16/23
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 17/23
about registering the current node.
Section 4: Known Issues
a) Final DB Startup fails if Flash Recovery is enabled
Screen output would be as follows:
...
...
Completed first phase of RMAN restore
Entering second phase of RMAN restore
\ 50% completed
Completed second phase of RMAN restore
Entering third phase of RMAN restore
- 50% completed
Completed third phase of RMAN restore
- 50% completed
ERROR while running Apply..
To find out why this failed, the information will be recorded in the log file created for this third phase restore-rac3.log
$ cat $ORACLE_HOME/appsutil/out/$CONTEXT_NAME/restore-rac3.log
...
...
...
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of startup command at 03/07/2011 16:33:43
ORA-38760: This database instance failed to turn on flashback database
Cause:
This was expected as the Restrictions section of Document 559518.1 - Cloning Oracle E-Business Suite Release 12 RAC-Enabled Systems with Rapid Clone states:
“During the cloning process, no allowance is made for the use of a Flash Recovery Area (FRA). If an FRA needs to be configured on the target system, it must be done manually. ”
Potential Solution/Workaround:
Complete the cloning steps manually. At the point the clone failed above, the only remaining steps are:
1. Enter the flash recovery parameters and check the control_file parameter
Add the flash recovery parameters to the $ORACLE_HOME/dbs/<SID>1_<hostname>_ifile.ora.
17 of 23 12/18/2012 6:00 PM
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 18/23
For example:
db_recovery_file_dest='+FLASH'
db_recovery_file_dest_size='120G'
Check the <ORACLE_HOME>/dbs/<sid>_APPS_BASE.ora and ensure it contains the correct control file entry:
control_files = +DATA/<SID>/controlfile_<SID>.ctl
2. Start the Database and run the FND_NODES cleanup process .
$ sqlplus / as sysdba
SQL> startup
SQL> exec fnd_conc_clone.setup_clean;
SQL> conn apps/<pass>
Tip: To avoid the clone terminating with this failure, it’s possible to create and update the <SID>_<host>_ifile.ora manually with the correct flash recovery parameters before running the adclone.pl command.
This file will not be referenced until the “Final DB startup ” phase where the Database is renamed and started with the init<SID>.ora parameter file.
b) On Secondary nodes, adclonectx.pl fails with Instance Number n used by host
Screen output would be as follows:
...
...
Please provide the details to connect to one of live RAC nodes
Host name of the live RAC node : ractrgt01
Domain name of the live RAC node : uk.oracle.com
Database SID of the live RAC node : R12RAC1
Listener port number of the live RAC node : 1521
Provide information for the new Node:
Host name : ractrgt02
Virtual Host name : vip-ractrgt02
Instance number : 2
Instance number '2' is used by host 'racnode02'
Instance number : ERROR: context creation not completed successfully
Cause:
If the clone does not completed successfully, it may not run the FND_NODES Cleanup process was not run. I did not run this manually either, which has caused this problem.
Solution:
Run FND_NODES Cleanup process manually
18 of 23 12/18/2012 6:00 PM
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 19/23
On Primary node
1. Run the following
SQL> exec FND_CONC_CLONE.SETUP_CLEAN
2.Run AutoConfig
$ cd $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME
$ ./adautocfg.sh
On Secondary node
3. Retry adclonectx.pl
c) RAC to RAC : Inventory Registration still references the source cluster nodes
This has been raised as Bug 11705106 - R12 RAPIDCLONE RAC CLONING DOES NOT UPDATE INVENTORY CORRECTLY
Problem:
After successful inventory registration, the inventory.xml in the Central Inventory still contains the source Cluster nodes, when they should now reflect the new cluster nodes.
For example:
Source nodes: racnode01 & racnode02
Target nodes: ractrgt01 & ractrgt02
Both target nodes still reference racnode01 & racnode02 :
ractrgt01: /etc/oraInventory/ContentsXML/inventory.xml contains:
<HOME NAME="R12RAC1_DB__oracle_R12RAC_db_tech_st_11_1_0" LOC="/oracle/R12RAC/db/tech_st/11.1.0" TYPE="O" IDX="3">
<NODE_LIST>
<NODE NAME="racnode01"/>
<NODE NAME="racnode02"/>
</NODE_LIST>
</HOME>
ractrgt02: /etc/oraInventory/ContentsXML/inventory.xml contains:
<HOME NAME="R12RAC2_DB__oracle_R12RAC_db_tech_st_11_1_0" LOC="oracle/R12RAC/db/tech_st/11.1.0" TYPE="O" IDX="3">
<NODE_LIST>
<NODE NAME="racnode01"/>
<NODE NAME="racnode02"/>
</NODE_LIST>
</HOME>
Cause:
The script called by the cloning process, ouicli.pl which launches runInstaller, does not pass the CLUSTER_NODES parameter
The RAC Cloning process runs the following, which was extracted from ohclone.log :
19 of 23 12/18/2012 6:00 PM
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 20/23
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 21/23
$ $ORACLE_HOME/oui/bin/detachHome.sh
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.
Actual 4909 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /etc/oraInventory
'DetachHome' was successful.
3. The detachHome.sh script will overwrite any existing oraclehomeproperties.xml file, so now restore the backup file taken in step 1.
Restore: oraclehomeproperties.xml.source
$ cd $ORACLE_HOME/inventory/ContentsXML
$ cp oraclehomeproperties.xml.source oraclehomeproperties.xml
4. Run the runInstaller command with the correct CLUSTER_NODES parameter.
Ensure you use the correct HOME_NAME for each node, as it’s based on the instance name R12RAC1, R12RAC2, not the service name R12RAC
Node 1: (HOME_NAME=R12RAC1_DB__oracle_R12RAC_db_tech_st_11_1_0 )
/oracle/R12RAC/db/tech_st/11.1.0/oui/bin/runInstaller -clone \
-silent -force -nolink -waitForCompletion \
-invPtrLoc /etc/oraInst.loc \
ORACLE_HOME=/oracle/R12RAC/db/tech_st/11.1.0 \
ORACLE_BASE=/oracle/R12RAC/db/tech_st/11.1.0 \ORACLE_HOME_NAME=R12RAC1_DB__oracle_R12RAC_db_tech_st_11_1_0 \
"CLUSTER_NODES={ractrgt01,ractrgt02}" \
-J-Doracle.installer.noLink=true
The cloning of R12RAC1_DB__oracle_R12RAC_db_tech_st_11_1_0 was successful.
Please check '/etc/oraInventory/logs/cloneActionsxxxx-xx-xx_xx-xx-xxXX.log' for more details.
f) Re-check /etc/oraInventory/ContentsXML/inventory.xml
<HOME NAME="R12RAC1_DB__oracle_R12RAC_db_tech_st_11_1_0" LOC="/oracle/R12RAC/db/tech_st/11.1.0" TYPE="O" IDX="3"><NODE_LIST>
<NODE NAME="ractrgt01"/>
<NODE NAME="ractrgt02"/>
</NODE_LIST>
</HOME>
g) Repeat the above on all Target RAC nodes
h) Finally, check the inventory metadata on all nodes:
$ORACLE_HOME/OPatch/opatch lsinventory -all_nodes
21 of 23 12/18/2012 6:00 PM
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 22/23
d) RAC to non-RAC : Inventory Registration still contains the source cluster nodes
Problem: RAC to non-RAC
After successful inventory registration, the inventory.xml in the Central Inventory still contains the source Cluster nodes, when they should now reflect a single non-cluster.
Source nodes: racnode01 & racnode02
Target nodes: mynode01
On mynode01 /etc/oraInventory/ContentsXML/inventory.xml contains:
<HOME NAME="R12NORAC_DB__oracle_R12NORAC_db_tech_st_11_1_0"
LOC="/oracle/R12NORAC/db/tech_st/11.1.0" TYPE="O" IDX="30">
<NODE_LIST>
<NODE NAME="racnode01"/>
<NODE NAME="racnode02"/>
</NODE_LIST>
</HOME>
Cause:
The cause is the same as the RAC-to-RAC clone, and raised in Bug 11705106. The runInstaller should include "CLUSTER_NODES={}" for RAC to Single clones.
The "CLUSTER_NODES " parameter will ensure the oraclehomeproperties.xm l inside the Local Home inventory ($ORACLE_HOME/inventory/ContentsXML) will be updated when run on each "LOCAL_NODE ".
Solution:
On the target database node, de-register the ORACLE_HOME, and re-run the correct runInstaller command manually.
a) Run detachHome.sh
$ $ORACLE_HOME/oui/bin/detachHome.sh
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.
Actual 4909 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /etc/oraInventory
'DetachHome' was successful.
b) Run the runInstaller command with the CLUSTER_NODES={}” parameter
/oracle/R12NORAC/db/tech_st/11.1.0/oui/bin/runInstaller \
-clone -silent -force -nolink -waitForCompletion \
-invPtrLoc /etc/oraInst.loc ORACLE_HOME=/oracle/R12NORAC/db/tech_st/11.1.0 \
ORACLE_BASE=/oracle/R12NORAC/db/tech_st/11.1.0 \
ORACLE_HOME_NAME=R12NORAC_DB__oracle_R12NORAC_db_tech_st_11_1_0 \
"CLUSTER_NODES={}" \-J-Doracle.installer.noLink=true
22 of 23 12/18/2012 6:00 PM
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=TROUBLESHOOT...
7/29/2019 Troubleshooting RAC RapidClone issues with Oracle Application R12
http://slidepdf.com/reader/full/troubleshooting-rac-rapidclone-issues-with-oracle-application-r12 23/23
The cloning of R12NORAC_DB__vm3_R12NORAC_db_tech_st_11_1_0 was successful.
Please check '/etc/oraInventory/logs/cloneActionsxxxx-xx-xx_xx-xx-xxXX.log' for more details.
c) Re-check /etc/oraInventory/ContentsXML/inventory.xml
<HOME NAME="R12NORAC_DB__oracle_R12NORAC_db_tech_st_11_1_0"
LOC="/oracle/R12NORAC/db/tech_st/11.1.0" TYPE="O" IDX="30"/></HOME>
References
NOTE:406982.1 - Cloning Oracle Applications Release 12 with Rapid Clone
NOTE:559518.1 - Cloning Oracle E-Business Suite Release 12 RAC-Enabled Systems with Rapid Clone
NOTE:603104.1 - Troubleshooting RapidClone issues with Oracle Applications R12
Related
Products
Oracle E-Business Suite > Applications Technology > Lifecycle Management > Oracle Applications Manager
Keywords
ADCFGCLONE.PL; TROUBLESHOOT; ADCLONE.PL; ADCLONECTX.PL; ADCONFIG.SH; ADPRECLONE.PL; E-BUSINESS; RAC
Errors
OUI-25023; ORA-38760; ORA-1103; RMAN-6054; RMAN-3002; RMAN-569; RMAN-571; 06054 ERROR
Back to top
Copyright (c) 2007, 2010, Oracle. All rights reserved. Legal Notices and Terms of Use | Privacy Statement
23 of 23 12/18/2012 6:00 PM