CookBook V1.10 Oracle 10g RAC AIX V5L

Installing Oracle 10g RAC on IBM

Quick Installation Guide Oracle 10g RAC

on IBM pSeries

on AIX 5L

Version 1.10 April 2005

pSeries with AIX 5L page 1/ 1

Document history :

Version Date Update Who Validated by 1.00

March 2005

Creation

Frederic Michiara Thierry Plumeau

Fabienne Lepetit Paul Bramy Alain Benhaim

1.10

April 2005

Updates Add Add chapter « Oracle necessary patches to apply » Add chapter «Synchronize the System Time on Cluster Nodes »

Frederic Michiara

Contributors :

o Thierry Plumeau : IBM France (EMEA ORACLE/IBM Joint Solutions Center)

o Paul Bramy : Oracle Corp. (EMEA ORACLE/IBM Joint Solutions Center)

o Alain Benhaim : IBM France (EMEA ORACLE/IBM Joint Solutions Center)

o Fabienne Lepetit : Oracle France (EMEA ORACLE/IBM Joint Solutions Center)

o Rick Piasecki : IBM US

o Frederic Michiara : Oracle France (EMEA ORACLE/IBM Joint Solutions Center)

o Michel Passet : IBM France Contact :

o EMEA ORACLE / IBM Joint Solutions Center [email protected]

Installing Oracle 10g RAC on IBM pSeries with AIX 5L page 2/ 2

1 The aim of this document ................................................................................. 5 2 What’s new......................................................................................................... 6 3 Hardware architecture....................................................................................... 7

3.1 Network architecture example with 2 nodes............................................................................7 4 Installation steps ............................................................................................... 8

4.1 Oracle CRS on raw devices and database on ASM................................................................8 4.2 Oracle CRS Data and database on GPFS ............................................................................10 4.3 Oracle CRS Data and database on raw devices part of an HACMP volume group resource12

5 CHECK LIST to use and follow....................................................................... 14 6 Preparing the system ...................................................................................... 15

6.1 Hardware requirements .........................................................................................................15 6.2 Software requirements...........................................................................................................15

6.2.1 AIX Requirements...........................................................................................................15 6.2.2 Oracle Software Requirements ......................................................................................20

6.3 Concurrent disks requirements..............................................................................................21 6.3.1 Raw Disks for CRS/Voting disks (ASM implementation) ...............................................22 6.3.2 Disks for ASM .................................................................................................................24 6.3.3 Disks for HACMP implementation ..................................................................................25

6.3.3.1 Creating a concurrent volume group.......................................................................25 6.3.3.2 Creating the concurrent logical volumes .................................................................30

6.4 Users and Groups..................................................................................................................34 6.5 Configure Kernel Parameters and Shell Limits......................................................................35

6.5.1 Configure Shell Limits.....................................................................................................35 6.5.2 Configure System Configuration Parameters.................................................................36 6.5.3 Configure Network Tuning Parameters ..........................................................................36

6.6 NETWORK CONFIGURATION .............................................................................................39 6.7 ORACLE environment setup..................................................................................................40

7 GPFS Implementation ..................................................................................... 41 7.1 INSTALLING GPFS ...............................................................................................................41 7.2 Creating a RSCT peer domain ..............................................................................................42 7.3 Creating a GPFS cluster ........................................................................................................43 7.4 Creating a GPFS filesystem...................................................................................................45

8 HACMP implementation .................................................................................. 48 8.1 Installing HACMP...................................................................................................................48 8.2 Post install tasks ....................................................................................................................48 8.3 Creating the cluster HACMP..................................................................................................49 8.4 Creating the resource HACMP ..............................................................................................58 8.5 STARTING HACMP...............................................................................................................62

9 Synchronize the System Time on Cluster Nodes ......................................... 66 10 Important TIPS for Oracle software and patchs Installation Instructions67

10.1 10g Installation on Aix 5.3, Failed with Checking operating system version must be 520067 10.2 10g recommended steps before installation and applying any Patch Set on AIX .............68

11 Install the Custer Ready Services (CRS) .................................................... 69 12 Install the Oracle 10g software.................................................................... 81

12.1 VIP Configuration Asssitant................................................................................................86 13 Oracle patches to apply ............................................................................... 93

13.1 Patch to apply on top of 10.1.0.2........................................................................................93 13.2 Patch to apply on top of 10.1.0.3........................................................................................94

14 CREATING THE DATABASE USING DBCA ................................................ 99 14.1 Database creation On GPFS .......................................................................................... 102 14.2 Database Creation On ASM............................................................................................ 107 14.3 Database Creation On Raw Devices .............................................................................. 116

14.3.1 Content example of Raw device DB configuration file ................................................ 116 Installing Oracle 10g RAC on IBM pSeries with AIX 5L page 3/ 3

14.4 Manual Database Creation ............................................................................................. 118 14.4.1 Tips to create a database in 10g Real Application Clusters. ................................... 118 14.4.2 Configure listener.ora / sqlnet.ora / tnsnames.ora ...................................................... 120 14.4.3 Configure Oracle Enterprise Manager......................................................................... 120

15 Grid control installation ............................................................................. 122 15.1 Grid Control Management Server Installation................................................................. 122 15.2 Metalink Note................................................................................................................... 138 15.3 Grid Control Agent Deployments .................................................................................... 139

16 Appendix A : Logical volumes creation.................................................... 146 17 Appendix B : examples of configuration files.......................................... 147

17.1 Network ........................................................................................................................... 147 17.2 listener.ora and tnsnames.ora Configuration example ................................................... 149

18 Appendix C : Oracle technical notes ........................................................ 150 18.1 CRS and 10g Real Application Clusters ......................................................................... 150 18.2 About RAC ...................................................................................................................... 160 18.3 About CRS ...................................................................................................................... 161 18.4 About VIP ... .................................................................................................................... 162 18.5 About manual database cration ... .................................................................................. 162 18.6 About Grid Control .......................................................................................................... 162 18.7 About TAF ... ................................................................................................................... 162 18.8 About Adding/Removing Node ... .................................................................................... 163 18.9 About ASM ... .................................................................................................................. 163 18.10 Note #2064876.102 : How to setup High Availability Group Services (HAGS) on IBM AIX/RS6000. .................................................................................................................................. 165 18.11 Note # 115792.1 , explaining how to setup HACMP cluster interconnect adapter. ........ 167

19 Appendix D : HACMP Cluster verification output .................................... 168 20 Appendix F : Filesets to be installed on the machines of the cluster. ... 171

20.1 RSCT 2.3.4 (provided with AIX 5.2) for all implementation............................................. 171 20.2 When implementing HACMP........................................................................................... 172

20.2.1 HACMP 5.1 filesets...................................................................................................... 172 20.2.2 RSCT 2.3.4 filesets for HACMP Implementation......................................................... 172

20.3 For GPFS implementation............................................................................................... 172


1 THE AIM OF THIS DOCUMENT This document is written to provide help installing Oracle10g Real Application Clusters (10.1) release 1 on IBM pSeries servers with AIX 5L.

We will describe step by step three different architecture:

- Oracle CRS (Cluster Ready Service) on raw devices and database on ASM (Automatic Storage Management), - Oracle CRS Data and database on GPFS (General Parallel File System), - Oracle CRS Data and database on raw devices part of an HACMP volume group resource.

With these three different implementations, you’ll be able to install some other combinations.

Metalink (http://metalink.oracle.com/metalink/plsql/ml2_gui.startup) Titre Origine Référence

Oracle Real Application Clusters Installation and Configuration Guide - 10g (10.1) for AIX-Based Systems

Oracle Metalink B10766-08

Oracle Real Application Clusters Administrator's Guide - 10g Release 1 (10.1)


Oracle Database Release Notes – 10g Release 1 (10.1.0.2.0) for AIX-Based Systems


Oracle Universal Installer Concepts Guide Release 2.2

Oracle Metalink A96697-01

Oracle Metalink

GPFS for AIX5L – AIX Clusters Concepts, Planning and Installation Guide

IBM GA22-7895-01

GPFS for AIX5L – AIX Clusters Administration and Programming Reference

IBM SA22-7896-01

GPFS on AIX Clusters – High Performance File System – Administration Simplified

IBM SG24-6035-00

The information contained in this paper resulted from :

- Oracle and IBM documentations - Workshop experiences done in the Oracle/IBM Joint Solutions Center - Benchmarks and POC implementations for customers performed by EMEA PSSC Montpellier - This documentation is a joint effort from Oracle and IBM specialists.

Please also refer to Oracle online documentation for more information : http://docs.oracle.com http://tahiti.oracle.comhttp://technet.oracle.com/docs/products/oracle9i/doc_library/release2/index.htmhttp://otn.oracle.com/products/oracle9i/content.html Oracle RAC home page : http://www.oracle.com/database/rac_home.htmlFor HACMP Documentation refer to : http://www-1.ibm.com/servers/eserver/pseries/library/hacmp_docs.htmlFor GPFS Documentation refer to : http://www-1.ibm.com/servers/eserver/pseries/library/gpfs.html

For more information : http://www-1.ibm.com/servers/eserver/pseries/software/sp/gpfs_faq.html


http://metalink.oracle.com/metalink/plsql/ml2_gui.startup

http://tahiti.oracle.com/

http://technet.oracle.com/docs/products/oracle9i/doc_library/release2/index.htm

http://www.oracle.com/database/rac_home.html

Your comments are important for us. We want our technical papers to be as helpful as possible.

Please send us your comments about this document to the EMEA Oracle/IBM Joint Solutions Center.

Use our email address :

[email protected] or our phone number :

+33 (0)4 67 34 67 49

2 WHAT’S NEW As shown is the figure below, the main difference between Oracle 9i RAC and 10g RAC is the clusterware. The three main implementations are:

- Oracle CRS Data and database on raw devices part of an HACMP volume group resource, - Oracle CRS (Cluster Ready Service) on raw devices and database on ASM (Automatic Storage Management), - Oracle CRS Data and database on GPFS (General Parallel File System).

Business Unit or Product Name

© 2004 JSCIBM / ORACLE Joint Solution Center | Workshop

New features Oracle10g Implementations choices

History for Oracle 9iRAC / AIX :

HACMP was mandatory with Oracle 9iRAC2 implementations types :

RAC / HACMP for database in concurrent Raw DevicesRAC / HACMP / GPFS pour database en cluster files system

10gRAC and AIX :

HACMP not mandatory3 implementations types :

10gRAC et HACMP / Database in concurrent raw devices (Only case for HACMP to be mandatory)10gRAC et ASM (Oracle Automated Storage Management) withoutHACMP nor GPFS10gRAC et GPFS (Without HACMP) / Database on cluster files system


3 HARDWARE ARCHITECTURE The cluster, is composed of three partitions on an IBM pSeries 595 using AIX 5L.

A private network (for instance a gigabit ethernet network, using a gigabit switch to link each cluster nodes) is designed only for Oracle interconnect use (cache fusion between instances). This dedicated network is mandatory.

Gigabit switch is mandatory even for only 2 nodes architecture for production (Cross-over cable can be used only for test purpose, and it’s not supported by Oracle Support, please read RAC FAQ on http://metalink.oracle.com). This network can also be used for GPFS. A second gigabit ethernet interconnect, with a different network mask, can be setup for security purposes or performance issues.

Oracle 10g RAC code is installed on the three machines of the cluster on internal disks. Data for CRS and the database are stored on DSX000 disks connected to all the nodes. All the disks must be physically attached to the three nodes. Because theses disks are connected on all the nodes, all the disks can be accessed by both of the nodes, in a concurrent mode. For this test we have used a IBM DS4000.

The cluster, named rac10g_cluster, is composed of three IBM pSeries using AIX 5L.

The IBM AIX clustering layer, HACMP filesets, should not be installed if you’ve chosen an implementation without HACMP. If this layer is implemented for other purpose, raw devices necessary to install and run CRS data will have to be part of an HACMP volume group resource.

3.1 NETWORK ARCHITECTURE EXAMPLE WITH 2 NODES


http://metalink.oracle.com/

4 INSTALLATION STEPS 4.1 ORACLE CRS ON RAW DEVICES AND DATABASE ON ASM

Hardware requirements (chap. 6.1)

Software requirements (chap. 6.2)

Disks for CRS on raw-disks (chap. 6.3.1)

Disks for ASM (chap. 6.3.2)

Users and Groups (chap. 6.4)

Network configuration (chap. 6.6)

Oracle environment setup (chap. 6.7)

Configure Kernel Parameters and Shell Limits (chap. 6.5)

Sy

End on next page ....

nchronize the System Time on Cluster Nodes (chap. 9)


Install the Cluster Ready Services (chap. 11)

Install the Oracle 10g software (chap. 12)

VIPCA Utility (chap. 12.1)

Creating the DB using DBCA (chap. 14 / 14.2) ASM Database


4.2 ORACLE CRS DATA AND DATABASE ON GPFS


GPFS Implementation (chap. 7)



Disks for CRS on raw-devices (chap. 6.3.1)


ork configuration (chap. 6.6)


(chap. 6.5)

stem Time on Cluster Nodes (chap. 9) Synchronize the Sy

Configure Kernel Parameters and Shell Limits

Netw


Creating the DB using DBCA (chap. 14/14.1)

GPFS Database

Install the Oracle 10g software (chap. 12)


Install the Cluster Ready Services (chap. 11) OCR and Voting Disks on HACMP


4.3 ORACLE CRS DATA AND DATABASE ON RAW DEVICES PART OF AN HACMP VOLUME GROUP RESOURCE




HACMP Implementation (chap. 8)

Disks for HACMP implementation (chap. 6.3.3)

Synchronize the System Time on Cluster Nodes (chap. 9)


Network configuration (chap. 6.6)


Configure Kernel Parameters and Shell Limits (chap. 6.5)



Install the Cluster Ready Services (chap. 11) OCR and Voting Disks on HACMP ressource

are ( hap. 12)

Creating the DB using DBCA chap 14 / 4.3)Raw-Device

( . 1

Install the Oracle 10g softw c


5 CHECK LIST TO USE AND FOLLOW

d do, before moving to Oracle Installation steps :

on each node : Yes/No ?

This is the list of operations you shoul

Done

Operations

Node 1

Node 2

Node 3

Node 4

Node 5

...

1 • Check the Hardware Requirements

2 • Check the Network Requirements

3 • Check the Software Requirements

4 equired UNIX Groups and User• Create R

5 • Configure Kernel Parameters and Shell Limits

6 • Identify Required Software Directories

7 • Identify or Create an Oracle Base Directory

8 ry• Create the CRS Home Directo

9 • Choose a Storage Option for Oracle CRS, Database, and Recovery Files

11 for Oracle CRS, Database, or • Create DirectoriesRecovery Files

12 • Configure Disks for Automatic Storage Management

• Or Configure Raw Devices And Verify the Cluster Software Configuration

• Or Configure GPFS for Shared Files System

And ACCESS Disk Storagall nodes participating to the R

e concurently from AC cluster

13 • Synchronize the System Time on Cluster Nodes

14 • Stop Existing Oracle Processes

15 • Configure the oracle User's Environment




6

6.1 RE

- RAM

- Internal disk >= 4 GB for the oracle code - P ck the ag

lsps –a

ace: The Oracle Universal Installer requires up to 400 MB of free e in th ory ck the free temporary space available:

tmp

u other file s of /tmp. Set the TEMP environment variable (used by Oracle) MPD e t variable to the new location. For example:

export TEMP=/nexport TMPDIR=

- CD-ROM drive.

FTWARE REQUIREMENTS

6.2.1 AIX Requirements

To have the latest information please refer to Metalink Note 282036.1. on http://metalink.oracle.com

PREPARING THE SYSTEM

HAR WARED QUIREMENTS

>= 512 MB. Command to check the physical memory: lsattr –El sys0 –a realmem

aging space = 2 x RAM, with a minimum of 400 MB and a maximum of 2 GB. To chep ing space configured :

- Temporary Disk Spspac

dfe /tmp direct–k /

. To che

You can se an sy tem instead and the T IR nvironmen

ew_tmp /new_tmp

6.2 SO

, this document include last update which is Version 7 (March 29, 2005)

Check that the required software and patches are installed on the system.

Check for Required Software

Depending on the products that you intend to install, verify that the following software is installe nrequiremen

d o the system. The procedure following the table describes how to check these ts.

AIX release supported with Oracle 10g RAC : • AIX 5L Version 5.2 ML04 and higher

Note: Maintenance Level 04 (ML04) contains both: 1) Recommended maintenance updates for all AIX 5L v5.2 installations. 2) Support code for POWER5 processor-based servers

• AIX 5L Version 5.3 and higher


To determine which version of AIX is installed, enter the following command: # oslevel -r

n is lower than AIX 5.2.0.0 Maintenance Level 4 (520 -04),

sion 5.2 maintenance packages are available from the following Web site: h /www-912

If the operating system versio 0upgrade your operating system to this level.

AIX 5L verttp:/ .ibm.com/eserver/support/fixes/

Filesets

• bos.adt.base • bos.adt.lib • bos.adt.libm • bos.perf.libperfstat • bos.perf.perfstat

bos.perf.proctools • rsct.basic.rte

bos.adt.base bos.adt.lib

bos.adt.libm bos.perf.libperfstat bos.perf.perfstat

bos.perf.proctools rsct.basic.rte

ther the required filesets are installed and committed, enter a to the following:

m itted, then install it. Refer to your operating system or software ut installing filesets.

AIX filesets required Os Release

Maintenance Level

AIX 5.2 ML 4 Minimum And higher

•

AIX 5.3

To ensure that the system meets these requirements, follow these steps:

• To determine whecommand similar

# lslpp -l bos.adt.base bos.adt.lib bos.adt.libm bos.perf.perfstat \ bos.perf.libperfstat bos.perf.proctools rsct.basic.rte

If a fileset is not installed and co mdocumentation for information abo

Check for Required Patches


Depending on the products thinstalled on the system. The prequirements.

AIX Patches (APAR) n(Note: If the PTF is not down

at you intend to install, verify that the following patches are rocedure following the table describes how to check these

eeded loadable, customers should request an efix through AIX customer support.) ALL (ASM, GPFS, RAW)

or all implementations (ASM, GPFS, RAW) IY43980: libperfstat.h not ANSI-compliant IY44810: DSI IN BMRECYCLE IY45462: Definition of isnan() in math IY45707: J2 READAHEAD/CIO INTERACTION IY46214: dropping partial connections leaves IY46605: exec of 32 bit application can fail on IY48525: SDK 1.4.1 32-BIT SR1: CA141-20030930 IY51801: race condition in aio_nwait_tim IY56024: CIO WRITE RETURNS INCORRECT LENGTH The following program technical fix (PTF) is required if you intend to use ASM for

database file storage: U496549: bos.rte.aio.5.2.0.15 IY64978 : Possible system hang while concurrently renaming and unlinking

under JFS. This APAR is currently available from the Fix Central download Web site locate

http://www-1.ibm.com/servers/eserver/support/pseries/aixfixes.html IY63366 : Loader may fail to find symbol even though the symbol is present in

to APAR availability, an emergency fix is available at: ftp://service.software.ibm.com/aix/efixes/iy63366/

the symbol table. This can cause applications that use dynamically loaded modules to fail. Prior

IY59082 : Heavily loaded systems running JFS2 file systems may hang. AIX lesets (bos.mp, bos.mp64, bos.up) at the 5.2.0.40

y the fix for this APAR.

ore of the following emergency

1 or later should install APAR IY64691. ith the chvg-B command that can cause data hich were converted from normal volume

5.2 systems with the kernels filevel, and using JFS2 file systems should appl

In addition, customers may need to install one or mfixes:

1. Systems running bos.rte.lvm 5.2.0.4APAR IY64691 fixes a problem wcorruption on Big volume groups wgroups. Prior to APAR availability, obtain the emergency fix for APAR IY64691 from: ftp://service.software.ibm.com/aix/efixes/iy64691/

2. Systems running bos.rte.lvm 5.2.0.50 should install APAR IY65001. APAR IY65001 fixes a possible corruption issue with mirrored logical volumes. This APAR also contains the fix for APAR IY64691. Prior to APAR availability, obtain the emergency fix for APAR IY65001 from:ftp://service.software.ibm.com/aix/efixes/iy65001/

3. Systems running bos.rte.aio 5.2.0.50 should install APAR IY64737. APAR

Os Release

Maintenance Level

5.2 ML 4 Minimum And higher

F

.h incorrect

them on so_q0 64 bit kernel

eout

d at:


IY64737 fixes a problem where applications that use Asynchronous I/O (AIO) can cause a system hang. Prior to APAR availability, obtain the emergency fix for APAR IY64737 from: ftp://service.software.ibm.com/aix/efixes/iy64737/

------------------------------------------------------------------------------------------ ---------

Note:Oracle CRS or database file storage,

ged nodes recycle IY43602: TO BE LOST IS NOT IY45695:

-----------------------------------------------------------------------------------------------------

Note:

GPFS 2.3 plus APAR IY63969 - or later) 2.3.0.1

Current GPFS 2.1 and 2.2 customers are encouraged to upgrade to GPFS

2806.1

IY60 ory.

IY54739: GPFS 2.2 mandatory service

U489058: mmfs.base.cmds.3.5.0.6

PTFs required for VisualAge C compiler (If needed) : f you

----------------------------------------------------------------------------------------------------- A ersion are installed:

055

--APARs and PTFs required for HACMP:

HACMP is required only if you want to use raw logical volumes for however, it is supported for all

installations. IY42783: CT:LX: RMC daemon may hang if mana

DISK FAILURES CAUSING QUORUM BASE FIXES FOR HACMP 5.1.0

U496124: cluster.es.server.rte.5.1.0.2

General Parallel File System (GPFS) GPFS is required only if you want to use a cluster file system for

Oracle CRS or database files. GPFS 2.3.0.1 (All new Oracle and GPFS customers on AIX 5L v5.2 should use GPFS– or later.

2.3.0.1 – or later. For more information regarding GPFS 2.3 and Oracle, see Metalink Note: 30 If installing GPFS on AIX 5L Version 5.2 ML04, the AIX PTF for APAR

609 is mandat APARs required for GPFS v2.2: GPFS 2.2.1.1 (GPFS PTF 6)

-OR-

APARs and PTFs required for GPFS v2.1: IY52454: DirectIO fixes for Linux, Inerop: backup extended

U496347: mmfs.gpfs.rte.3.5.0.10 U496395: mmfs.gpfs.rte.2.1.0.10

-----------------------------------------------------------------------------------------------------

U489726: vac.C.6.0.0.4 (or later) Note: These APARs are required only iare using the associated JDK version.

PAR required for JDK depending on which v JDK 1.4.1.1 (64-bit): IY48526 JDK 1.3.1.11 (32-bit): IY47


JDK 1.2.2.18: IY40034

eeded adable, customers should request an efix throPAR

For all implementations (ASM, GPFS, RAW)

IY59386 "libdepend.mk files are all empty" IY60930 Unable to delete network routes

AIX Patches (APAR) n(Note: If the PTF is not downlo ugh AIX customer support.) Os Release

Maintenance Level

A

5.3 And higher IY58143 "REQUIRED UPDATE FOR AIX 5.3"

IY66513 LDR_CNTRL TURNS ON UNDESIRABLE OPTION WHEN INITIALIZED WITH INCORRECT VALUE (customers should efix hrough

----------------------------------------------------------------------------------------------------

umes for CRS or database file storage, however, it is supported for all

n)

CT GW ADDR FOR DEFAULT ROUTE AFTER ALIAS DELETE

91: THREADS BLOCKED IN _GLOBAL_LOCK_COMMON

HACMP V5.2 plus (RAW Devices implementation): TO SUPPORT

HACMP V5.1 plus (RAW Devices implementation): U498114 and all co-requisite PTFs.

---------------------------------------------------------------------------------------

File System (GPFS) e a cluster file system for

AIX customer support)

---

Note: HACMP is required only if you want to use raw logical volOracleinstallations.

For either HACMP 5.2 or HACMP 5.1 (RAW Devices implementatio IY61034: INCORRE

IY61770: MISC SERVICE UPDATES IY621

IY60759: "ODM PERMISSION CHANGES REQUIRED ORACLE 9i"

----------------

General Parallel Note: GPFS is required only if you want to usOracle CRS or database files. GPFS (Cluster files system implementation) : For latest GPFS information, see: http://publib.boulder.ibm.com/clresctr/library/gpfs_faqs.html

GPFS 2.3.0.1 (GPFS 2.3 plus APAR IY63969 - or later)

All Oracle and GPFS customers on AIX 5L v5.3 should use GPFS 2.3.0.1

For more information regarding GPFS 2.3 and Oracle, see Metalink Note: 302806.1

-OR- GPFS 2.2.1.0 (GPFS PTF 5) plus: IY59339: "Misc Service Updates"

-OR- Installing Oracle 10g RAC on IBM pSeries with AIX 5L page 19/ 19

GPFS 2.2.1.1 (GPFS PTF 6)

---------------

PTFs required for VisualAge C compiler (If needed) :

---------------------------------------------------------------------------------------- APAR required for JDK depending on which version are installed: IBM JDK 1.3.1.11(32 bit) IY47055 IBM JDK 1.4.1.2 (32 bit) IY47536 IBM JDK 1.4.1.1 (64 bit) IY47538

-------------------------------------------------------------------------------------------------------

VAC 6 C and C++ for AIX July PTF (6.0.0.4) U489726

meets these requirem

• To ensure that the system ents, follow these steps: To determine whether an APAR is installed, enter a command wing:

fix -i -k "IY43980 IY44810 IY45462 IY457 7 IY46214 IY46605 \

, download it from the following Web site and install it: bm.com/eserver/support/fixes/

similar to the follo# /usr/sbin/inst IY48525 IY51801 IY56024"

0

If an APAR is not installedhttp://www-912.i To determine whether a PTF is installed, enter a command similar to the following:

85561 ...

If a PTF is not installed, download it from the following Web site and install it: /eserver/support/fixes/

# lslpp -l -B U489726 U4

http://www-912.ibm.com

6.2.2 Oracle Software Requirements

Orale CD’s needed for the RAC installation :

• Oracle Database 10g Release 1 (10.1.0.2)

rvices Release 1 (10.1.0.2) for AIX5L Based Systems (1 CD)

Enterprise/Standard Edition for AIX5L Based Systems ()

• Oracle Cluster Ready Se


6.3 ONCURRENT DISKS REQUIREMENTS C

Choose a Storage Option for Oracle CRS, Database, and Recovery Files The following table shows the storage options supported for storing Oracle Cluster Ready Services (CRS) files, Oracle database files, and Oracle database recovery files. Oracle database files include datafiles, control files, re o log files, the server parameter file, and the password file. Oracle CRS files include the Oracle Cluster R stry (OCR) and the CRS voting disk.

ant to use for Oracle CRS files and Oracle databa ackups during the installation, you must also choose the storage option

for RAC

http://metalink.oracle.com

degi

For all installations, you must choose the storage option that you wse files. If you want to enable automated b that you want to use for recovery files (the flash recovery area).

Note: For the most up-to-date information about supported storage options installations, refer to the Certify pages on the OracleMetaLink Web site:

ported File Types Sup

Storage Opt CRS Database Recovery ion

Autom Yes atic Storage Management No Yes

Cluster file system

Note: R

Yes Yes Yes equires GPFS

equires HACMP

raw disk devices Yes Yes

the following guidelines when choosing the storage options that you want to use for each file can choose any combination of the supported storage ments listed for the chosen storage options.

Standard Edition installatiomatic Storage Management cannot be used to store Oracle CRS files, because these files m

using HACMP, you cannot use shared raw logical volumes for CRS or database fil information about how to configure disk storage before you start the installation, refer tong sections depending on your choice:

Shared raw logical volumes

Note: R

Yes Yes No

Shared No - Use type: - You options for each file type as long as you satisfy any require- For ons, ASM is the only supported storage option for database or recovery files. - Aut ust be accessible before any Oracle instance starts. - If you are not e storage. - For one of the followi

prepare RAW o To use ASM for database or recovery file storage, refer to chapters 6.3.1 to disks for OCR/Voting disks and 6.3.2 to prepare disks for ASM implementation.

o To use GPFS cluster file system for Oracle CRS, database, or recochapter 7 for GPFS implementation to prepare cluster files system for OCR/Voting disks (seen as files).

o To use raw devices (disks or logical volumes) for Oracle CRS or dat

. very file storage, refer to

abase file storage, refer to chapter 8 for HACMP installation/implementation and chapter 6.3.3 to prepare RAW devices for OCR/Voting disks .


6.3.1 Raw Disks for CRS/Voting disks (ASM implementation)

Identify two disks available for OCR and Votingg for nodes used in our cluster. The LUN fo

disks, on each node. The screen before shows the LUN mappinhelp us

r OCR disk and Voting disk have ids 1 and 2. These ids will on to identify which hdisk will be used.

B

U1.9-P1-I1/Q1-W200300A0B80C5404-

cfg –vl hdisk3

U1.9-P1-I1/Q1-W200300A0B80C5404-(500) Disk Array Device

:root-/> rsh node2

node2:root-/> lscfg –vl hdisk3

hdisk3 U1.9-P1-I1/Q1-W200300A0B80C5404-L2000000000000 3552 (500) Disk Array Device

- If these disks do not have a PVID assign one to it as follow : node1:root-/> lspv

e carefull hdisk2 on node1 is not necessary hdisk2 on node2. node1:root-/> lscfg –vl hdisk2

hdisk2 L1000000000000 3552 (500) Disk Array Device

node1:root-/> ls

hdisk3 L2000000000000 3552

node1

hdisk2 U1.9-P1-I1/Q1-W200300A0B80C5404-L1000000000000 3552 (500) Disk Array Device

node2:root-/> lscfg –vl hdisk4


hdisk0 0033c670e214eac5 rootvg

None

k5 none None

0e214eac5 rootvg

None

2 0033c670b0f3ee84 None

hdisk3 0033c670b0f3fb87 None

hdisk4 none None

hdisk5 none None

for ocr and voting disks on each nodes. ve_policy=no_reserve

node1:root-/> rsh node2

node2:root-/> chdev-l hdisk3 -a reserve_policy=no_reserve

……

- Issue the command lsattr –E –l hdisk2 to vizualize all attributes for hdisk2 - As described before, disks might have different names from one node to another for example hdisk2 on node1 might be hdisk3 on node2, etc… Create a device on each node called /dev/ocr_disk and /dev/vote_disk with the right major and minor number.

node1:root-/> ls -l /dev/*hdisk2

brw------- 1 root system 20, 4 Oct 18 17:52 /dev/hdisk2

crw------- 1 root system 20, 4 Oct 18 17:52 /dev/rhdisk2

node1:root-/> mknod /dev/ocr_disk c 20 4

node1:root-/> mknod /dev/vote_disk c 20 5


node2:root-/> ls -l /dev/*hdisk3


crw------- 1 root system 12, 9 Oct 18 16:2é2 /dev/rhdisk3

node2:root-/> mknod /dev/ocr_disk c 12,9

node2:root-/> mknod /dev/vote_disk c 12,10

……..

- Change owner, group and permission for /dev/ocr_disk and /dev/vote_disk on each node of the cluster: node1:root-/> chown oracle.dba /dev/ocr_disk

node1:root-/> chown oracle.dba /dev/vote_disk

node1:root-/> chmod 660 /dev/ocr_disk

node1:root-/> chmod 660 /dev/vote_disk

active

hdisk1 none None

hdisk2 none

hdisk3 none None

hdisk4 none None

hdis

node1:root-/> chdev-l hdisk2 -a pv=yes

node1:root-/> chdev-l hdisk3 -a pv=yes

node1:root-/> lspv

hdisk0 0033c67active

hdisk1 none

hdisk

- Change the “reserve_policy” attribute to “no_reserve” node1:root-/> chdev-l hdisk2 -a reser



node1:root-/> ls –l /dev/* | grep “20, 4”


brw------- 1 root system 20, 4 Oct 18 17:52 /devr/hdisk2

crw-rw---- 1 oracle dba 20, 4 Oct 19 13:27 /dev/ocr_disk

node1:root-/> ls –l /dev/* |grep “20, 5”


crw------- 1 root system 20, 5 Oct 18 17:52 /dev/rhdisk3

crw-rw---- 1 oracle dba 20, 5 Oct 19 13:28 /dev/vote_disk


node2:root-/> chown oracle.dba /dev/ocr_disk

…….

6.3.2 Disks for ASM

- Identify two disks available for ASM( see § 6.3.1


Raw Disks for CRS/Voting disks) :

dedicated to ASM, on each nodes of the cluster: node1:root-/> chdev-l hdisk6 -a pv=yes


….

- Change owner, group and permission for ASM disks on each node of the cluster: node1:root-/> chown oracle.dba /dev/rhdisk6

node1:root-/> chown oracle.dba /dev/rhdisk7


node2:root-/> chown oracle.dba /dev/rdisk6

node2:root-/> chown oracle.dba /dev/rhdisk11

…….

6.3.3 Disks for HACMP implementation 6.3.3.1 Creating

node1:root-/> lsdev -Ccdisk

- If these disks do not have a PVID, assign one to it and change the “reserve_policy” attributes for each disks

a concurrent volume group

The database files and/or ocr and voting disk are stored on external concurrent disks.




HACMP running

X X X

X X X

Creating a concurrent Volume Group (VG)

Create vg

X

ON

OFF

Unk

now

n

Node 1 VG mode

Node 2 VG mode

Action

Varyon X

X

On

As

cline

onc.

X

Def

ined

A

s co

nc.

ON

OFF

Unk

now

n

X

Varyoff x X X

X

Import X

X xx

X

Varyon X X

X X

Varyoff X X

Start HACMP

ne

onc.

ned

onc.

ne

nc.

X X

Def

ined

A

s co

nc.

Onl

ine

As

conc

.

Def

inA

s c

ed

onc.

ON

Onl

iA

s c

Def

iA

s c O

N

Onl

iA

s co

- Che tween the servers of the cluster and identify which one will ck if the target disks are physically shared bebe used, as shown in § 6.3.1


Raw Disks for CRS/Voting disks .

Remenber the disk name can be different, depending on the others disks connected on each machine.

Create at the AIX level on the first machine (node1) a concurrent volume group, oradatavg: node1:root-/> smit vg Add a Volume Group Type or select values in entry fields. Press Enter AFTER making all desired changes. [Entry Fields] VOLUME GROUP name [oradatavg] Physical partition SIZE in megabytes 64 + * PHYSICAL VOLUME names [hdisk4] + Activate volume group AUTOMATICALLY no See Note 1 + at system restart? Volume Group MAJOR NUMBER [80] See Note 2 +# Create VG Concurrent Capable? yes + Auto-varyon in Concurrent Mode? no +

Note 1: Never choose YES to activate at system restart, neither auto-varyon in concurrent mode. These sks have to be managed by HACMP. The volume group just has to be created with concurrent capability.

Note 2: You must choose the major number to be sure the volume groups have the same major number on ll the nodes (caution, before choosing this number, you must be sure it is free on all the nodes). The vlstmajor command lists the first free major number on a machine.

The major number for oradatavg volume group is 80. node1:root-/> ls –al /dev/oradatavg

crw-rw---- 1 root system 80, 2 Oct 12 13:39 /dev/oradatavg

Start manually the volume group on node1 : node1:root-/> varyonvg oradatavg

Create all the logical volumes (raw devices) you need for your database, including raw devices for ocr and oting and shared configuration disks. Stop oradatavg on node1 :

node1:root-/> varyoffvg oradatavg

Import oradatavg volume group on the second machine (node2) node1:root-/> smit vg Import a Volume Group Type or select values in entry fields. Press Enter AFTER making all desired changes. [Entry Fields] VOLUME GROUP name [oradatavg] * PHYSICAL VOLUME name [hdisk4] + Volume Group MAJOR NUMBER [80] +# Make this VG Concurrent Capable? yes + Make default varyon of VG Concurrent? no +

he physical volume name (hdisk) could not have the same number on both sides. Check the PVID (lspv) of the isk, because it’s the only information reliable and unique thru the cluster.

e sure to have the same major number. This number has to be undefined on all the nodes.

The import of a volume group resets to YES the field “Activate volume group automatically at system start”, which has to be set to NO. To do it manually : chvg –a ‘n’ or you can also modify this flag using smit

vg

-

ta

al

-

-

- v-

-

Td

B

re


- Varyon manually the volume group on node2 :

node2:root-/> varyonvg oradatavg

Repeat the import off the oradatavg on the third node. Do not forget to stop the volume group on the previous ode.

he new volume group is now defined on the all the machines of the cluster, with the concurrent capable feature t on.

Test the availability of the volume group on the two nodes

he volume group is now defined on all the nodes of the cluster, with concurrent capability. But it is not yet aried on in concurrent mode. This is the goal of HACMP.

efore HACMP is up and running, the oradatavg volume group is defined on all the nodes, but can be varied on t one time on a single machine only. Don’t try to varyon oradatavg manually in concurrent mode, this task is evoted to HACMP only.

To list the volume groups defined on the machine : lsvg

To list the active volume groups : lsvg –o

To switch the volume group from one node to the other one, do : varyoffvg oradatavg on the machine where it is varied on varyonvg oradatavg on the other node lsvg oradatavg : to list the volume group characteristics

OLUME GROUP: oradatavg VG IDENTIFIER: 033c67000004c00000000ff87432e75 G STATE: active PP SIZE: 64 megabyte(s) G PERMISSION: read/write TOTAL PPs: 79 (5056 egabytes) AX LVs: 512 FREE PPs: 79 (5056 megabytes) LVs: 0 USED PPs: 0 (0 megabytes) OPEN LVs: 0 QUORUM: 2 TOTAL PVs: 1 VG DESCRIPTORS: 2 STALE PVs: 0 STALE PPs: 0 ACTIVE PVs: 1 AUTO ON: no Concurrent: Enhanced-Capable Auto-Concurrent: Disabled VG Mode: Non-Concurrent MAX PPs per PV: 1016 MAX PVs: 128 LTG size: 128 kilobyte(s) AUTO SYNC: no HOT SPARE: no BB POLICY: relocatable

You should have a similar output on all the nodes.

Note that at this point of the disks setup, the VG Mode is not be concurrent yet. But it must be concurrent after the starting of HACMP !

- n

Tse

-

Tv

Bad

-

-

-

V0VVmM


6.3.3.2 Creating the concurrent logical volumes Adding a Logical Volume in an active Volume Group (VG) (Via Logical Volume Manager) Mklv * x

X X X X

varyoff xx X X

X X

export X X

X

X X

* At this point this new LV is known on this node

Active VG

X

X X

Def

inA

s co

nced .

OFF

Node 1 de VG mo

ode 2

Action

Def

is

cone

d nc

. A

Varyon X X

X

Stop HACMP

X X X

Unk

now

n

ON

OFF

Unk

now

n

NVG mode

ON

Onl

ine

As

conc

.

Onl

ine

As

conc

.

X

import X X

* At this point this new LV is unknown on this node

Start HACMP




Node 1VG mode

Node 2 VG mode

D

efi

s co

nc

ON

OFF

nk Onl

iA

s co

Adding a Logical Volume in an active Volume Group (VG) (Via HACMP)

Active VG

X

X X

X X X

ned .

A U D A

now

n

ON

OFF

Unk

now

n

Action

Smit hac pm

Cluster manager Conc. LVM Conc LV Add conc LV

x X X X X X

X

efin

ed

s co

nc.

ne

nc.

Onl

ine

As

conc

.

In a raw-device implementation, each logical volume created is used as a raw-device and will represent one datafile (or redolog or control file…). We will need a raw-device for ocr disk and another one for voting disk.

ollowing raw devices :

name Purpose

See the table #1 below to have a list of logical volumes required for a minimal database implementation.

Table #1 . A minimal database implementation requires the f

LSize ogical Volume name Raw device

100 MB ocr_disk /dev/rocr_disk OCR disk 20 MB vo _disk /dev/rvote_disk Voting disk 2 MB rac_system /dev/rrac_system System

te51512 MB rac_undotbs01 /dev/rrac_undotbs1 UNDO Tablespace (instance #1) 512 rac_ MB undotbs02 /dev/rrac_undotbs2 UNDO Tablespace (instance #2) 200 MB 1 /devlog1 /rlog11 Redolog Thread #1, Group #1 200 MB log12 /dev/rlog12 Redolog Thread #1, Group #2 200 MB log21 /dev/rlog21 Redolog Thread #2, Group #1 200 MB log22 /dev/rlog22 Redolog Thread #2, Group #2

32 MB rac_control01 /dev/rrac_control01 Control File # 1 32 MB rac_control02 /dev/rrac_control02 Control File # 2 32 MB rac_control03 /dev/rrac_control03 Control File # 3 32 ac_spfile For spfile (if used) MB rac_spfile /dev/rr

/dev/rsrvconfi128 M g For srvctl tool (if used) B srvconfig MB rac_data 512 /dev/rrac_data DATA tablespace

512 /dev/rrac_index INDEX tablespace MB rac_index 256 MB rac_temp /dev/rrac_temp TEMP tablespace

es have to be dedicated on each instance.

script is provided in A gical volumes creation) to create the volumes groups and the logical olume as specified above.

datavg

he volume group, oradatavg, is build with a physical partition (PP) size (of 64MB for example). The size of the

o new entries appears in the /dev directory :

rnew_lv where the “r” stands for raw device. This is the device name to use with Oracle for a raw-vice implementation.

Important for raw-device implementation : check that the owner of /dev/rnew_lv is oracle, group dba, and that oracle have the read and write privilege.

If needed, do as root (on all the nodes) : node1:root-/> chown oracle:dba /dev/*rac_*

node1:root-/> chmod go+rw /dev/*rac_*

Once new logical volumes are created or modified on a node, the other node has to update its ODM (AIX internal repository), by retrieving the information on the concurrent disk.

node1:root-/> redefinevg –d <disk name> oradatavg

This command can also done with the following sequence : node1:root-/> varyoffvg oradatavg

node1:root-/> exportvg oradatavg

The redologs and undo logical volum

Av

ppendix A (Lo

Some samples command from this appendix : node1:root-/> mklv -y'rac_systemlv' -t'jfs2' ora

Tlogical volume is expressed in number of PP, not in Kbytes.

Once a logical volume new_lv is created on a node, tw

- /dev/new_lv which is normally used by LVM for file systems. /dev/-

de


node1:root-/> importvg –y oradatavg –V<major <diskname>

number>

node1:root-/> varyonvg oradatavg

hat all the ODM databases of all the nodes have the same level of update about the logical volumes of the concurrent disk. This point is checked by HACMP during the synchronization process.

You shouldnodacc oradatavg:

/A undotbs01 jfs2 8 8 1 open/syncd N/A

jfs2 8 8 1 open/syncd N/A jfs2 4 4 1 open/syncd N/A

4 1 open/syncd N/A rac_control01 jfs2 1 1 1 open/syncd N/A

s2 1 cd N/A

config jfs2 2 pen/syncd N/A jfs2 8

x jfs2 8 pen/syncd N/A rac_temp jfs2 4 N/A

The same command have to be executed on the others nodes, as soon as an update has been made on one node. It is important for HACMP t

have a similar output on all the nodes. e1:root-/> lsvg –l oradatavg (raw-devices implementation, ording table #1):

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT ocr_disk jfs2 2 2 1 open/syncd N/A vote_disk jfs2 2 2 1 open/syncd N/A rac_system jfs2 8 8 1 open/syncd N

undotbs02 log11 log12 jfs2 4 4 1 open/syncd N/A log21 jfs2 4 4 1 open/syncd N/A log22 jfs2 4

rac_control02 jfrac_control03 jfs2 1

1 1 open/syn1 1 open/syncd N/A

rac_srv 2 1 orac_data rac_inde

8 1 open/syncd N/A 8 1 o4 1 open/syncd


6.4 SERS AND GROUPS U

bers (203, 204 and 205 in our case) are identical thru the nodes.

- smit g o create the following groups dba rimary group for the oracle user. hags oinst

of the oracl group for oracle user.

- smit or

The orasecondary gr account.

Verifica ould be different) h u

dba:!:204:oracle

o

tion: ulimit –f ations :

e file /etc/security/limits which shows the limits for each user. The default stanza applies user to be c le can be modified by root with vi.

sh set to unlimited, except for core (e.g. –1 in the file /etc/security/limits). To turn n to , use smit users.

- Set a password to swd oracle

This setup has to be done on all the nodes of the cluster. Be sure that all the groups and user num

roup tP

user For high availability (if not already created, and if HACMP used). all The ora inventory group. This group is not mandatory. If it exists, it will be the group owner

e code files. This group is a secondary

user to create the users acle Owner of the database.

cle user must have dba as primary group, oinstall and hagsuser as secondary groups. Also add the oup hagsuser (if HACMP used) to the root

tion : check if the file /etc/group contains lines such as : (the numbers cags ser:!:203:oracle, root

install:!:205:oracle

- Check if there is some AIX default limitations (especially on the file size) : File size limita All limit ulimit –a

See also thto all new reated. This fi

The default limits ould be some user limitatio unlimited

oracle user, the same for all the nodes of the cluster, with the command : pas


6.5 C

Configuring Shell Limits, Sy ork Tuning Parameters (Extract from Ora

The parameter and shell limit values shown in this section are recommended

stem. Refer to your operating .

ONFIGURE KERNEL PARAMETERS AND SHELL LIMITS

stem Configuration, and Netwcle Documentation)

Note:

values only. For production database systems, Oracle recommends that you tune these values to optimize the performance of the sysystem documentation for more information about tuning kernel parameters

and network tuning parameters as described in this section on all cluster nodes.

6.5.1 Configu

Verify that the shell limits shown in the following table are set to the values shown. The procedure following the table describe nd set the values.

Oracle recommends that you set shell limits, system configuration parameters,

re Shell Limits

s how to verify a

Shown in smit) RecommendeShell Limit (As d Value

Soft FILE size -1 (Unlimited)

e -1 (Unlimited) Soft CPU tim Note: This is the default value.

Soft DATA d) segment -1 (Unlimite

size -1 (Unlimit

current value specified for these shell limits, an

er the following command:

Soft STACK ed)

To view the d to change them if necessary, follow these steps:

1. Ent# smit chuser

3. shown for the soft limits listed in the previous

n ting value.

4. 0 to exit.

2. In the User NAME field, enter the user name of the Oracle software owner, for exampleoracle.

Scroll down the list and verify that the valuetable is -1.

If ecessary, edit the exis

When you have finished making changes, press F1


6.5.2 Configure System Configuration Parameters

Verify that the maximum number of processes allowed per user is set to 2048 or greater :

Note: For production systems, this value should be at least 128 plus the sum of the PROCESSES and PARALLEL_MAX_SERVERS initialization parameters for each database running on the system.

1. Enter the following command: # smit chgsys

2. Verify that the value shown for Maximum number of PROCESSES allowed per user is greater than or equal to 2048.

If necessary, edit the existing value.

3. When you have finished making changes, press F10 to exit.

6.5.3 Configure Network Tuning Parameters

Verify that the network tuning parameters shown in the following table are set to the values shown or higher values. The procedure following the table describes how to verify and set the values.

Network Tuning Parameter Recommended Value

ipqmaxlen 512

rfc1323 1

sb_max 2*655360

tcp_recvspace 65536

tcp_sendspace 65536

Udp_recvspace 655360

Note: The recommended value of this parameter is 10 times the value of the udp_sendspace parameter. The value must be less than the value of the sb_max parameter.

Udp_sendspace 65536

Note: This value is suitable for a default database installation. For production databases, the minimum value for this parameter is 4 KB plus the value of the database DB_BLOCK_SIZE initialization parameter multiplied by the value of the DB_MULTIBLOCK_READ_COUNT initialization parameter:

(DB_BLOCK_SIZE * DB_MULTIBLOCK_READ_COUNT) + 4 KB

To view the current value specified for these parameters, and to change them if necessary, follow these steps :


1. To check the current values of thto the following:

e network tuning parameters, enter commands similar

2. f any parameter, enter the following command to the system is running in compatibility mode:

# /usr/sbin/lsattr -E -l sys0 -a pre520tune

ty mode, the output is similar to the following, showing 20tune attribute is enable:

e ng compatibility mode True

3. in ty mode, follow these steps to change the

n following to change the value of each

name=value

recvspace=655360

llowing to the /etc/rc.net file for each parameter vious step: then ndspace=65536

o endspace=65536 o ecvspace=65536 o 23=1 o x=2*655360

/usr/sbin/no -o ipqmaxlen=512

c/rc.net file, the values persist when the system

4. tibility mode, enter commands similar to the lues:

r len=512

/usr/sbin/no -p -o parameter=value

eter, you must restart the system.

# /usr/sbin/no -a | more

If you must change the value odetermine whether

If the system is running in compatibilithat the value of the pre5pre520tune enabl Pre-520 tuni

If the system is runn g in compatibiliparameter values:

a. Enter comma ds similar to the parameter:

# /usr/sbin/no -o parameter_

For example: # /usr/sbin/no -o udp_

b. Add entries similar to the fothat you changed in the pre

if [ -f /usr/sbin/no ] ; /usr/sbin/no -o udp_se /usr/sbin/no -o udp_recvspace=655360 /usr/sbin/n -o tcp_s /usr/sbin/n -o tcp_r /usr/sbin/n -o rfc13 /usr/sbin/n -o sb_ma

fi

By adding these lines to the /etrestarts.

If the system is not running in compafollowing to change the parameter va

• ipqmaxlen parameter: /usr/sbin/no - -o ipqmax

• Other parameter:

Note: If you modify the ipqmaxlen param

unables/nextboot file, causing the attribute values to These commands modify the /etc/t


persist when the system restarts.


6.6

Set up le rsh, rcp, rlogin commands. You should have the entries on eac ot/oracle home directory $HOME/.rhosts.

node1

node2

node3

node2_vip

rconnect RAC & GPFS

ode1_gpfs

gpfs

ot

root

n

n

n

node1_vip oracle

node2_vi

node3_vip oracle

node1_gp

node2_gpfs oracle

node3_gpfs oracle

node1:ro $HOME/.rhosts

nod

node2

node3 root

nod root

nod ot

node3_vip root

node1_gpfs root

NETWORK CONFIGURATION

user equivalence for the oracle account, to enabh node for : /etc/hosts, /etc/hosts.equiv and on ronode1:root-/> pg /etc/hosts

# Public Network

10.2.12.81

10.2.12.82

10.2.12.83

# Virtual IP address

10.2.12.181 node1_vip

10.2.12.182

10.2.12.183 node3_vip

# Inte

10.10.12.81 n

10.10.12.81 node2_gpfs

10.10.12.81 node3_

node1:root-/> pg /etc/hosts.equiv

node1 root

node2 root

node3 ro

node1_vip root

node2_vip root

node3_vip root

node1_gpfs root

node2_gpfs root

node3_gpfs

ode1 oracle

ode2 oracle

ode3 oracle

p oracle

fs oracle

ot-/> pg

e1 root

root

e1_vip

e2_vip ro


node2_gpfs root

node3_gpfs root

oracle

oracle

cle

cle

acle-/home/oracle> pg $HOME/.rhosts

root

e

1_vip oracle

e

e

e

_gpfs oracle

possible, but not advised because of security reasons, to put a “+” in hosts.equiv and .rhosts files.

Test if econdary cluster machine). You are logged on node1

oot-/> rlogin node2 (=> no password)

oot-/> rcp /tmp/toto node2:/tmp/toto

oracle

2 :oracle-/home/oracle> rsh node3 date

6.7

home directory

node1 oracle

node2 oracle

node3 oracle

node1_vip oracle

node2_vip

node3_vip

node1_gpfs oracle

node2_gpfs ora

node3_gpfs oracle

node1:root-/> su - ora

node1:or

node1 root

node2 root

node3 root

node1_vip

node2_vip root

node3_vip root

node1_gpfs root

node2_gpfs root

node3_gpfs root

node1 oracle

node2 oracl

node3 oracle

node

node2_vip oracl

node3_vip oracl

node1_gpfs oracl

node2

node3_gpfs oracle

Note : It is

the user equivalence is correctly set up (node2 is the sas root: node1:r

node2:r

node2:root-/> su -

node

node2 :oracle-/home/oracle> sdfsvdv<sdv<vd<d 2004

ORACLE ENVIRONMENT SETUP

Oracle environment : $HOME/.profile file in Oracle’s


export ORACLE_BASE=/oh10g

CLE_BASE/db10g

port ORACLE_BASE=/oh10g

PE=S

To

The or an internal disk and propagated on the other machines of the cluster. The cluster-wide installation, that is done only once. Regular file systems are

for Oracle code. This file system of 4 GB, is generally located on an t

- To l

volume group called oraclevg : 32' hdisk1

- Cre stem /oracle in the previous volume group (large file enabled) : > crfs -v jfs2 -a bf=true -g'oraclevg' -a

-a -a ag='64'

mount /oracle

e

The Oracle code can also be located on shared concurrent disks on a GPFS file-system.

7 G TATION

ation, see: ulder.ibm.com/clresctr/library/gpfs_faqs.html

export ORACLE_HOME=$ORA

export ORACLE_CRS=$ORACLE_BASE/crs

export PATH=$ORACLE_HOME/bin:$PATH

ex

export AIXTHREAD_SCO

umask 022

be done on each node.

acle code can be located onOracle Universal Installer manage theused for Oracle code.

On both nodes, create the file system in ernal disk.

ist the internal disks : node1:root-/> lsdev –Ccdisk | grep SCSI

- Create anode1:root-/> mkvg -f -y'oraclevg' -s'

ate a 4GB file synode1:root-/size='8388608' -m'/oracle' -A'yes' -p'rw' -t'no' nbpi='8192'

node1:root-/>

node1:root-/> chown oracle:dba /oracl

PFS IMPLEMEN7.1 INSTALLING GPFS

For latest IBM GPFS informhttp://publib.bo

GPFS 2.3.0.1 (GPFS 2.3 plus APAR IY63969 - or later)

e to GPFS 2.3.0.1

or more information regarding GPFS 2.3 and Oracle, see Metalink Note:

All new Oracle and GPFS customers on AIX 5L v5.2 should use GPFS 2.3.0.1 – or later.

Current GPFS 2.1 and 2.2 customers are encouraged to upgrad– or later.

F302806.1

Doc ID: Note:302806.1 Subject: IBM General Parallel File System (GPFS)

and Oracle RA AIX 5 C on L and IBM eServer

Content Type:

TEXT/X-HTML

Creation ate:

25-MAR-2005

Last Revision Date:

31-MAR-2005

pSeries D

Type: BULLETIN Status: PUBLISHED


On the idden file .toc exists. If not, when located on this directo

If you all, this file always exists on the installation media.

smit

Type F ”all latest”.

To be only before proceeding to the real install.

Filseset

paragraph “GPFS” to check that all the necessary filesets have been installed.

This appendix provides the result of the command :

Post in

Add GPFS binaries in root’s path. fs/bin

Note : the steps listed here are for a GPFS implementation with GPFS version 2.2, specificities of version 2.3 are added i .

7.2 T PEER DOMAIN

Note f er domain is not needed. Skip this paragraph and go to“6.3 Creating a GPFS cluster

- Prepare ain with the preprpnode command. You must issue the command for eac node. On node1:

preprpnode node3

> preprpnode node1

de2:/> preprpnode node3

At this n.

- Cre

Examp

Example : st –V demo10g

- Start the peer dom

ine:

directory containing all the filesets, check if the hry, make inutoc .

are using a cdrom as source for this inst

install specify the directory containing the filesets

4 to list the filesets to install, rather than choosing

sure of what you are doing, you can use the field preview

s mmfs.xx.xxx

See Ap endix H : Filesets to be installed on the machines of the cluster, p

lslpp -L | grep mmfs

stall task

export PATH=$PATH:/usr/lpp/mm

n when necessary

CREATING A RSC

or GPFS 2.3 : RSCT pe“.

the nodes to be part of the peer domh node part of the peer domain on all node1:/> preprpnode node1

node1:/> preprpnode node2

node1:/>

On node2: node2:/

node2:/> preprpnode node2

no

…..

point, all the commands are only issued once, on one of the node part of the peer domai

ate the node list file which will contain hostnames part of the peer domain.

le : node1:/> cat /var/mmfs/etc/rpd_node.list

node1

node2

node3

- Create a new peer domain.

node1:/> mkrpdomain –f /var/mmfs/etc/rpd_node.li

ain: node1:/> startrpdomain demo10g

Check the configuration and check if all nodes are onl


node1:/> lsrpdomain demo10g Name OpState RSCTActiveVersion MixedVersions TSPort GSPort

demo10g Online 2.3.3.0 NO 12347 12348

de1:/> lsrpdnode

de2 Online 2.3.3.0

7.3 C ATING A GPFS CLUSTER

y done: /usr/lpp/mmfs/bin

- Create the GPFS nodefile which will contain IP

Examp

N

no

Name Opstate RSCTVersion

node1 Online 2.3.3.0

no

node3 Online 2.3.3.0

RE

Follow these steps to configure GPFS :

- Include the path of GPFS binaries in root’s path, if not alreadexport PATH=$PATH:

hostnames of GPFS interconnect network.

le: node1:/> cat /var/mmfs/etc/node.list

node1_gpfs

node2_gpfs

node3_gpfs

ote for GPFS 2.3 : node1:/> cat /var/mmfs/etc/node.list

ter –t lc –p node1_gpfs –s node2_gpfs -n c/node.list

node1_gpfs:1:quorum

node1_gpfs:2:quorum

node1_gpfs:3:quorum

- Create the gpfs cluster : node1:/> mmcrclus/var/mmfs/et

Note for GPFS 2.3 : ster –p node1_gpfs –s node2_gpfs -n

ar/mmfs de.list –C MyCluster –A

- Creluster –

Note f

node1:/> mmcrclu/etc/no/v

ate the nodeset node1:/> mmconfig –n /var/mmfs/etc/node.list –A –C MyCD /tmp

or GPFS 2.3 : this step is not needeed (this command does not exist anymore in 2.3).

You can check the GPFS creation with the mmlscluter and the cluster configuration with mmlsconfig. In our examp

ster

information

ype: lc

: gpfs1083860813

ain name: demo10g

and: /usr/bin/rsh

/usr/bin/rcp

rvers:

le you see : node1:/> mmlsclu

GPFS cluster

GPFS cluster t

GPFS cluster id

RSCT peer dom

Remote shell comm

Remote file copy command:

GPFS system data repository se


-------------------------------------

Primary server: node1_gpfs

Secondary server: node2_gpfs

Nodes for nodeset MyCluster:

-------------------------------------------------

1 node1_gpfs 10.10.12.81 node1_gpfs



Note for GPFS 2.3 : some lines change/ disapear in the previous output

guration data for nodeset MyCluster:

-----------------

recgroup GpfsRec.MyCluster

nodeset MyCluster:

node1:/> mmlsconfig

Confi

-------------------

pagepool 80M

dataStructureDump /tmp/mmfs

multinode yes

autoload yes

useSingleNodeQuorum no

wait4RVSD no

comm_protocol TCP

clusterType hacmp

group Gpfs.set1

File systems in

-----------------------------

(none)


- Start GPFS

mmfs Subsystem has been started.

node1:/> mmstartup –C MyCluster

Wed Sep 02 17:38:26 NFT 2004: mmstartup: Starting GPFS ...

node1_gpfs: 0513-059 TheSubsystem PID is 65653.

node2_gpfs: 0513-059 The mmfs Subsystem has been started. Subsystem PID is 48626.

Node3_gpfs: 0513-059 The mmfs Subsystem has been started. Subsystem PID is 58734.


You can check the GPFS sub-system is started :

node1:

tus

aixmm 65653 active

Note f

/> lssrc –s mmfsd

Subsytem Group PID Sta

mmfsd

or GPFS 2.3 : don’t check the status of the subsystem with this command because even if GPFS is started, the output of lssrc will show “inoperative“ status.


7.4 M

- Create a “disk descriptor file” containing the disks devices that will be shared in the GPFS cluster Examp

Cr ared disk with :

You ca and mmlsnsd :

File system Disk name Primary node Backup node

-----------------------------------------------------

(free disk) gpfs1nsd node1_gpfs node2_gpfs

(free disk) gpfs2nsd node1_gpfs node2_gpfs

- Create the file-system with mmcrfs node1:/> mmcrfs /oradata_gpfs /dev/oradata_gpfs -F /var/mmfs/etc/nsddisk.list –A yes -B 512k -n 4 -N 80000

-F <list of disks and descriptors> modified from mmcrlv

-A Auto mount the file system at GPFS startup

-B Block size and Stripe size

-n estimated number of nodes

-N Number of I-Nodes

node1:/> mmlsnsd

File system Disk name Primary node Backup node

-----------------------------------------------------

/dev/oradata_gpfs gpfs1nsd node1_gpfs node2_gpfs

/dev/oradata_gpfs gpfs2nsd node1_gpfs node2_gpfs

node1:/> cat /etc/filesystems | grep -p /oradata_gpfs

/oradata_gpfs:

dev = /dev/oradata_gpfs

vfs = mmfs

nodename = -

mount = false

type = mmfs

account = false

- Mount the newly created file-system : node1:/> mount /oradata_gpfs

CREATING A GPFS FILESYSTE

le : node1:/> cat /var/mmfs/etc/nsddisk.list

hdisk12:node1_gpfs:node2_gpfs:dataAndMetadata:

hdisk15:node1_gpfs:node2_gpfs:dataAndMetadata:

eate the network sh- mmcrnsd command node1:/> mmcrnsd –F /var/mmfs/etc/nsddisk.list

-F <list of disks to use with descriptors>

n list the nsd that were created by commnode1:/> mmlsnsd


- Set the permissions and the ownership of the file-systems

fs

.

node1:/> chown oracle:dba /oradata_gp

node1:/> chmod go+rw /oradata_gpfs

Note :

- GPFS block size should be around (256K, 512K, 1024K) - GPFS block size is the file system stripe size. - Not that important for regular database I/O since Direct I/O is used. - Very important for operations that increase data file size

Note for GPFS 2.3 : if you have a 2-nodes cluster you must add a tie-breaker disk. So, you should have kept a disk dedicated to the tie-breaker.

To do this:

- create a disk descriptor file : node1:/> cat /var/mmfs/etc/nsddisk_tiebreaker.list

s/etc/nsddisk_tiebreaker.list

ig tiebreakerDisks=”<name_of_the_disk>”

mand)

hdisk17:node1_gpfs:node2_gpfs::

- create the NSD: node1:/> mmcrnsd –F /var/mmf

do not mount a GPFS on this network shared disk if GPFS is started, stop it with mmshutdown and define the newly created NSD as a tie-breaker disk

node1:/> mmchconf

(you can see the nam

e of the disk in the disk descriptor file after it has been updated by the mmcrnsd com

Tuning suggestions

- aioservers : AIO should be enabled. The general rule for heavy I/O environments is to initially set the

rs='80’

rs='40'

See the “Tuni n in the “AIX 5L Version 5.1 Performance Management

epool

“maxservers” value to at least 10*(number of disks accessed asynchronously) and the “minservers” value can be set to half of the “maxservers” value. For an 8 disk GPFS file system, an example of the command to modify would be:

node1:/> chdev -l aio0 -a maxserve

node1:/> chdev -l aio0 -a minserve

ng Asynchronous Disk I/O” sectioGuide” for more discussion and details.

- pag : the pagepool is used to cache user data and indirect blocks. It is the GPFS pagepool mechanism nt read as well as write requests asynchronously. Increasing the size of pagepool buffer cache pinned memory available. The default value for pagepool is 20 MB

set" of to change pagepool to 100 MB:

ion data for nodeset MyCluster:

--------------------

autoload yes

that allows GPFS to implemeincreases the amount of GPFSand the maximum allowable value is 512 MB. Applications that may benefit from a larger pagepool (compared to the default) potentially include those that either reuse data, those that have a random I/O pattern, and/or those that have a higher per client pe ent. The size of the pagepool will depend on the "working rformance requirem

I/O data that needs to be cached. For instancenode1:/> mmchconfig pagepool=100M

node1:/> mmlsconfig

Configurat

----------------

pagepool 100M

dataStructureDump /tmp/mmfs

multinode yes


useSingleNodeQuorum no

wait4RVSD no

comm_protocol TCP

clusterType hacmp

group Gpfs.set1

recgroup GpfsRec.MyCluster

File systems in nodeset MyCluster:

-----------------------------

/oradata_gpfs

- ipqmaxlen : the ipqmaxlen network option controls the number of incoming packets that can exist on the IP interrupt queue. Since both GPFS and IBM Virtual Shared Disk use IP, the default value of 128 is often insufficient. This is especially important if your virtual shared disks are configured over IP. The recommended

tting is 512. node1:/> no –a ipqmaxlen=512

node1:/> rsh node2 no –a ipqmaxlen=512

se


8 HACMP IMPLEMENTATION 8.1 INSTALLING HACMP

HACMP (High Availability Cluster Management Protocol) is a product which provide a high availability on a cluster of machines (from 2 to 32 nodes). HACMP/ES is the enhanced version of HACMP (Enhanced Scalability). HACMP/CRM (Concurrent Ressource Manager) layer includes the CLVM, which enable the concurrent logical volume manager.

In all this CookBook, HACMP means HACMP/ES/CRM.

The instances of the same parallel database have a concurrent access on the same external disks. It is a concurrent access, and not a shared one.

In this Oracle10g RAC installation guide, the purpose is not to set up all the parameter stuff of HACMP. We will focus on the concurrent volumes groups, which is enough to install and run Oracle10g RAC. In this cookbook, HACMP 5.1 ptf 5 is used and the RSCT 2.3.4. RSCT (RS/6000 Cluster Technology) are the base filesets used by HACMP to manage the cluster.

Exactly the same filesets have to be installed on all the machines of the cluster (node1, node2 & node3)

All the official HACMP documentation is available at : http://hacmp.aix.dfw.ibm.com/

How to install HACMP using smit install :

On the directory containing all the filesets, check if the hidden file .toc exists. If not, when located on this directory, make inutoc .

If you are using a cdrom as source for this install, this file always exists on the installation media.

specify the directory containing the filesets

Type F4 to list the filesets to install, rather than choosing ”all latest”.

you are doing, you can use the field preview only before proceeding to the real install.

See § 20.2.1 HACMP 5.1 filesets on page 172 and § 20.2.2RSCT 2.3.4 filesets for HACMP Implementation on page 172 to check that all the necessary filesets have been installed.

This appendix provides the result of the command : node1:/> lslpp -l | grep cluster

node1:/> lslpp -l | grep rsct

During the installation, select all the common filesets, the HACMP support one, but not the PSSP support one.

8.2 POST INSTALL TASKS

You must have at least the filesets listed in § 20.2.1 HACMP 5.1 filesets on page 172 and § 20.2.2RSCT 2.3.4 filesets for HACMP Implementation on page 172, with a release number equal or higher.

node1:/> lslpp -l | grep cluster

node1:/> lslpp -l | grep rsct

add in the PATH environment variable the following directories /usr/es/sbin/cluster

/usr/es/sbin/cluster/utilities

/usr/es/sbin/cluster/sbin

sr/es/sbin/cluster/diag

smit install

To be sure of what

/u




After both RSCT and HACMP have been installed successfully on all the nodes, all the machines have to been rebooted before going on with HACMP configuration.

8.3 CREATING THE CLUSTER HACMP

Main steps for creating an HACMP configuration :

Consult the HACMP documentation, available on http://hacmp.aix.dfw.ibm.com/

All this steps have to be done using one node only. During the synchronization process, HACMP will copy the whole configuration to every node in the cluster. This functionality is called SPOC (Single Point Of Control)

Every time you have modified the topology or resources of the cluster, don’t forget to synchronize. Don’t try to m

ake cluster configuration at the same time on two nodes.

Configuring HACMP using Smit menus

Cluster Configuration

Extended topology

Configure HACMP cluster

Add/Change/Show an HACMP cluster

Configure HACMP node

Add a node to the HACMP Cluster

Configure HACMP Networks

Add an IP-based Network to the HACM Cluster

HACMP Verification and Synchronization

Configure HACMP Communication Interfaces/Devices

Add Communication Interfaces/Devices

Configure HACMP Networks

Change/Show a Network in the HACMP Cluster

Discover HACMP-related Information from Configured Nodes

HACMP Extended Resource Group Configuration

Extended Resource Configuration

Add a Resource Group

Change/Show Resources and Attributes for a Resource Group

HACMP Verification and Synchronization

On one of the nodes of the cluster, configure the cluster : smit hacmp, and follow the screens.

ardware topology, which includes the nodes participating to the cluster, the First of all, define the cluster hnetwork, and network interfaces.


In the step under you give an identifier and a name to your cluster.


Configure cluster nodes:

Here, you enter the name of the nodes participating to the cluster. Those names are the hostname, not the name planned for interconnect network (in our case node1, node2 and node3).


Configure the network

In the case of the implementation of a concurrent volume group, only a private network is needed. It’s not necessary to define boot, service and standby addresses like in cascading or rotating HACMP configurations. In RAC environments, you will need a high performance private network between the machines participating to the RAC cluster. The network type of this interconnect can be HPS (High Performance Switch) or ethernet (use a point to point gigabit ethernet, for example).


Configure the adapters

There must be one adapter by machine. This adapter is configured for the interconnect network. It is not the hostname.





Synchronize the cluster topology

It is better, but not mandatory, that HACMP is stopped when starting synchronizing.

8.4 CREATING THE RESOURCE HACMP

The second configuration step is to define the concurrent volume group, which has to be varied on by HACMP.

ent resources

This is called the resource.

Discover concurr


Define a resource group

icipating nodes must be entered. The node relationship must be “concurrent”. All part



Be careful to enter the concurrent volume group name in the line Concurrent Volume Group, me Group !

and not Volu

Synchronize the cluster resources

Important : your cluster must be synchronized after each new modification (topology or resource)

To check your HACMP cluster configuration, see § 19 Appendix D : HACMP Cluster verification output Page 168


8.5 STARTING HACMP

Is HAC

You hav

The con p is not available in concurrent mode access (i.e. varyon)

To chec ume groups lsvg –o

In the comm

If the volume group is available on one of the nodes, make on this node.

Do the following on each node of the cluster.

MP correctly configured ? Checklist.

e to check if the concurrent Volume Group is active or not on both nodes.

current volume group is managed by HACMP. When HACMP is stopped, the volume grou (i.e. varyoff) When HACMP is started, the volume group is varyon

k the available vol

and output, you should see rac92_cvg only when HACMP is up and running.

varyoffvg rac92_cvg


The lock manager does not have to be started. The locks are managed by Oracle.

Is HACMP up and running ?

When you have OK at the previous smit screen, this does not mean that HACMP is running yet. It means that the start process is engaged. Because HACMP is very long to be up (synchronization between the nodes), wait for 10 minutes. If after this time HACMP is not running, there is a problem, and you have to have a look into the logs.

HACMP logs : /tmp/hacmp.out on each node

A useful command : tail –f /tmp/hacmp.out

clstat , xclstat : programs displaying the state of the HACMP cluster and every node participating.


You can also check the HAGS sockets and the cluster sub-systems.

HAGS socket

The HAGS socket needs to be writable by "oracle" an"oracle". By configuring the group and permissions for

d the "cldomain" executable needs to be executable by the "grpsvcsdsocket.<domain>" file the instance will be

r, perform the following tasks :

ssions on the "cldomain" executable : # chmod a+x /usr/sbin/cluster/utilities/cldomain

# chgrp hagsuser /var/ha/soc/grpsvcsdsocket.rac92_cluster

- change the group permissions for the "grpsvcsdsocket.rac92_cluster" socket :

# chmod g+w /var/ha/soc/grpsvcsdsocket.rac92_cluster

Jun 19 16:58 em.clsrv.rac92_cluster

ot system 512 Jun 19 17:51 grpsvcs.clients.rac92_cluster

512 Jun 19 16:57 haem 512 Apr 02 23:30 hats 512 Jun 19 16:57 topsvcs

s

cs 65698 active grpsvcs grpsvcs 63872 active grpglsm grpsvcs 68862 active emsvcs emsvcs 69132 active emaixos emsvcs 62336 active clstrmgrES cluster 70086 active clsmuxpdES cluster 65306 active

You have also another way to see if HACMP is up : list the online volume groups. If your concurrent volume group oradatavg is varyon on all the nodes, all is OK ! This is the only use of HACMP for RAC databases

roviding concurrent access to disks. To list the online volume groups : lsvg –o on each node. To see th

7000004c00000000ff87432e75 TE: active PP SIZE: 64 megabyte(s)

G PERMISSION: read/wri e TOTAL P s: 79 (5056 megabytes) MAX LVs: 512 FREE PPmegabytes) LVs: 2 USED PPs: 6 (384 megabytes) OPEN LVs: 0 QUORUM: 2 TOTAL PVs: 1 VG DESCRIPTORS: 2 STALE PVs: 0 STALE PPs: 0 ACTIVE PVs: 1 AUTO ON: no Concurrent: Enhanced-Capable Auto-Concurrent: Disabled

able to communicate with HAGS and the Oracle instance will mount.

On all the nodes of the cluste- check hagsuser group exists, else create it

place "oracle" into the "hagsuser" group - change the permi

- change the group to "hagsuser" for the "svcsdsocket.rac92_cluster” socket :

You should have something like the following : ls –l /var/ha/soc total 4 srw-rw-rw- 1 root haemrm 0srw-rw---- 1 root haemrm 0 Jun 19 16:58 em.rmsrv.rac92_cluster drwxrwxrwx 2 rosrw-rw-rw- 1 root hagsuser 0 Jun 19 16:57 grpsvcsdsocket.rac92_cluster drwxrwx--- 2 root haemrm drwxr-x--- 2 root system drwxr-xr-x 2 root system

Daemons, etc…

Execute the following command : l src -a | egrep 'svcs|ES'

It should give the following output : topsvcs topsv

: pVOLUME GROUP: oradatavg VG IDENTIFIER: 0033c6G STAVV t P

s: 79 (4672


VG Mode: Concurrent Node ID: 1 Active Nodes: 2 3 MAX PPs per PV: 1016 MAX PVs: 128 LTG size: 128 kilobyte(s) AUTO SY C: HOT SPARE: no BB POLICY: relocatable

N no


9 SYNCHRONIZE THE SYSTEM TIME ON CLUSTER NODES MANDATORY

To ensure that RAC operates efficiently, you must synchronodes. Oracle recommends that you use xntpd for this purimplementation of the Network Time Protocol (NTP) version 3 standard and is more accurate than timed.

To configure xntpd, follow these steps on each cluster no

1. Enter the following command to create required file# touch /etc/ntp.drift /etc/ntp.trace /etc/ntp.conf

2. Using any text editor, edit the /etc/ntp.config file: # vi /etc/ntp.conf

3. Add entries similar to the following to the file: # Sample NTP Configuration file # Specify the IP Addresses of three clock server systems. server ip_address1server ip_address2server ip_address3 # Most of the routers are broadcasting NTP time information. If your # router is broadcasting, then the following line enables xntpd # to listen for broadcasts. broadcastclient # Write clock drift parameters to a file. This enables the system # clock to quickly sychronize to the true time on restart. driftfile /etc/ntp.drift tracefile /etc/ntp.trace

4. To start xntpd, follow these steps:

a. Enter the following command: # /usr/bin/smitty xntpd

b. Choose Start Using the xntpd Subsystem, then choose BOTH.

nize the system time on all cluster pose. xntpd is a complete

de:

s, if necessary:


10 IMPORTANT TIPS FOR ORACLE SOFTWARE AND PATCHS INSTALLATION INSTRUCTIONS

10.1 10G INSTALLATION ON AIX 5.3, FAILED WITH CHECKING OPERATING SYSTEM VERSION MUST BE 5200

Doc ID: Note:293750.1 Subject:

10g Installation on Aix 5.3, Failed with Checking operating system version must be 5200 Failed

Type: PROBLEM Status: MODERATED

05

@ (AuthWiz 2.0) Created from SR 4219914.995

Content Type: TEXT/X-HTMLCreation Date: 13-DEC-2004 Last Revision Date: 02-MAR-20

pid Visibility (RaV) process, and w.

. @ Click here to edit in wizard. This document is being delivered to you via Oracle Support's Ratherefore has not been subject to an independent technical revie The information in this document applies to:

racle Server - Enterprise Edition - Version: 10.1.0.2 to 10.1.0.2 OAIX 5.3 Based Systems (64-bit)

ymptoms S 10g Installation on Aix 5.3, Faile with Checking operating system version must be 5200 Failed Installation Log show following details : Using paramFile: /u06/OraDb10g/Disk1/install/oraparam.ini Checking installer requirements... Checking operating system version: must be 5200 Failed << Cause

d

ng message or similar may

This issue is same as the following : Oracle bug fixes for AIX 5L v5.3 interoperability: When running the Oracle Universal Installer (OUI) the followi appear: "OUI-18001: The operating system “AIX Version 5300.0x” is not supported." Fix

Workaround is to run the OUI as follows:

This parameter tell the installer to not stop because encountering an OS version greater than e

./ runInstaller -ignoreSysPrereqs

xpected.

References Note 282036.1 - Minimum software versions and patches required to Support Oracle Products on IBM pSeries.


10.2 10G RECOMMENDED STEPS BEFORE INSTALLATION AND APPLY

Description

ING ANY PATCH SET ON AIX

on AIX platforms do not

Patch Set or during upgrade pplied.

CLE_HOME (being sure to stance ).

HOME

The patch set instructions for Installation or Patch Sets include instructions to run "slibclean" before installing. This can lead to write errors and / or strange other errors during the application of the/ operation of a database after the Patch Set has been a

The recommended steps before installation and applying any Patch Set on AIX are:

1. Shutdown all instances which use the target ORAexit the SQLPLUS session used to shut down each in

2. Stop all listeners started from the target ORACLE_

3. Stop all client application code / daemons which use the target ORACLE_HOME

4. Run the AIX command "/usr/sbin/slibclean" as "root" to clean all unreferenced libraries from memory.

5. Follow the install steps for the Patch Set

Possible Symptoms • Write Errors during Patch Set installation

<Note:169705.1> describes some of the "write" application of the Patch Set if slibclean is not run

Explanation

errors which can occur during .

ibrary is kept in

but subsequent startup

When AIX loads a shared library into memory the image of that lmemory even if no process is using the library. If the on-disk copy of the library is altered then applications using that library still use the in-memory copy and not the updated disk copy. This is normal expected behaviour on AIX.

In the case of applying an Oracle Patch Set then shutting down all the instances , listeners and applications still leaves shared libraries in memory (eg: libjox9.a stays in memory). Application of the Patch Set updates the disk copy, of an instance uses the in-memory library images (if they are still present). Hence the version banner can show the old release, and upgrade steps may fail as the instance is running an unsupported combination of libraries.

Running "slibclean" before starting the upgrade flushes libraries which are not currently in use from memory.


11 INSTALL THE CUSTER READY SERVICES (CRS)

Please read following Metalink note about CRS and 10gRAC :

CRS and 10g Real

Application Clusters259301.1

15-MAR-2005

Oracle Server Enterprise Edi

Generic BULLETIN

Oracle Cluster Ready Services installation is necessary and mandato one only starting from one node. Once the first node is installed, Oracle OUI automatically starts the copy of the mandatory files on the others nodes, using rcp command. This step sh t think the OUI is stalled, and look at the network traffic before can

- n tio

ry. This installation just have to be d

ould not last long. But in any case, don’celing the installation !


You can create a staging area. The name of the subdirectories is in the format “Disk1”

As root user, execute : xhost +

ogin as oracle and follow the procedure hereunder…

Setup and export your DISPLAY, TMP and TEMP variables

- export DISPLAY=....... - export TMP=/tmp With /tmp or other destination having enoug- export TEMP=/tmp

IF AIX5L release 5.3 is used, do modify the file oraparam.ini, a

update entries AIX5200 to AIX5300 on both files, and exe$/<cdrom_mount_point>/runInstaller

Or execute : ./runInstaller -ignoreSysPrereqs

L

h free space, about 500Mb

nd cluster.ini in Disk1/installer

cute :

quirements for

OUI (Oracle Universal Installer) check the operating system re AIX5L 5.2.0.0. If AIX

aintenance level 1, 2, 3 or/and 4 are installed, the installer will notice (no further actions) and will go to the ext step.

To chek AIX mantenance level installed on each node :

-> instfix –i|grep ML

All filesets for 5.2.0.0_AIX_ML were found.

All filesets for 5200-01_AIX_ML were found.

All filesets for 5200-02_AIX_ML were found.

l filesets for 5200-03_AIX_ML were found. All filesets for 5200-04_AIX_ML were found.

OUI (Oracle Universal Installer) asks you to run rootpre.sh as root. At the OUI Welcome screen, click Next.

mn

Al

Make sure to execute rootpre.sh on each node before you click to the next step.


At the O

UI Welcome screen

Just click Next ...

where you want to create the ry directory. ult it will be creatLE_BASE

SpecifyinventoBy defa ed in the $ORAC Operating system group name should be

Then click Next ...

set as dba

The OUI ask the user to run orainstRoot.sh in a separate window, if it’s the first Oracle product install on this machine. This script creates the file /etc/oraInst.loc, which is used by OUI for the list of installed products. Connect as root on node 1, and run the orainstRoot.sh located in $ORACLE_BASE/oraInventory This will change permissions, and group name to dba on the /etc/oraInst.loc file. Then click Continue ...


Specdestin

name and

in The dinsid

ORACLE_

Th Next ...

Language Selection :

choose ge you nsh should always be selected in

addition of the others.

r example, we’ll just need English.

click Next ...

Just Engli

the langua eed.

In ou

Then

Specify file locations Don’t change t h

ify an ORACLE_HOME ation directory for the CRS

stallation.

estination directory should be e the $ BASE

en click

he source pat


Cluster Configuration :

our

pecify Public Node Name and Private ode Name for all nodes to be part of e Oracle cluster.

ublic correspond to IP address linked to k

Private correspond to IP address linked to the RAC interconnect.

In our case, we are using the same interconnect link for GPFS and RAC.

If you have problems at this stage when clicking on Next (er

Check your network configuration, and User equivalence con

Specify a cluster name, crs in xample e

SNth

Pconnected to the public networ

Then click Next ...

ror messages) figurations on all nodes.

Private Interconnect

nforcement : pecify which network card correspond

ork, and which one ivate network (RAC

Interconnect)

n0 must exist as private on each ode.

n1 must exist as public on each node.

Then click Next ... At this stage, you must have already configured the shared storage. At least for : Oracle Cluster Registry Voting Disk

You must know where to install these 2 files to be reachable by all nodes participating to the RAC cluster

You have 3 implementations choices : Raw Disks with Oracle ASM (Automated Storage Management) Cluster files system with IBM GPFS Concurrent Raw Devices with HACMP

ESto the public netwcorrespond to the pr

In our example :

En

E


Oracle Cluster Registry :

Specify the OCR location, this must be a shared location on the shared storage reachable from all nodes.

ou must have the read/wright permissions on this shared location from all nodes. If ASM implementation is used,

fy raw disk location If GPFS implementation is used,

specify files system location If Raw Devices implementation is

specify raw device location Then click Next ...

And y

speci

used,

Voting Disk :

g Disk location, this d location on the shared

storage reachable from all nodes. And you must have the read/wright permissions on this shared location from all nodes.

• If ASM implementation is used, specify raw disk location

• If GPFS implementation is used, specify files system location

• If Raw Devices implementation is used, specify raw device location

Then click Next ...

tinSpecify the Vo

must be a share


orainstRoot.sh : execute the orainstRoot.sh on all nodes. The file is located in the installation path

Then click Continue ...

of the CRS on each nodes.

Check Cluster Nodes and Remote Nodes lists. The OUI will install the Oracle CRS software on to the local node, and then copy this information to the other selected nodes. Then click Install ...

Summary :

Install : The Oracle Universal Installer will proceed the installation on the first node, then will copy automatically the code on the 2 others selected nodes. During the copy, you may be prompt to enter the source location for Disk2. Just wait for the next screen ...


Install : Copying installed files from node 1 to the other selected nodes. Node 2 and 3 in our example.

Just wait for the next screen ...

Install : Setup Privileges

At this stage, you need to run the “root.sh” script on each selected nodes. Start with node 1 and wait for the result before executing on node 2, then when finished on node 2, execute on node 3. This file is located in

$ORACLE_BASE/crs directory on each node.


Step 1

Execute root.sh on node 1 And wait …


ileges

Step1

Install : Setup Priv

node1”

When finished, CSS deamon should be active on node 1 and inactive on node 2 and 3 Check for line “CSS is active on these nodes.

Install : Setup Privileges Step 2 Execute root.sh on node 2. When finished, CSS deamon should be active 2, and inactiv

on node 1 and e on node 3

Check for line “CSS is active on node1

Step 3

these nodes.

node2”

When finished, CSS deamon should be active on node 1, 2,

e1 node2 node3”

If CSS is not active on all nodes, or on one of the nodes, this means that you could have a problem with the network configuration, or the shared disks configuration for accessing OCR and Voting Disks.

Check your network , shared disks configuration, and owner and access permissions (read/write) n OCR and Voting disks from e ch participating node. And execute again the root.sh script on node

having the problem.

Execute root.sh on node 3

and 3. Check for line “CSS is active on these nodes. nod

o a


Metalink note to use in case of problem with CRS ... Subject Doc ID Modified Date Product Platform Type

- Generic H 1 10G: How to Stop the

Cluster Ready Services (CRS)

263897.1

14-SEP-2004

Oracle ServerEnterprise Edition

OWTO

2 How to verify if CRS install is Valid

295871.1

04-FEB-2005

Oracle Server - Enterprise Edition

Generic HOWTO

10g RAC: 3Troubleshooting CRS Reboots

265769.1

16-MAR-2005


Generic TROUBLE SHOOTING

4. CRS and 10g Real Application Clusters

259301.1

15-MAR-2005


BULLETIN

5 Repairing or

- Generic

Restoring an Inconsistent OCR in RAC

268937.1

02-MAR-2005

Oracle ServerEnterprise Ed

6 Placement of voting

- ition

Generic BULLETIN

-

Generic B

ition

nager for Generic

and OCR disk files in 10gRAC

293819.1

01-MAR-2005


Generic BULLETIN

7 10g RAC: How to Clean Up After a Failed CRS Install

239998.1

23-FEB-2005


ULLETIN

8 CRS 10g Diagnostic Collection Guide

272332.1

15-NOV-2004


Generic BULLETIN

10 10g RAC: Stopping Reboot Loops When CRS Problems Occur

239989.1

03-MAY-2004

Oracle Server - Enterprise Ed

Generic PROBLEM

11 HOW TO REMOVE CRS AUTO START AND RESTART FOR A RAC INSTANCE

298073.1

11-FEB-2005

Enterprise MaRDBMS

BULLETIN

12 HOW TO REMOVE CRS AUTO START AND RESTART FOR A RAC INSTANCE

298069.1

11-FEB-2005

Enterprise Manager for RDBMS

Generic BULLETIN

14 CRS Home Is Only Partially Copied to Remote Node

284949.1

05-DEC-2004


Generic PROBLEM



Just click OK to continue ...

Configuration Assistants :

.

heck for the result to be successful.

C

Then click Next ...

Then click Exit ...

Coming back to this previous screen.

2 configuration assistants will be automatically executed

End of Installation


Post-Installation Oracle unix Profile Update :

To be done on each node. Oracle environment : vi $HOME/.profile file in Oracle’s home dir Add the entries in bold blue color export ORACLE_BASE=/oh10g export AIXTHREAD_SCOPE=S (S for system-wide thread scope) umask 022 export ORACLE_CRS=$ORACLE_BASE/crs export ORACLE_CRS_HOME=$ORACLE_BASE/crs export LD_LIBRARY_PATH=:$ORACLE_CRS/lib:ORACLE_CRS/lib32 export PATH=$ORACLE_CRS/bin:$PATH

At this stage :

• The Oracle Cluster Registry and Voting Disk are creates a• The Oracle Cluster Ready Services is installed, and started on all nodes.

Post-Installation Check list:

• Check CRS processes on each nodes :

ps –ef|grep d.bin

s oracle user :

• Command to start/stop the CRS deamons :

/etc/init.crs start to start the CRS /etc/init.crs stop to stop the CRS

• If necessary and only for troubleshooting purpose, disable the automatic eboot of AIX nodes when

node fail to communicate with CRS daemons, or fail to access OCR and Voting disk.

ectory

nd configured

• Execute crs_stat on each node a

r


12 INSTALL THE ORACLE 10G SOFTWARE

Oracle RAC option installation just have to be done only starting from one node. Once t ed, Oracle OUI automatically starts the copy of the mandatory files on the step could last long, depending on the network speed (one hour…), witOUI is stalled, and look at the network traffic before canceling the instYou can also create a staging area. The name of the subdirectories is in As root user, execute : xhost + Login as oracle and follow the procedure hereunder…


- export DISPLAY=....... - export TMP=/tmp With /tmp or other destination having enough free space, about 500Mb - export TEMP=/tmp

he first node is installsecond node, using rcp command. This hout any message. So, don’t think the allation ! the format “Disk1” to “Disk3”


IF AIX5L release 5.3 is used, do modify the file oraparam.ini, a

update entries AIX5200 to AIX5300 on both files, and execute : $/<cdrom_mount_point>/runInstaller


nd cluster.ini in Disk1/installer

OUI (Oracle Universal Installer) chek the operating system requirements for A

aintenance level 1, 2, 3 or/and 4 are installed, the installer will notice (no fur e step.

To chek AIX mantenance level installed on each node : -> instfix –i|grep ML All filesets for 5.2.0.0_AIX_ML were found. All filesets for 5200-01_AIX_ML were found. All filesets for 5200-02_AIX_ML were found. All filesets for 5200-03_AIX_ML were found. All filesets for 5200-04_AIX_ML were found. OUI (Oracle Universal Installer) asks you to run rootpre.sh as root. At the OUI Welcome screen, click Next.

IX5L 5.2.0.0. If AIX ther actions) and will go to thm

next

Make sure to execute rootpre.sh on each node (Should be already done with the CRS Installation), before you click to the next step


At the OUI Welcome screen

Just click Next ...

Specify File Locations : Do not change the Source field Specify a different ORACLE_HOME Name with its own directory for the Oracle software installation.

This ORACLE_HOME must be different then the CRS ORACLE_HOME.

Then click Next ...

If you don’t see the following screen with Node selection, it might be that your CRS is down on one or

all nodes. Please check if CRS is up and running on all nodes.

Specify Hardware Cluster Installation Mode : Select the other nodes on to which the Oracle RDBMS software will be installed. It is not necessary to select the node on which the OUI is currently running. Click Next.

Then click Next ...


Enterprise, Standard Editionto proceed.

creating a dat

abase by .

specific Do

Prerequis

Select the installation type : You have t o choose

, or Custom

Choose the “Custom” option to avoid

default

Then click Next ...

he option t

The installer will check some product-

ite.

n’t take care of the 3 lines with checking at status “Not executed”, These are just warnings because AIX

aintenance level might be higher then ur example

Then click Next ...

m5200, which is the case in oML04). (

Available Product Components :

ts for racle Database 10g.

Make sure to select the “Oracle eal Application Clusters 10.1.0.2.0”

option

Then click Next ...

Select the product componenO

R


P

rivileged Operating Systems Groups :

erify the UNIX primary group name of the installation of

e Oracle10g software.

d.

of unix oracle user) to be t for both entries.

Then click Next ...

Vthe user which controlsth(Use unix command id to find out) And specify the Privileged OperatingSystem Groups to the value founIn our example, this must be “dba” (Primary groupse

Create Database : Choose “No”, we don’t want to create a database at this stage.

Then click Next ...

Summary : The Summary screen will be presented. Confirm that the RAC database software and other selected options will be installed. Check Cluster Nodes and Remote Nodes lists. The OUI will install the Oracle 10g software on to the local node, and then copy this information to the other selected nodes.

Then click Install ...


Install : The Oracle Universal Installer will

on on the first node, then will copy automatically the code on the 2 others selected nodes.

Just wait for the next screen ... During the copy, you may be prompt to enter the source location for Disk2 and Disk3.

proceed the installati

Set

up Privileges : At this stage, run the “root.sh” script on each selected nodes

your x-windows terminal to be able to proceed with the VIP graphical assistant when executing root.sh on the first node Execute the root.sh on each node ...

CONF

Setup the Display value as root in

12.1 VIP IGURATION ASSSITANT

On the first node as root user, you must setup the DISPLAY before running the root.sh script

Wait for Next Screen ...


The VIP “Welcome” graphical screen

t.sh script

c .

will appear at the end of the roo

Then lick Next ..

s the RACork.

erconnect vate

e as root.

The click Next

2 :

e for node1 and stroke the TAB key to

alidityeeding.

f the entries b re

The click Next

1 of 2 : Select one and only one network

Select the network interface corresponding to the Public Network

ork card on each node must have the same name, “en1” for example in our case. En0 i Int , or prinetwPlease check with “ifconfig –a” on each nod Select “en1” in our case

n ...

interface.

Remember that each public netw

2 of In the Virtual IPs for cluster nodes screen, you must provide the VIP node nam

automatically feel the rest. Check v o efoproc

n ...


VIP conficeed with c

uration Aseation, conflication

t will ion an

d e the

The Summary screen will appear, please validate the entries.

Then click Finish ...

The g sistanpro r igurat d startup of all app resources on all selected nodes. VIP, GSD an ONS will bapplication resources to be created.

Wait while progressing ...

Th click OK

l be

configuration is 100% completed.

en ...

If you don’t get any errors, you’lprompted to click OK as the


Run the root.sh script on o befor

proceeding to the next stage.

VI sistant a her nodes

when executing the root.sh script.

In our le, we’ll run root.sh on Node 2 and node 3

oo

co

for wo IP.

If problems occurs with VIP configuration assistant, please e specified in this cha

each ther node e

No displ

P screen asyed on the ot

will be

examp

as r

t user.

Then click Next ...

ifconfig –a

check that each network card configured Using “if nfig –a” on each node,

Public net rk is mapping a virtual

use th metalink notes pter.

Check the Configuration results.

Then click Exit ...


Metalink notes to use for VIP : Subject Doc ID Modified Date Pro 1 Configuring the IBM

perating

Oracle 10g VIP

296856.1 17-FEB-2005 Oracle Serve pris

2

Changin

r - Enter e Edition

g the check or the Oracle

10g VIP

294336.1 23-FEB-2005 Oracle Server - Enterprise Edition

Generic BULLETIN

2 Modifying the VIP of Node

276434.1 13-JAN-2005 Oracle Server - Enterprise Edition

Generic BULLETIN

ng3 Modifyi the default address the Oracle

10g VIP

298895.1 07-MAR-2005 Oracle ServEnterprise Edition

er - Gene

ri N

4 How to C

c BULLETI

onfigure s for 10g

RAC

264847.1 24-FEB-2005 Orac

I

tall : Setu rivileges

Just cli OK to continue ...

duct Platform Type

AIX 5L OSystem for the

interval f

a Cluster

gatewayused by

Virtual IPle Server -

Enterprise Edition

Generic HOWTO

ns p P

Coming back to this previous screen.

ck


http://updates.oracle.com/ARULink/PatchDetails/process_form?patch_num=4003062&release=80101030&plat_lang=212P&email=frederic.michiara%40usa.net&userid=MICHIARA&


Configuration Assistants : The Universal Installer will proceed with the “Oracle Net Configuration Assistant”

Wait for next screen ...

Th click Next ...

End of Installation :

his screen p

Check that it is successful and write dow lpplications that have been deployed

(isqlplus, …).

en

T

will automatically ap ear.

n the URL ist of the J2EE a

Th click Exit ...

Select “Perform Typical Configuration”

en


Post-Installation Oracle unix Profile Update :

Oracle environment : vi $HOME/.profile file in Oracle’s home directory Add the entries in bold blue color export ORACLE_BASE=/oh10g export AIXTHREAD_SCOPE=S umask 022 export ORACLE_CRS=$ORACLE_BASE/crs export ORACLE_CRS_HOME=$ORACLE_BASE/crs export ORACLE_HOME=$ORACLE_BASE/db10g export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_CRS/lib: …. …. $ORACLE_HOME/lib32:$ORACLE_CRS/lib32 export PATH=$ORACLE_HOME/bin:$ORACLE_CRS/bin:$PATH At this stage :

• The Oracle 10g Software with the Real Application Cluster option is installed. • The VIP (Virtual IP), GSD ans ONS application resources are configured on all

nodes. • The listeners are configured and started.

Post-Installation Check list:

• Execute crs_stat on each node as oracle user • Make sure VIP is present and configured on each node :

as root, execute ipconfig -a

To be done on each node.


13 ORACLE PATCHES TO APPLY Before any patch update, backup your installation, database, and OCR (Oracle Cluster

If you apply patchset 1 (10.1.0.3), do apply : Patch

Registry) disks

Description Release Updated Size

3761843 Oracle Database Family: Patchset 10.1.0.3 PATCH SET FOR ORACLE DATABASE SERVER

10.1.0.3 17-AUG-2004 916M

And go to “Patch to apply on top of 10.1.0.3“

13.1 PATCH TO APPLY ON TOP OF 10.1.0.2

Extracted from http://metalink.oracle.com (5th of April 2005), please consult Metalink for last

pdated patchs list.

All patches are not necessary, just apply patches that will solve problems you may encounter.

Results for Platform : AIX5L Based Systems (64-bit) Patch

u

Description Release

Updated Size

0305140 Oracle Database Family: Patch PLEASE PROVIDE MLR ON TOP OF 10.1.0.2 FOR SECURITY ROLLUP

10.1.0.2 21-JAN-2005 29M

3838804 Oracle Database Family: Patch MERGE LABEL REQUEST ON TOP OF 10.1.0.2

10.1.0.2 10-NOV- 4.0M

3458343

2004

10.1.0.2 15-AUG- 91K

10.1.0.2 06-AUG-2004

Oracle Database Family: Patch SRVCTL CONFIG DATABASE FAILS FOR 9.2 TARGET

10.1.0.2 28-SEP-2004 16K

3785451 Oracle Database Family: Patch NOT USING VALUES RETURNED BY IOCTL WHEN THE DEVTYPE IS DD_SCDISK.

10.1.0.2 02-SEP-2004 109K

3811942 Oracle Database Family: Patch MERGE LABEL REQUEST ON TOP OF 10.1.0.2

10.1.0.2 31-AUG-2004

19M

3572753 RDBMS Server: Patch ORA-600: [KGHSTACK_UNDERFLOW_INTERNAL_2] TAKING BACKUP OF CONTROLFILE VIA RMAN

2004

3712203 RDBMS Server: Patch BACKPORT THE ZSUBTYPE PATCH FOR VERITAS DEVICE FOR 9.2.0.X AND 10.1.0.X.

338K

3452566 RDBMS Server: Patch ORA-15131: BLOCK 43008 OF FILE 3 IN DISKGROUP 1 COULD NOT BE READ

10.1.0.2 29-JUL-2004 34K


3728908 RDBMS Server: Patch

ORA-600:[KCCSBCK_FIRST] HAPPEN AFTER RESTART INSTANCE BY CRS WITH HACMP

3612581

10.1.0.2 23-JUL-2004 342K

10.1.0.2 06-JUL-2004 15K

10.1.0.2 20-JUN-2004 14K

2 12-MAY-

RDBMS Server: Patch KPOFDR-LONG INTERNAL ERROR DUING SAP-SD BENCHMARK

3417593 RDBMS Server: Patch PIECEWISE FETCHING OF VARCHAR2 COLUMN FAILED IN SOLARIS CLIENT -> NT SERVER MODE

3564573 RDBMS Server: Patch ORA-01017 WHEN 10G CLIENT CONNECTS TO 9I OR 8I DATABASE WITH DIFF CHAR SET

10.1.0.2004

16K

3527628 Oracle Database Family: Patch ROOT.SH SHOULD BE MODIFIED TO RECOGNISE SILENT INSTALLS

10.1.0.2 11-MAY-2004

35K

8559233 RDBMS Server: Patch 10G-SH9: ASM RESOURCE RACGMAIN CORE DUMP WITH SHARED ORACLE HOME

10.1.0.2 07-MAY- 37K

3492889

2004

0.1.0.2 30-APR-2004 421K

0.1.0.2 21-APR-2004 15K

lems you may encounter.

R

RDBMS Server: Patch THREAD SPAWN FAILURES IN THE OCR LOGS CAUSING SESSIONS TO HANG

1

3540898 RDBMS Server: Patch MERGE LABEL REQUEST ON TOP OF 10.1.0.2

1

13.2 PATCH TO APPLY ON TOP OF 10.1.0.3

Extracted from http://metalink.oracle.com (5th of April 2005) Please consult Metalink for last updated patchs list. All patches are not necessary, just apply patches that will solve prob

Results for Platform : AIX5L Based Systems (64-bit)

Patch Description elease

Updated Size

4130275 RDBMS Server: Patch ORA-354 ON LOGICAL STANDBY HOT MINING AFTER PRIMARY TRANSPORT ERROR

10.1.0.3 05-APR-2005 342K

4054910 RDBMS Server: Patch THE DEFAULT VALUE FOR FAIL_WHEN_ALL_LINK_DOWN SHOULD BE 1 (TRUE) ON AIX5L

10.1.0.3 05-APR-2005 21K

4274434 RDBMS Server: Patch 10103ROOT.SH REQUESTING LIB/LIBCLNTSH.A(SHR.O)

10.1.0.3 04-APR-2005 330K

4061535 RDBMS Server: Patch ORA-07445 [KNLCINLINEDATA()+793] IN CAPTURE PROCESS

.0.3 04-APR-2005 45K 10.1



ORA-7445 KOKLMIFRDHT WHEN RUNNING LOB RELATED TESTS WITH ARCHIVING ON

.1.0.3 01-APR-2005 59K

4258825

10

1.0. 31-MAR- 12005

1 24-MAR-2005

RDBMS Server: Patch CORRUPTED R-TREE INDEX - CONTAINS ORPHAN ROWID'S

10. 3 4K

4128156 RDBMS Server: Patch ASM FREE SPACE NOT USED WHEN TABLESPACE CREATION USES MANY, SMALLER FILES

0.1.0.3 17K

1 54238 8APPSST10G: PLS-907 CANNOT LOAD LIBRARY UNIT

2005 RDBMS Server: Patch 10.1.0.3 16-MAR- 29K

423 4063 RDBMS Server: Patch MERGE LABEL REQUEST ON TOP OF 10.1.0.3 FOR

10.1.0.3 15-MAR-

BUGS 4149653 3819875

662

2005 566K

268 9 RDBMS Server: Patch 10.1.0IMPORT DOES NOT WORK WELL WHEN USE A

SCHEMA

.3 14-MAR-2005

846K

409 79

PARTICULAR XML

08 RDBMS Server: Patch 10.1.0.3 09-MAR-MERGE LABEL REQUEST ON TOP OF 10.1.0.3 FOR BUGS 3777747 3849723 3590881 3863235

2005 1.2M

3417593 RDBMS Server: Patch PIECEWISE FETCHING OF VARCHAR2 COLUMN FAILED IN SOLARIS CLIENT -> NT SERVER MODE

10.1.0.3 09-MAR-2005

14K

4091881 Oracle Database Family: Patch MERGE LABEL REQUEST ON TOP OF 10.1.0.3 FOR BUGS 3982434 4087215

10.1.0.3 07-MAR-2005

2.0M

4206159 RDBMS Server: Patch OPROCD IS PRONE TO TIME REGRESSION DUE TO CURRENT API USED

10.1.0.3 03-MAR-2005

18M

3910149 RDBMS Server: Patch KGLHDDEP SGA LEAK IN RAC

10.1.0.3 23-FEB-2005 55K

4090777 RDBMS Server: Patch CSS CORE DUMPS ON ALL 5 NODES

10.1.0.3 22-FEB-2005 732K

3635177 RDBMS Server: Patch PSRC: ORA-7445 [KXCCUIN] POSSIBLE FROM DML

10.1.0.3 21-FEB-2005 15K

4113001 RDBMS Server: Patch MERGE LABEL REQUEST FOR PEOPLESOFT

10.1.0.3 18-FEB-2005 52K

3972424 Oracle Database Family: Patch 10.1.0.3.0 LISTENER HANGS INTERMITTENTLY (ONS)

10.1.0.3 10-FEB-2005 491K

3857039 RDBMS Server: Patch RMAN COMMAND FAILS WITH ORA-6502 ERROR

10.1.0.3 02-FEB-2005 740K

3228560 RDBMS Server: Patch ORA-00600: [QKEBVALIDATEOPN:2], [198]

10.1.0.3 01-FEB-2005 69K



INCORRECT EXCEPTION HANDLING IN QMX0.C:QMXUPARSEXOBWITHPROPCS()

10.1.0.3 27-JAN-2005 76K

4003062 RDBMS Server: Patch PLEASE PROVIDE MLR ON TOP OF 10.1.0.3 FOR SECURITY ROLLUP

10.1.0.3 18-JAN-2005 2.8M

4034746 RDBMS Server: Patch WRONG RESULTS WHEN USING HASH JOIN WITH PARALLEL EXECTION.

10.1.0.3 17-JAN-2005 18K

4120185 RDBMS Server: Patch MERGE LABEL REQUEST ON TOP OF 10.1.0.3 FOR BUGS 3672097 4011806

10.1.0.3 17-JAN-2005 96K

3572753 RDBMS Server: Patch ORA-600: [KGHSTACK_UNDERFLOW_INTERNAL_2] TAKING BACKUP OF CONTROLFILE VIA RMAN

10.1.0.3 16-JAN-2005 91K

3966811 RDBMS Server: Patch NEW INIT.CSSD PROVIDED IN BUG#3923542 MIGHT CAUSE INCORRECT BEHAVIOR

10.1.0.3 06-JAN-2005 877K

3828269 RDBMS Server: Patch WRONG RESULT OCCURS WHEN SPACE CHARACTER IS SCANNED WITH UNIQUE INDEX

10.1.0.3 28-DEC-2004 56K

4026166 RDBMS Server: Patch CAPTURE PROCESS DISCARDS MVDD INFO FOR RENAMED TABLES

10.1.0.3 21-DEC-2004 74K

4327738 RDBMS Server: Patch EXTRANEOUS TRACE FILES PRODUCED

10.1.0.3 08-DEC-2004 193K

4051111 RDBMS Server: Patch BUNDLE FOR BUGS3992673 AND 3942568

3455036

10.1.0.3 06-DEC-2004 1.6M

1

1 29-NOV-2004

RDBMS Server: Patch CRSD/EVMD PIDFILE/LOCK MECHANISM IS FAILURE PRONE.

0.1.0.3 06-DEC-2004 21K

3854375 RDBMS Server: Patch ORA-7445 WHEN UPDATING A VIEW WITH A TRIGGER ON THE VIEW

0.1.0.3 20K

3844049 RDBMS Server: Patch APPSPERF: RDBMS: 600 ERROR [QCTCTE1]

10.1.0.3 25-NOV-2004 20K

4000840 RDBMS Server: Patch UPDATE STATEMENT FAILS WITH OERI [QERTBFETCHBYROWID],OERI [4511],..

10.1.0.3 16-NOV-2004 45K

3712203 RDBMS Server: Patch BACKPORT THE ZSUBTYPE PATCH FOR VERITAS DEVICE FOR 9.2.0.X AND 10.1.0.X.

10.1.0.3 16-NOV-2004 316K

3953420 Oracle Database Family: Patch MLR: ONS ROLLUP ON TOP OF 10.1.0.3

10.1.0.3 12-NOV-2004 540K

3924482 RDBMS Server: Patch APPSST10103: ORA-31693 ORA-604 PLS-306 ON

10.1.0.3 09-NOV-2004 70K


1159 APPS DB DURING DATAPUMP EXPORT

3814603 RDBMS Server: Patch ORA-600[KCCSBCK_FIRST]

10.1.0.3 05-NOV-2004 728K

3625392 RDBMS Server: Patch WRONG RESULTS OR ORA-7445 [EVAOPN2] WITH STAR TRANSFORMATION HINT

V-2004 251K

3941443

10.1.0.3 05-NO

10.1.0.3 RDBMS Server: Patch ISSCHEMAVALID CRASH IN QMXINSERTNODEBEFORE WITH CDATA TAG DOCUMENT

04-NOV-2004 111K

3942568 RDBMS Server: Patch NODE CRASHING WHEN HEARTBEAT CABLE IS PULLED ON OTHER (MASTER) NODE

10.1.0.3 01-NOV-2004 733K

3672097 RDBMS Server: Patch WRONG RESULTS JOINING CHAR OR NCHAR FROM 10G TO 9I & 8I ACROSS DB LINK

10.1.0.3 31-OCT-2004 21K

3880640 RDBMS Server: Patch QUERY WITH ALL OPERATOR HANGS

3800614

10.1.0.3 18-OCT-2004 44K

RDBMS Server: Patch ORA-600 [12410] ON SIMPLE QUERY WITH ANALYTIC FUNCTION

10.1.0.3 18-OCT-2004 14K

3412553 RDBMS Server: Patch FXD: ORA-942 DURING FUNCTIONAL INVOCATION OF CONTAINS ON ANOTHER USER'S TABLE

10.1.0.3 13-OCT-2004 29K

3555800 RDBMS Server: Patch 0G CLIENT CAN'T CONNECT VIA ASO TO 10G DB ON DIFFERENT ENDIAN ORDER PLATFORM

10.1.0.3 08-OCT-2004 41K

3476889 RDBMS Server: Patch DP EXPORT FAILS WITH ORA-600 [KTCSETSCN-1] WITH LARGE SCN VALUE

3785451

10.1.0.3 07-OCT-2004 50K

-

10.1.0.3 20-SEP-2004 53K

Oracle Database Family: Patch NOT USING VALUES RETURNED BY IOCTL WHEN THE DEVTYPE IS DD_SCDISK.

10.1.0.3 01-OCT-2004 115K

3612581 RDBMS Server: Patch KPOFDR-LONG INTERNAL ERROR DUING SAP-SD BENCHMARK

10.1.0.3 01-OCT-2004 15K

3755693 RDBMS Server: Patch NO ROWS FOUND WHEN USING BINDVARIABLES AND CBO

10.1.0.3 28-SEP-2004 57K

3554494 RDBMS Server: Patch ALTER TABLE SPLIT PARTITION FAILS WITH ORA00933 ERROR FROM RECURSIVE SQL

10.1.0.3 20-SEP-2004 50K

3686661 RDBMS Server: Patch EXCESSIVE REDO GENERATION ON 10G PRODUCTION RELEASE WHEN COMPARED TO


ORACLE 9I

3877070 RDBMS Server: Patch MLR FOR CSFB BUG FIXES ON TOP OF 10.1.0.3

10.1.0.3 13-SEP-2004 1.8M


14 CREATING THE DATABASE USING DBCA Connect as Oracle unix user from first node, and setup your DISExecute dbca & to lanch the database configuration assistant

PLAY

DBCA Welcome Screen

Select the “Oracle Real Application Cluster Database” option.

Then click Next ...

Step 1: Operations Select the “Create a Databse” option.

Then click Next ...

Step 2 : Node Selection Make sure to select all RAC nodes.

Then click Next ...


Step 3 : Database Templates

Or “Custom Database” if you want to generate the creation scripts.

Select “General Purpose”

Then click Next ...

Step 4 : Database Identification Specify the “Global Database Name” The “SID Prefix” will be automatically updated. (by default it is the Global Database Name)

Then click Next ...

Step 5 : Management Options Check “Configure the database with Enterprise Manager” if you want to use the Database Control (local administration). Or Don’t check if you plan to administrate the database using the Grid Control (global network administration)

Then click Next ...


Step 6 : Database Credentials Specify same password for all administrator users, or specify individual password for each user.

Then click Next ...

Step 7 : Storage Options

depending of the storage setup you did choose in the storage setup part :

o Cluster File system (GPFS)o Automatic Storage

Management (ASM) o Raw Device (HACMP)

Then click Next ...

Depending storage option you did select, please go to next step as specified bel

o Cluster File system (GPFS) Go to 14.1 Database On GPFS : Architecture GPFS to implement

o Automatic Storage Management (ASM)

Go to 14.2 Database On ASM : Architecture ASM to implement o Raw Device (HACMP)

Go to 14.3 Database On RAW DEVICES : Architecture RAW DEVICE to implement

Select the storage option

ow :


14.1 DATABASE CREATION ON GPFS

Step 8-a : Storage Options

Choose Cluster File System

Then click Next ...


Step 9-a : Database File Locations

base managed.

nd specify directory if necessary cation for all

atabase”)

Then click Next ...

Specify the way you want the datafiles to be A(option “Common lod

Step 10-a : Recovery Configuratio

n

pecify if you want to use a Flash if so :

o Specify destination directory

...

SRecovery Area, and

o and area size

Then click Next

Step 11-a : Database Content Select “Sample Schemas” if you need And add Custom scripts if necessary

Then click Next ...


Step 12-a : Database Services DeAp n Failover) strategy at this poi TAF can be configured later

Then click Next ...

fine the TAF (Transaction plicationt, or move on to next step.

Step 13-a : Initialization Parameters

r

...

Check-up default parameters, oSpecify your own parameters. Or move to next step

Then click Next

Step 14-a : Database Storage

es.

Then click Next or Finish...

Check-up the entri


Step 15-a : Summary Read/save the summary, and move to n

Then click OK ...

ext step

Step 16-a : Creation Options

ssing ...

Just wait while proce

Step 17-a : Passwords Management Enter in password management, if you need to change pasword, and unlock some user accounts that are locked by default (for security purpose).

Then click Exit ...


At this stage :

• The GPFS Oracle Cluster Database is created on the shared Cluster File Sysem configured with IBM GPFS, with

1 database, 2 databases instances (1 on each nodes),

• export ORACLE_SID=GPFS1 on node1 to access database instance with sqlplus


14.2 DATABASE CREATION ON ASM

#chown oracle:dba /dev/rhdiskn #chmod 660 /dev/rhdiskn /usr/sbin/chdev –l hdiskn –a reserve_lock=no /usr/sbin/chdev –l hdiskn –a reserve_policy=no_reserve /usr/sbin/chdev –l hdiskn –a pv=yes

n standing for disk number

MANDATORY

Change on each nodes for all ASM Disks (LUN’s) candidates to be used in the ASM Disks Group.



Step 8-b : Storage Options

Choose Automatic Storage Management (ASM)

t ...

Step 8-b Create ASM Instance

• Specify SYS password for the ASM Instance

• Choose the type of

parameter file

Then click Next ...

Step 9-b Create ASM Instance

Then click OK..

Then click Nex


ASM Disk Groups Then Click on

Step 10-b : Create Disk Group

Then Select Disks And External (for no ASM mirroring) Then click Ok ...

Step 11-b : Create Disk Group ASM Disk Group Creation Just Wait

“Create New” ...

Select Show Candidates

while processing ...


Step 12-b : Create Disk Group Now the ASM Disks Group is created Select your DiskGroup for the

Then click Next ...

Step 13-b : Database File Locations Select Use Oracle-Managed Files Then click Next ...

Step 14-b : Create Disk Group For Flash Recovery Area Select Show Candidates Then Select Disks And External (for no ASM mirroring) Then click Ok ...

database



Step 15-b : Create Disk Group

ASM Disks Group eat d

ct your DiskGroup for Flash Recovery Area

Then click Ok ...

Step 16-b : Database File Locations

p 17-b : Database Content

needed

anager

ext ...

Now the eis cr

Selethe

Select Use Oracle-Managed Files Then click Next ...

Ste Select the options And Enterprise MRepository

k NThen clic


Step 18-b : Database Services and TAF (Transaction Application Failover) policy Then click Next ...

9-b : Initialization Parameters

meters needed Then click Next ...

Step 20-b : Database Storage

tafiles organization

Step 1 Select the para

Check the da

Then click Next ...


Step 21-b : Creation Options

ct the options needed

• Create Database

Save as a Database Template

Database Creation

Step 22-b : Summary Check the description Save the HTML summary file if needed

ck Ok ...

Step 23-b : The template is created.

Sele

•

• GenerateScripts

Then click Finish ...

Then cli

Then click Ok ...



Step 24-b : The script have been generated Then click Ok ...

Step 25-b : Database creation on progress Just wait while processing ...

Step 26-a : Passwords Management Enter in password management, if you need to change pasword, and unlock some user accounts that are locked by default (for security purpose).

Then click Exit ...

At this stage :

• The ASM Oracle Cluster Database is created on Shared ASM Disks Group with

1 database, es),

2 ASM instances (1 on each nodes).

lus

2 databases instances (1 on each nod

• export ORACLE_SID=+ASM1 on node1 to access ASM instance with sqlplus • export ORACLE_SID=ASMDB1 on node1 to access database instance with sqlp


14.3 DATABASE CREATION ON RAW DEVICES

reate a parameter file $HOME/dbca_raw_config which will be used by DBCA to map the typical tablespaces

and makes asier to create the database.

Set DBCA_RAW_CONFIG env variable in the Oracle user, before running DBC

EXPORT DBCA_RAW_CONFIG=$HOME/dbca_raw_config

14.3.1 Content example of Raw device DB configuration file $HOME/dbca_raw_config

Raw devices parameter file for the Database Configuration A system=/dev/rrac_system undo1=/dev/rrac_undotbs01 undo2=/dev/rrac_undotbs02 redo01=/dev/rlog11 redo02=/dev/rlog12 redo03=/dev/rlog21 redo04=/dev/rlog22 control1=/dev/rrac_control01 control2=/dev/rrac_control02 control3=/dev/rrac_control03 data=/dev/rrac_data index=/dev/rrac_index temp=/dev/rrac_temp

Cto raw devices. This parameter file needs to be pointed to by the environment variable DBCA_RAW_CONFIGe

A :

ssistant


Step 8-c

Choose Raw Devices

At this stage :

• The Oracle Cluster Database is created on concurent Shared RAW DEVICES with

1 database, 2 databases instances (1 on each nodes),

• export ORACLE_SID=RACDB1 on node1 to access database instance with sqlplus

: Storage Options

Then click Next ...


14.4 MANUAL DATABASE CREATION

Doc ID: Note:240052.1 Subject: 10g Manual Database Creation in Oracle

(Single Instance and RAC) Type: BULLETIN Status: PUBLISHED

Content Ty TEXT/PLAIN Creation Da

Last RevisiDate: 2004

PURPOSE ------- The purpose of this bulletin is to give an example of a manual database creation in 10g. MANDATORY ========== FROM 10g RELEASE WE HAVE SYSAUX TABLESPACE MANDATORY FOR STATISTICS WORKLOAD REPOSITORY FACILITIES (SWRF) GOOD PRACTICE =============

G THE DATABASE. So whenever DBA will

entioning he DEFAULT TABLESPACE clause

andatory to use the init.ora parameompatible must be >=10.0"

Oracle recommends using the Database Configuration Assistant (DBCA) to create your database. hese steps are available for DBAs who want to manually create a

10g database either in single instance or Real Application Cluste

14.4.1Tips to create a database in 10g Real Application Clusters

=================================================

Manual Database Creation steps for Real Application Clusters

================================================ Here are the steps to be followed to create a Real Application Clusters database: 1. Make a init<SID>.ora in your $ORACLE_HOME/dbs d file is in $ORACLE_HOME\database. To simplify, you can init<SID>.ora and modify the file. Remember that your cont be pointing to a pre-existing raw device or cluster file system location. *** Path names, file names, and sizes will need to be modified Example parameter settings for the first instance: Cluster-Wide Parameters for Database "RAC": db_block_size=8192 db_cache_size=52428800 background_dump_dest=/u01/32bit/app/oracle/product/9.0.1/rdbms/log core_dump_dest=/u01/32bit/app/oracle/product/9.0.1/rdbms/log user_dump_dest=/u01/32bit/app/oracle/product/9.0.1/rdbms/log

control_files=("/dev/RAC/control_01.ctl", "/dev/RAC/control_02.ctl")

pe: te: 30-MAY-

2003 on 07-DEC-

CREATE DEFAULT TABLESPACE, WHILE CREATINcreate a new user it will, use the default permanent tablespace, unless DBA is m twhile creating the user. To use default tablespace option, it is m ter "C SCOPE & APPLICATION -------------------

Trs mode.

.

=============

=============

irectory. On Windows this copy init.ora to rol file must

=

timed_statistics=TRUE


db_name=RAC shared_pool_size=52428800 sort_area_size=524288 undo_management=AUTO cluster_database=true cluster_database_instances=2 remote_listener=LISTENERS_RAC Instance Specific Parameters for Instance "RAC1":

instance_name=RAC1 instance_number=1 local_listener=LISTENER_RAC1

thread=1 undo_tablespace=UNDOTBS * The local_listener parameter requires that you first add th address to the TNSNAMES.ORA - remember to do so on both ** You can also use an spfile as described in Note 136327.1.

e listener ode 1 and Node 2. N

2. Run the following sqlplus command to connect to the database: sqlplus '/ as sysdba' 3. Startup up the database in NOMOUNT mode: SQL> startup nomount 4. Create the Database (All raw devices must be pre-created) : *** Path names, file names, and sizes will need to be modified CREATE DATABASE <db_name>

CONTROLFILE REUSE MAXDATAFILES 254 MAXINSTANCES 32 MAXLOGHISTORY 100 MAXLOGMEMBERS 5 MAXLOGFILES 64 DATAFILE '/dev/RAC/system_01_400.dbf' SIZE 900M segment space management auto REUSE AUTOEXTEND ON NEXT 10240K MAXSIZE UNLIMITED UNDO TABLESPACE "UNDOTBS" DATAFILE '/dev/RAC/undotbs_01_210.dbf' SIZE 200M REUSE

DEFAULT TABLESPACE USER_DEFAULT DATAFILE '/u01/oracle/rbdb1/user_default_1.dbf' size 2000M REUSE segment space management auto SYSAUX DATAFILE '/u01/oracle/rbdb1/sysaux_1.dbf' size 500M REUSE segment space management auto CHARACTER SET US7ASCII LOGFILE GROUP 1 ('/dev/RAC/redo1_01_100.dbf') SIZE 100M REUSE, GROUP 2 ('/dev/RAC/redo1_02_100.dbf') SIZE 100M REUSE; 5. Create a Temporary Tablespace: *** Path names, file names, and sizes will need to be modified CREATE TEMPORARY TABLESPACE "TEMP" TEMPFILE '/dev/RAC/temp_01_50.dbf' SIZE 40M REUSE 6. Create a 2nd Undo Tablespace:


*** Path names, file names, and sizes will need to be modif CREATE UNDO TABLESPACE "UNDOTBS2" DATAFILE '/dev/RAC/undotbs_02_210.dbf' SIZE 200M REUSE NEXT 5120K MAXSIZE UNLIMITED; 7. Run the necessary scripts to build views, synonyms, et The primary scripts that you must run are: i> CATALOG.SQL--creates the views of data dictionary ta es and dynamic performance views ii> CATPROC.SQL--establishes the usage of PL/SQL functionality and creates many of the PL/SQL Oracle supplied packages iii> CATPARR.SQL--creates RAC specific views 8. Edit init<SID>.ora and set appropriate values for the 2nd instance on the 2nd Node: *** Names may need to be modified instance_name=RAC2

instance_number=2 local_listener=LISTENER_RAC2 thread=2 undo_tablespace=UNDOTBS2 9. From the first instance, run the following command: *** Path names, file names, and sizes will need to be modified alter database add logfile thread 2 group 3 ('/dev/RAC/redo2_01_100.dbf') size 100M,

group 4 ('/dev/RAC/redo2_02_100.dbf') size 100M; alter database enable public thread 2; 10. Start the second Instance. (Assuming that your cluster configuration is up and running)

14.4.2 Configure listener.ora / sqlnet.ora / tnsnames.ora Use netca and/or netmgr to check the configuration of the listener and configure Oracle Net services (by default the Net service may be equal to the global database name (see instance parameter service_names ).

14.4.3 Configure Oracle Enterprise Manager Then start the OEM agent: $agentctl start

- Check /etc/oratab

The file should contain a reference to the database name, not to the instance name. The last field should alway be “N” on a RAC environment to e name to be started.

- Register the database with srvctl (this should not be necessary if the da

DBCA) srvctl add db –p <db_name> –o <ORACLE_HOMEsrvctl add instance –p <db_name> –i <SID1>

ied

c.:

bl the

s avoid 2 instances of the sam

tabase was not created by

path> –n <node1>


srvctl add instance –p <db_name> –i <SID2> –n <node1>


15 GRID CONTROL INSTALLATION

15.1 GRID CONTROL MANAGEMENT SERVER INSTALLATION

Port 1521 must not be used by any other Oracle Listener, or other process

IF AIX5L release 5.3 is used, please apply same AIX required APAR/Filesets for RAC installation on

AIX5.3 (read first chapters of this cookbook) As root user, execute : xhost + Login as oracle on the server which will host the Grid Control Management Server

Grid Control Server Must be different then RAC nodes

nd follow the procedure hereunder…


- export DISPLAY=....... - export TMP=/tmp With /tmp or other destination having enough free space- export TEMP=/tmp

A

, about 500Mb


IF AIX5L release 5.3 is used, do modify the file oraparam.ini, and cl ler

update entries AIX5200 to AIX5300 on both files, and execute : $/<cdrom_mount_point>/runInstaller


uster.ini in Disk1/instal

nt actions) and will go to

OUI (Oracle Universal Installer) chek the operating system requireme s for AIX5L 5.2.0.0. If AIX maintenance level 1, 2, 3 or/and 4 are installed, the installer will notice (no further the next step. To chek AIX mantenance level installed on each node : -> instfix –i|grep ML All filesets for 5.2.0.0_AIX_ML were found. All filesets for 5200-01_AIX_ML were found. All filesets for 5200-02_AIX_ML were found. All filesets for 5200-03_AIX_ML were found. All filesets for 5200-04_AIX_ML were found.

OUI (Oracle Universal Installer) asks you to run rootpre.sh as root. At the OUI Welcome screen, click Next.

Make sure to execute rootpre.sh before you go to the next step

he

Loader for AIX Configure and load

t_kernel Configure Asynchronous I/O And POSIX Asynchronous I/O Then Return to previous screen to

Running trootpre.sh

Install Oracle Kernel Extension

pw-syscall.64bi

answer yes ...


RunInstaller and Java Cryptography Extension (JCE)

patch to the jce-1_2_2.zip file

Then Return ...

Provide

RunInstaller and Java Cryptography Extension (JCE) Checking the Operating System Level

Then Return ...

Welcome Screen

Then click Next ...



Specify Inventory directory and redentials

validate efault directory.

nd Choose group “dba”

Then click Next ...

c Enter Inventory directory, ord A

OrainstRoot.sh As root, execute OrainstRoot.sh In the specified directory

Then click Next ...

{node1:root}/oragridctl/oraInventory -> ./orainstRoot.sh Creating the Oracle inventory pointer file (/etc/oraInst.loc) Changing groupname of /oragridctl/oraInventory to dba.

ode1:root}/oragridctl/oraInventory ->

Content of /etc/oraInst.loc : inventory_loc=/oragridctl/oraInventory inst_group=dba

rainstRoot.sh

O

Then click Next ... {n

Specify File Locations

Then click Next ...

Enter the Grid Control ORACLE-HOME and name

Select a Product to Install

Select “Enterprise Manager 10g Grid Control Using a New Database”

Then click Next ...

Product-specific Prerequisite Checks Don’t take care of ‘Not executed’ message, it’s just because the AIX Maintenance level is higher than specified

Then click Next ...


Set Up Administrator for Enterprise Manager Enter password for sysman user Choose email notification or not

Then click Next ...

Sysman Password

SMAN password must start with at east one

mber.

Then click OK ...

SYleast 5 character and at lnu

Specify Database Passwords Specify same password for all administrator users, or specify individual password for each user.

Then click Next ...


Metalink and Proxy Information :

Enter Metalink user and password, if available ...

Then click Next ...

Privileged Operating System Groups Specify group “dba” for both group

Then click Next ...

Database Identification Specify the “Global Database Name” The “SID Prefix” will be automatically updated. (by default it is the Global Database Name)

Then click Next ...


Database File Location

Specify directory destination for Grid Control database datafiles

Then click Next ...

Summary

Then click Nex

t ...

Install

Just wait while processing ...


Install : Setup Privileges Need to execute root.sh script as root

...

Execute Root.sh as root in the specified directory Then return to previous screen to click on Ok ...

Configuration Assistants Check all status to be “Succeeded”

Then click Next ...


Database Configuration Assistant

Just wait while processing ...

Configuration Assistants

Carry on processing …

Check all status to be “Succeeded”

Then click Next ...


Configura

on Session r Oracle Agent

Then click Next ...

Specify File locations :

nter ORACLE_HOME name and irectory destination for Grid Agent.

rid Control

RACLE_HOME

Accept the default proposed values

Then click Next ...

Select a product to install : Choose “Additional Management Agent”

Then click Next ...

tion Assistants

Launching Second Installatifo

Ed Must be Different ORACLE_HOMEthen the GO



Specify Oracle Manaement Service

Specify the name of the management server (Actual server on which you are running the Grid installation) Memorize the default management service port : 4889

Then click Next ...

arning :

ead the warning windows

Then click OK ...


Location :

W R

Summary Check the details and


Installation in progress

Then click Next ...

Setup Privileges : Open a telnet session, and ... Execut

Executing root.sh as root

Then go back to previous screen to click OK ...

Wait while processing ...

e root.sh as root

...




ssistants will execute automatically

Then click Next ...

nd of Installation :

ccess the README_EM.txt fil for techn

Then click Exit ...

A Verify status is succeded at the end,and retry if necessary

E A

ical details.

ps -ef|grep emrep

racle 123082 1 0 12:15:58 - 0: o 00 ora_qmn0_emrep 2 1 0 12:15:58 - 0:02 ora_ckpt_emrep

gwr_emrep emrep

ep ep

C

:09 C:01 C:00 C:00 C

Check the running processes

Then click Next ...

on. Al----------------------------------

//node4:1830/emd/main/ -09-21 12:21:51

arted by user : oracle

: (none) : (none)

Total Megabytes of XML files uploaded so far : 0.00 Number of XML files pending upload : 0 Size of XML files pending upload(MB) : 0.00 Available disk space on upload filesystem : 37.82% --------------------------------------------------------------- Agent is Running and Ready

agent status

Then click Next ...

oracle 17210 oracle 217310 1 0 12:15:58 - 0:00 ora_l oracle 233494 1 0 12:15:58 - 0:00 ora_cjq0_ oracle 258186 1 0 12:15:58 - 0:00 ora_smon_emrep oracle 303274 1 0 12:15:58 - 0:00 ora_dbw0_emr

oracle 307318 1 0 12:15:58 - 0:00 ora_pmon_emr oracle 331942 139416 0 13:06:48 pts/0 0:00 grep emrep oracle 479436 1 0 12:22:20 - 0:00 ora_j002_emrep

emrep oracle 487664 1 0 12:15:58 - 0:00 ora_reco_ oracle 503952 1 0 12:22:47 - 0:00 oracleemrep (LOCAL=NO) oracle 516274 1 0 12:22:20 - 0:00 ora_j001_emrep

oracle 520250 1 0 12:22:51 - 0:00 oracleemrep (LO AL=NO) oracle 561186 1 0 12:22:20 - 0:12 ora_j000_emrep

oracle 589866 1 0 12:22:20 - 0:00 ora_j004_emrep oracle 602154 1 0 12:22:20 - 0:00 ora_j003_emrep oracle 610348 1 0 12:22:21 - 0:00 ora_j005_emrep oracle 626846 1 0 12:22:52 - 0 oracleemrep (LO AL=NO) oracle 655480 1 0 12:22:51 - 0 oracleemrep (LO AL=NO) oracle 659588 1 0 12:22:51 - 0 oracleemrep (LO AL=NO)

oracle 663696 1 0 12:22:52 - 0 oracleemrep (LO AL=NO) oracle 671816 1 0 12:22:51 - 0:01 oracleemrep (LOCAL=NO) oracle 729092 1 0 12:59:39 - 0:02 oracleemrep (LOCAL=NO)

oracle 741454 1 0 12:59:39 - 0:00 oracleemrep (LOCAL=NO)

ctl status agent em

Check GridOracle Enterprise Manager 10g Release 10.1.0.2.0.

opyright (c) 1996, 2004 Oracle Corporati l rights reserved. C

-----------------------------ersion : 4.0.1.0.0V

Agent Home : /oragridctl/grid10g gent Process ID : 581686 A

Parent Process ID : 557086 Agent URL : http:Started at : 2004StLast Reload : 2004-09-21 12:21:51 Last successful upload

Last attempted upload


To start/Stop all components from Grid Control (oms, http, OC4J, WebCache, ..) $ORACLE_HOM

E/opmn/bin/opmnctl startall

n/ ll

n and all managed processes...

acc l interface 2/drivers/etc/hosts ame http://Om User to login : sysman

opmnctl: starting opmn and all managed processes... $ORACLE_HOME/opmn/bin/opmnctl status

$ORACLE_HOME/opm bin/opmnctl stopta

opmnctl: stopping opm

To ess Enterprise Manager Grid Control htm

Add entrie in your local winnt/system3

OmsServerIP OmsServerNex.: 10.2.12.21 dbs13

Then :

sServerName:4889/em

Password : ********


Post-Installation Oracle Grid Control unix Profile Update :

To Oracle in Oracle’s home directory Add th export ORACLE_BASE=/oh10g exporumask 022 export ORACLE_HOME=$ORACLE_BASE/gridcontrol export LD_LIBRARY_PATH=$ORACLE_HOME/lib: $ORACLE_HOME/lib32 expor

15.2

Doc I

be done on each node.

environment : vi $HOME/.profile file

e entries in bold blue color

t AIXTHREAD_SCOPE=S

t PATH=$ORACLE_HOME/bin: $PATH

METALINK NOTE

D: Note:266769.1 Subject: EM 10g GRID Control

Release Notes 10.1.0.2.0 Type: README Status: PUBLISHED

Content Type: TEXT/X-HTML Creation Date: 29-MAR-2004 Last Revision Date: 04-AUG-2004


15.3 GRID CONTROL AGENT DEPLOYMENTS

As root user on the first RAC cluster node, execute : xhost +

nd follow the procedure hereunder…

AY, TMP and TEMP variables export DISPLAY=....... export TMP=/tmp With /tmp or other destination having enough free space, about 500Mb exp MP=/tmp

is used, do modify the file oraparam.ini, and cluster.ini in Disk1/installer

d execute : >/runInstaller


Login as oracle, a

Setup and export your DISPL

ort TE

IF AIX5L release 5.3

update entries AIX5200 tcdrom_mount_point

o AIX5300 on both files, an$/<

perating system requirements for AIX5L 5.2.0.0. If AIX , the installer will notice (no further actions) and will go to the

x To-> All filesets for 5.2.0.0_AIX_ML were found. All filesets for 5200-01_AIX_ML were found. All filesets for 5200-02_AIX_ML were found. All filesets for 5200-03_AIX_ML were found. All filesets for 5200-04_AIX_ML were found. OU ller) asks you to run rootpre.sh as root. At ick Next.

OUI (Oracle Universal Installer) chek the o

aintenance level 1, 2, 3 or/and 4 are installedmne t step.

chek AIX mantenance level installed on each node : instfix –i|grep ML

I (Oracle Universal Instathe OUI Welcome screen, cl

!!! You must do it , if not already done !!!


RunInstaller and Java Cryptography Extension

Provide patch to the jce-1_2_2.zip file Then Return

...

(JCE)

RunInstaller and Java Cryptography Extension (JCE) Checking the Operating System Level

Then Return ...

Welcome Screen

Then click Next ...



Specify Hardware Cluster Installation Mode : “Cluster Installation” (Select N

Specify File Locations ... Specific ORACLE_HOME Name

Then click Next ...

ode

lation” (Need to n

each node)

Select a product to install :

And ORACLE_HOME directory

Name) Or “Local Instalproceed the full installation process o

Then click Next ...

“Additional Management Agent”

Then click Next ...

Checking the summary for remote nodes ...

C

hecking the summary ...


Installation in progress ...

Then click Next ...


Installation in progress ... Copying binaries from node 1 to remote

nodes.

Then click Next ...

Setup Privileges : A root.sh script has to be run on each node where Grid Control Agent is deployed.

Then click Next ...


Setup Privileges :

where the agent is deployed

Then click Next ...

Running root.sh on each nodes


e Agent configuration

Then click Next ...

Going thru thassistant.


If you get this error message, it will

node 1 (able to start and stop the agent), and not completed on other nodes.


mean that Grid Control Agent has been well deployed and configured on

Solutions : Running a local Grid ControAgent installation on each other nodes,

l

modifications on each node.

Then click OK ...

or copying binaries from node 1 to the others, and process configuration files

Then click N

To start/stop the Oracle Grid Control Intelligent Agent : agentctl start agentctl stop

Assistants :

ext to Finish ...

agentctl status

Configuration


16 APPENDIX A : LOGICAL VOLUMES CREATION. This is for raw devices implementation

disk' –t’jfs2’ oradatvg 2 ext_disk # 128 MB

B

# 256 MB # 256 MB

lv -y'r sk # 64 MB

'rac_srvconfig' –t’jfs2’ oradatvg 2 ext_disk # 128 MB

s2’ oradatvg 8 ext_disk # 512 MB

.dba /dev/r* /dev/r*

#!/bin/ksh # Creation of a concurrent volume group # To be executed on the primary node (node1) # 64 is a free major number. It has to be unused on all the nodes mkvg -f –c -y oradatvg –V'80' -s'64' <disk list> # Creation of the minimum logical volumes (raw devices)

64 MB physical partitions) # The number is the size of the LV (number of# To be exectued on the primary node (node1)

lv -y'ocr_disk' –t’jfs2’ oradatvg 2 ext_disk # 128 MB mkmklv -y'vote_ mklv -y'rac_system' –t’jfs2’ oradatvg 8 ext_disk # 512 M mklv -y'rac_undotbs01' –t’jfs2’ oradatvg 8 ext_disk # 512 MB

512 MB mklv -y'rac_undotbs02' –t’jfs2’ oradatvg 8 ext_disk #

oradatvg 4 ext_disk # 256 MB mklv -y'log11' –t’jfs2’mklv -y'log12' –t’jfs2’ oradatvg 4 ext_disk mklv -y'log21' –t’jfs2’ oradatvg 4 ext_disk mklv -y'log22' –t’jfs2’ oradatvg 4 ext_disk # 256 MB mklv -y'rac_control01' –t’jfs2’ oradatvg 1 ext_disk # 64 MB

sk # 64 MB mklv -y'rac_control02' –t’jfs2’ oradatvg 1 ext_diac_control03' –t’jfs2’ oradatvg 1 ext_dimk

mklv -y'rac_spfile' –t’jfs2’ oradatvg 1 ext_disk # 32 MB

lv -ymk

lv -y'rac_data' –t’jfmkmklv -y'rac_index' –t’jfs2’ oradatvg 8 ext_disk # 512 MB mklv -y'rac_temp' –t’jfs2’ oradatvg 8 ext_disk # 512 MB varyoffvg oradatvg # To be exectued on the secondary node (node2) importvg -y oradatvg –V80 <one of the disks> varyonvg oradatvg chfs –c oradatvg

To be exectued on all the nodes (node1 & node2) when oradatvg is varied on # chown oracle

mod go+rw ch


17 PENDIX B : EXAMPLES OF CONFIGURATION FILES AP17.1 NETWORK

This appendix provides examples of the configuration files that are mentioned in the document. /etc/host

2.12.81 node1

10.2.12.82 node2

10.2.12.83 node3

# Virtual IP address

s (on all nodes)

ork

root

.rhosts

In the root’s and oracle’s home directory, put the list of machines. $HOME/.rhosts

node1 root

# Public Netw

10.

10.2.12.181 node1_vip

10.2.12.182 node2_vip

10.2.12.183 node3_vip

# Interconnect RAC & GPFS

10.10.12.81 node1_gpfs

10.10.12.81 node2_gpfs

10.10.12.81 node3_gpfs

/etc/hosts.equiv (on all nodes)

node1 root

node2 root

node3 root

node1_vip

node2_vip root

node3_vip root

node1_gpfs root

node1 oracle

node2 oracle

e1_vip oracle

nod

nod

e2_vip oracle

e3_vip oracle

node1_gpfs oracle

nod

node3_gpfs oracle

node2_gpfs root

node3_gpfs root

node3 oracle

nod

e2_gpfs oracle


node2 root

node3

node3_vip root

node1_gpfs root

node2_gpfs root

node1 oracle

node2 oracle

node3 oracle

node1_vip oracle

node2_vip oracle

node3

node1_gpfs

node2_gpfs oracle

node3_gpfs

root

root

root

_vip oracle

oracle

oracle

node1_vip

node2_vip

node3_gpfs root


17.2 LISTENER.ORA AND TNSNAMES.ORA CONFIGURATION EXAMPLE

listener.ora

LISTENER = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = NODE1_VIP)(PORT = 1521)) ) ) ) SID_LIST_LISTENER = (SID_LIST =

ID_DESC = (S (ORACLE_HOME = /oracle/pr )

) t

(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = node2_vip)(PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = RAC)

RAC1 = (DESCRI (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = nod

ONNECT_DATA = (SERVICE_NAME = RAC) (INSTANCE_NAME = RAC1) ) )

oduct/10.1.0) (SID_NAME = RAC1)

nsnames.ora implementing TAF

RAC2 =

PTION =

e1_vip)(PORT = 1521))

RAC =

(ADDRESS = (PROTOCOL = TCP)(HOST = node1_vip)(PORT = 1521))

(FAILOVER_MODE = LECT) PRECONNECT) 20)

(INSTANCE_NAME = RAC2) ) )

) (C

(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = node2_vip)(PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = RAC)

(TYPE = SE (METHOD = (RETRIES = (DELAY = 60) ) ) )


18 APPENDIX C : ORACLE TECHNICAL NOTES

x provides some useful not i Oracl . These not n M nk.

I

This appendi es com ng from e support etali

18.1 CRS AND 10G REAL APPL USTERS

oc ID: Note:259301.1 ubject: CRS andS 10g Real Application

TStatus: PUBLISHED

ent HTM

tion st Rev

Date: -MAR-

2005

--

ov i inf Sein 10g Real Application Clusters. COPE & APPLICATION

This document is intended for RAC Database Administrators and Oracle support enginneers. CRS and 10g REAL APPLICATION CLUSTERS ------------------------------------- CRS (Cluster Ready Services) is a new feature for 10g Real Application Clusters that provides a standard cluster interface on all platforms and performs new high availability operations not available in previous versions. CRS KEY FACTS ------------- Prior to installing CRS and 10g RAC, there are some key points to remember about CRS and 10g RAC: - CRS is REQUIRED to be installed and running prior to installing 10g RAC. - CRS can either run on top of the vendor clusterware (such as Sun Cluster, HP Serviceguard, IBM HACMP, TruCluster, Veritas Cluster, Fujitsu Primecluster, etc...) or can run without the vendor clusterware. The vendor clusterware was required in 9i RAC but is optional in 10g RAC. - The CRS HOME and ORACLE_HOME must be installed in DIFFERENT locations. - Shared Location(s) or devices for the Voting File and OCR (Oracle

Clusters ype: BULLETIN

Cont Type: TEXT/X-L

CreaLa

EC-2003

PURPOSE -----

is document is to prrvices)

ide add tional ormation on CRS (Cluster

S-------------------

es can be found i

CATION CL

D

Date: 05-Dision 15

Th Ready




Configuration The

Repository) file must be available PRIOR to installing CRS.

voting file should be at least 20MB and the OCR file should be at least

CRS and RAC require that the following network interfaces be configured

g CRS r RAC: - Public Interface

ac f

For more information on this, see N

100MB. -prior to installin o

- Private Interf- Virtual (Publi

ce ) Inter ace

ote 264847.1. - at the of th tion starts the CRstThe root.sh scripack.

t end e CRS installa S

If your CRS stack does not start, see Note 240001.1. - Only one set of CRS daemons can be running per RAC node.

"r

or more nodes may reboot automatically to prevent data corruption.

met tart CRS bo hi NUAL S

THE CRS STACK IS NOT SUPPORTED UNTIL 10.1.0.4 OR HIGHER. - The supported method to stop is shutdown the machine or use "init.crs o

- Killing CRS daemons is not supported unless you are removing the CRS i a 998.1

- On Unix, the CRS stack is run from entries in /etc/inittab with espawn".

- If there is a network split (nodes loose communication with each other). One

- OFThe supported

hod to s is oting the mac ne. MA TARTUP

st p".

nstallation vi Note 239 cau

r mod

flag files ca

at the OS.

become ched.

or maintenance,

Once the stack is cesses

tarted, ld able to see a of the n

[rac1]/u01/home/beta> ps -ef | grep crs

1

or

acle 1002 1 0 11:2 0:0 u01/crs_h n/ocssd

Here is a short de

D:

cription of e CRS daemon ocesses

- E- M

ngine for HA anages 'appli

ration ion reso

be se n mismat

- F go to single use e

s you shou be ll daemoprowith a ps -ef command:

oracle 1363 999 0 11:23:21 ? 0:00 /u01/crs_home/bin/evmlogger.bin -o /u0 oracle 999 1 0 11:21:39 ? 0:01 /u01/crs_home/bin/evmd.bin root 1003 1 0 11:21:39 ? 0:01 /u01/crs_home/bin/crsd.bin

1:39 ? 1 / ome/bi .bin

CRS DAEMON FUNCTIONALITY ------------------------

s of each th pr : CRS

opecat urces'

- Starts, stops, and fails 'application resources' over - Spawns separate 'actions' to start/stop/check application resources


- Maintains confRepository)

iguration profiles in the OCR (Oracle Configuration

Stores current known state in the OCR. as root started aut al e

- OCSSD is part of RAC and Single Instance with ASM - Provides access to node membership - Provides group services

ctew e

hout integration to ve tware - Runs as Oracle.

u

- Spawns a permanent child evmlogger - Evmlogger, on demand, spawns children - Scans callout directory and invokes callouts. Runs as Oracle.

------------------

S Home

$ORA_CRS_HOME/crs/log - This directory includes traces for CRS resources thjoining, leaving, restarting, and relocating as identified by CRS.

Any core dumps for the crsd.bin daemon should be

ere. Note 1812.1

-- Runs - Is re omatic ly on failur OCSSD:

- - -

Provides basic Integrates with Can also runs wit

luster lexisting

ocking vendor clus are, when pr

ndor clussent

- --- This is a feature to prevent data corruption in event of a split brain. Failure exit ca ses machine reboot.

EVMD: - Generates events when things happen

- - Restarted automatically on failure

CRS LOG DIRECTORIES - When troubleshooting CRS problems, it is important to review the directories un der the CR

at are

.

$ORA_CRS_HOME/crs/init -written h could be used to debug these.

reconfigurations, missed checkins , connects, and disconnects from the clCSS listener . In some cases the logger logs messages with the category of

or th s acle uld king thexact time when the reboot occured.

ss daemon whose death is treated as fatal are located here. If there are bnormal restarts for css then the core files will have the formats of

8

$ORA_CRS_HOME/css/log - The css logs indicate all actions such as

ient

(auth.crit) fe

e reboot done by or . This co be used for chec

$ORA_CRS_HOME/css/init - Core dumps from the ocssd primarily and the pid for the cacore.<pid>. Note 1 12.1 could be used to se. ORA_CRS_HOME/evm/log - Log files for the evm and evmlogger daemons. Not sed

$ORA_CRS_HOME/evm/init - Pid and lock files for EVM. Core files for EVM should also be written here. Note 1812.1

debug the

$uas often for debugging as the CRS and CSS directories.


could be used to debug these.

http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=271297.1

$ORA_CRS_HOME/srvm/log - Log files for OCR.

E-

After installing RAC and running the VIPCA (Virtual IP Configuration

launched with the RAC root.sh, you should be able to see all of your CRS resources with crs_stat. Example:

cd $ORA_CRS_HOME/bin

NAME=ora.rac1.gsd

pplication TARGET=ONLINE

NAME=ora.rac1.oem TYPE=application

STATE=ONLINE

s

TARGET=ONLINE

vip

TARGET=ONLINE

NAME=ora.rac2.gsd TYPE=application

NAME=ora.rac2.oem TYPE=application

E

2.onson

.vip

TARGET=ONLINE

ST--ATUS FOR CRS R--------------

SOURCES -------

Assistant)

./crs_stat

TYPE=a S TATE=ONLINE

TARGET=ONLINE

N TAME=ora.rac1.onYPE=application

STATE=ONLINE

N TYPE=application AME=ora.rac1.

STATE=ONLINE

T SARGET=ONLINE TATE=ONLINE

TARGET=ONLINSTATE=ONLINE N TAME=ora.racYPE=applicatiTARGET=ONLINE STATE=ONLINE

N AME=ora.rac2TYPE=application

STATE=ONLINE


There is also a script available to view CRS resources in a format that is as st cre te a s sc

--------------------------- Begin Shell Script ------------------------------- #!/usr/bin/ksh # # Sample 10g CRS resource status query script # # Description: # - Returns formatted version of crs_stat -t, in tabular # format, with the complete rsc names and filtering keywords # - The argument, $RSC_KEY, is optional and if passed to the script, will # limit the output to HA resources whose names match $RSC_KEY. # Requirements: # - $ORA_CRS_HOME should be set in your environment RSC_KEY=$1 QSTAT=-u AWK=/usr/xpg4/bin/awk # if not available use /usr/bin/awk # Table header:echo "" $AWK \ 'BEGIN {printf "%-45s %-10s %-18s\n", "HA Resource", "Target", "State"; printf "%-45s %-10s %-18s\n", "-----------", "------", "-----";}' # Table body: $ORA_CRS_HOME/bin/crs_stat $QSTAT | $AWK \ 'BEGIN { FS="="; state = 0; } $1~/NAME/ && $2~/'$RSC_KEY'/ {appname = $2; state=1}; state == 0 {next;} $1~/TARGET/ && state == 1 {apptarget = $2; state=2;} $1~/STATE/ && state == 2 {appstate = $2; state=3;} state == 3 {printf "%-45s %-10s %-18s\n", appname, apptarget, appstate; state=0;}' --------------------------- End Shell Script ------------------------------- Example output: [opcbsol1]/u01/home/usupport> ./crsstat HA Resource Target State ----------- ------ ----- ora.V10SN.V10SN1.inst ONLINE ONLINE on opcbsol1 ora.V10SN.V10SN2.inst ONLINE ONLINE on opcbsol2 ora.V10SN.db ONLINE ONLINE on opcbsol2 ora.opcbsol1.ASM1.asm ONLINE ONLINE on opcbsol1 ora.opcbsol1.LISTENER_OPCBSOL1.lsnr ONLINE ONLINE on opcbsol1 ora.opcbsol1.gsd ONLINE ONLINE on opcbsol1 ora.opcbsol1.ons ONLINE ONLINE on opcbsol1 ora.opcbsol1.vip ONLINE ONLINE on opcbsol1 ora.opcbsol2.ASM2.asm ONLINE ONLINE on opcbsol2 ora.opcbsol2.LISTENER_OPCBSOL2.lsnr ONLINE ONLINE on opcbsol2 ora.opcbsol2.gsd ONLINE ONLINE on opcbsol2 ora.opcbsol2.ons ONLINE ONLINE on opcbsol2 ora.opcbsol2.vip ONLINE ONLINE on opcbsol2 CRS RESOURCE ADMINISTRATION

e

ier to read. Ju a hell ript with:


---------------------------

examp

--------------------------------------------------------------------------

tabase-name> -i <instance-name> >[,<instance-

el>] se-name> -s <service-name>[,<service-name-

<level>] <node-name>]

ode_name>

stances and all services. srvctl status database -d ORACLE -v f named instances with their current services.

atus of all nodes supporting database applications. srvctl status node

------------------------------------------------------

ART CRS RESOURCES

-options>]

-name-list>] [-o <start-options>] [-c <connect-string> | ]

ice-name>[,<service-name-

[-i <instance-name>] [-o <start-options>] [-c <connect-string> | -q]

Start the database with all enabled instances. srvctl start database -d ORACLE

srvctl start instance -d ORACLE -i RAC03, RAC04 es are started as needed.

srvctl start service -d ORACLE -s CRM

srvctl start service -d ORACLE -s CRM -i RAC04

srvctl start nodeapps -n myclust-4

You can use srvctl to manage these resources. Below are syntax and

les. -----

CRS RESOURCE STATUS srvctl status database -d <database-name> [-f] [-v] [-S <level>] srvctl status instance -d <daname-list>] [-f] [-v] [-S <levsrvctl status service -d <databalist>] [-f] [-v] [-Ssrvctl status nodeapps [-n srvctl status asm -n <n EXAMPLES: atus of the database, all inSt

Status o srvctl status instance -d ORACLE -i RAC01, RAC02 -v Status of a named services. srvctl status service -d ORACLE -s ERP -v St ------------------------- ST srvctl start database -d <database-name> [-o < start [-c <connect-string> | -q]

srvctl start instance -d <database-name> -i <instance-name> [,<instance-qsrvctl start service -d <database-name> [-s <servst>]] li

srvctl start nodeapps -n <node-name> srvctl start asm -n <node_name> [-i <asm_inst_name>] [-o <start_options>] AMPLES: EX

Start named instances.

Start named services. Dependent instanc Start a service at the named instance.

Start node applications.


---------------------------------------------------------------------------

STOP CRS RESOURCES vctl stop database -d <database-name> [-o <stop-options>]

vctl stop instance -d <database-name> -i <instance-name> [,<instance-

[-o <stop-options>][-c <connect-string> | -q] -name>[,<service-name-

st>]]

Stop the database, all instances and all services.

Stop named instances, first relocating all existing services.

srvctl stop service -d ORACLE -s CRM top the service at the named instances.

srvctl stop service -d ORACLE -s CRM -i RAC04 ons. Note that instances and services also stop.

nodeapps -n myclust-4

-------------------------------------------------------------------------------

omain_name>] [-p

<spfile>] [-A <name|ip>/netmask] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY}] [-s <start_options>] [-n <db_name>] srvctl add instance -d <name> -i <inst_name> -n <node_name> srvctl add service -d <name> -s <service_name> -r <preferred_list> [-a <available_list>] [-P <TAF_policy>] [-u] srvctl add nodeapps -n <node_name> -o <oracle_home> [-A <name|ip>/netmask[/if1[|if2|...]]] srvctl add asm -n <node_name> -i <asm_inst_name> -o <oracle_home> OPTIONS: -A vip range, node, and database, address specification. The format of address string is: [<logical host name>]/<VIP address>/<net mask>[/<host interface1[ | host interface2 |..]>] [,] [<logical host name>]/<VIP address>/<net mask> [/<host interface1[ | host interface2 |..]>] -a for services, list of available instances, this list cannot include preferred instances -m domain name with the format “us.mydomain.com” -n node name that will support one or more instances -o $ORACLE_HOME to locate Oracle binaries -P for services, TAF preconnect policy - NONE, PRECONNECT -r for services, list of preferred instances, this list cannot include

----

sr [-c <connect-string> | -q] srname-list>] srvctl stop service -d <database-name> [-s <serviceli [-i <instance-name>][-c <connect-string> | -q] [-f] srvctl stop nodeapps -n <node-name> srvctl stop asm -n <node_name> [-i <asm_inst_name>] [-o <start_options>] EXAMPLES:

srvctl stop database -d ORACLE

srvctl stop instance -d ORACLE -i RAC03,RAC04 Stop the service.

S Stop node applicati

srvctl stop

ADD CRS RESOURCES

srvctl add database -d <name> -o <oracle_home> [-m <d


available instances. -s spfile name -u updates the preferred or available list for the service to support he specified instance. Only one instance may be specified with the -u switch. Instances that already support the service should not be included.

EXAMPLES:

ps -n myclust-1 -o $ORACLE_HOME –A 139.184.201.1/255.255.255.0/hme0 Add a new database. srvctl add database -d ORACLE -o $ORACLE_HOME

to an existing database. ance -d ORACLE -i RAC01 -n myclust-1

srvctl add instance -d ORACLE -i RAC02 -n myclust-2 srvctl add instance -d ORACLE -i RAC03 -n myclust-3

referred instances (-r) and available instances (-a). Use basic failover to the available instances. srvctl add service -d ORACLE -s STD_BATCH -r RAC01,RAC02 -a

ances in list one and

vailable instances. 1,RAC02 -a

C04 -P PRECONNECT -------------------------------------------------------------------------

OVE CRS RESOURCES

ase -d <database-name> -i <instance-name>] <service-name> [-i <instance-

me>] ode-name>

ORACLE move the applications for named instances of an existing database.

srvctl remove instance -d ORACLE -i RAC03

ances.

t

Add a new node: srvctl add nodeap

Add named instances srvctl add inst

Add a service to an existing database with p

RAC03,RAC04 Add a service to an existing database with preferred inst

available instances in list two. Use preconnect at the asrvctl add service -d ORACLE -s STD_BATCH -r RAC0

RAC03,RA

------ REM srvctl remove databsrvctl remove instance -d <database-name> [rvctl remove service -d <database-name> -s snasrvctl remove nodeapps -n <n EXAMPLES: Remove the applications for a database. srvctl remove database -dRe srvctl remove instance -d ORACLE -i RAC04 move the service. Re

srvctl remove service -d ORACLE -s STD_BATCH Remove the service from the inst srvctl remove service -d ORACLE -s STD_BATCH -i RAC03,RAC04 emove all node applications from a node. R srvctl remove nodeapps -n myclust-4


---- -----------------------------------------------------------------------

- ---

me>] [-m <domain>] PHYSICAL_STANDBY | LOGICAL_STANDBY}]

y instance -d <database-name> -i <instance-name> -n <node-name> name> {-s <asm_inst_name> | -r}

srvctl modify service -d <database-name> -s <service_name> -i <instance-name>

stance-name> [-f] service -d <database-name> -s <service_name> -i <instance-

stance name (-i) is replaced

ame (-t) modified to be a preferred

stan add at node level

dency

. srvctl modify instance -d ORACLE -n myclust-4

2

ctl modify service -d ORACLE -s HOT_BATCH -i RAC02 –r

--------------------------------------------------------------

-s <service-name> [-i <instance-

MODIFY CRS RESOURCES srvctl modify database -d <name> [-n <db_name] [-o <oho [-p <spfile>] [-r {PRIMARY | [-s <start_options>]

srvctl modifrvctls modify instance -d <name> -i <inst_

-t <infysrvctl modi

name> -r [-f] srvctl modify nodeapps -n <node-name> [-A <address-description> ] [-x] OPTIONS:

me> the in-i <instance-name> -t <instance-nay the b instance ni <instance-name> -r the named instance is

ce in-A ress-list for VIP application,-s <asm_inst_name> add or remove ASM depen

EXAMPLES: odify an inM stance to execute on another node Modify a service to execute on another node. srvctl modify service -d ORACLE -s HOT_BATCH -i RAC01 -t RAC0Modify an instance to be a preferred instance for a service.

srv ---------------- - RELOCATE SERVICES srvctl relocate service -d <database-name> name >]-t<instance-name > [-f] XAMPL S: E E Relocate a service from one instance to another srvctl relocate service -d ORACLE -s CRM -i RAC04 -t RAC01


-------------------------------------------------------------------------------

ABLE CRS RESOURCES (The resource may be up or down to use this function)

-name> name> -i <instance-name> [,<instance-

able the database.

ervice -d ORACLE -s ERP,CRM

-------------------------------------------------------------------------

e down to use this function)

me-list>] [-i <instance-name>]

tabase globally.

------

ers Guide - Appendix B

RELATED DOCUMENTS ----------------- Oracle10g Real Application Clusters Installation and Configuration Oracle10g Real Application Clusters Administrator’s Guide

EN srvctl enable database -d <databasesrvctl enable instance -d <database-name-list>] srvctl enable service -d <database-name> -s <service-name>] [, <service-

e>] name-list>] [-i <instance-nam EXAMPLES: En srvctl enable database -d ORACLE Enable the named instances. srvctl enable instance -d ORACLE -i RAC01, RAC02 Enable the service. srvctl enable sEnable the service at the named instance. srvctl enable service -d ORACLE -s CRM -i RAC03 ------

DISABLE CRS RESOURCES (The resource must b srvctl disable database -d <database-name> srvctl disable instance -d <database-name> -i <instance-name> [,<instance-name-list>] rvctl disable service -d <database-name> -s <service-name>] [,<service-sna EXAMPLES: isable the daD srvctl disable database -d ORACLE Disable the named instances. srvctl disable instance -d ORACLE -i RAC01, RAC02 Disable the service globally. srvctl disable service -d ORACLE -s ERP,CRM Disable the service at the named instance. srvctl disable service -d ORACLE -s CRM -i RAC03,RAC04 ------------------------------------------------------------------------- For more information on this see the Oracle10g Real Application ClustAdministrator’s




18.2 ABOUT RAC ...

Subject Doc ID Modified Date Product Platform Type

1 Minimum software versions

and patches required to Support Oracle Products on ...

282036.1 21-FEB- Oracle2005 - Enterprise

Edition

Server

IBM RS 6000 AIX 5L

BULLETIN

2 Pre-Install checks for 10g RDBMS on AIX

283743.1 10-DEC- OSS Support IBM RS 6000 AIX 5L

RULE SETS 2004 Tools

3 RAC: Frequently Asked Questions

220970.1 16-MAR- Oracle Server rprise

Gen2005 - Ente

Edition

eric FAQ

4 Raw Devices and Cluster Filesystems With Real Application Clusters

183408.1 19-FEB- Oracle2005 - Ente

Server rprise

Generic BULLETIN

Edition5 10g Installation on Aix 5.3,

Failed with Checking operating system version mu...

293750.1 02-MAR- Oracle2005

Server - Enterprise

AIX-Based Systems (64-bit)

PROBLEM

Edition

18.3 ABOUT

ject

CRS ...

Sub Doc ID Modified Date Product Platform Type 1 10G: How to Stop the

Cluster Ready 1 2004 Enterprise Edition Services (CRS)

263897. 14-SEP- Oracle Server - Generic HOWTO

2 How to verify if CRS install

295871. 04-FEB- Oracle Server - Generic HOWTO is Valid 1 2005 Enterprise Edition

3 10g RAC: 265769. 16-MAR- Oracle Server - Generic TROUBLTroubleshooting CRS 1 2005 Enterprise Edition SHOOTINReboots

4. CRS and 10g Real 259301. 15-MAR- Oracle Server - Generic BULLETIN

E G

Application Clusters 1 2005 Enterprise Edition 5 Repairing or

Restoring an 268937.1

02-MAR-2005


Generic BULLETIN

Inconsistent OCR in RAC

6 Placement of voting and OCR disk files in

293819. 01-MAR- Oracle Server - Generic BULLETIN

10gRAC7 10g RAC: How to 239998.1 23- Oracle Server - Generic BULL

1 2005 Enterprise Edition

ETIN Clean Up After a FEB- Enterprise Edition Failed CRS Install 2005

8 CRS 10g Diagnostic Collection Guide

272332.1 15-NOV-


Generic BULLET

2004 9 How to Restore a 279793.1 09- Oracle Server - Generic PROBLEM

IN

Lost Voting Disk in 10g

OCT-2004

Enterprise Edition

10 10g RAC: Stopping Reboot Loops When

239989.1 03-MAY-


Generic PROBLEM

CRS Problems Occur

2004

11 HOW TO REMOVE CRS AUTO START

298073.1 11-FEB-

Enterprise Manager for

Generic BULLETIN

AND RESTART FOR A RAC

2005 RDBMS

INSTANCE12 HOW TO REMOVE

CRS AUTO START 298069.1 11-

FEB-Enterprise Manager for

Generic BULLETIN

AND RESTART 2005 RDBMS FOR A RAC INSTANCE

14 CRS Home Is Only 284949.1 Partially Copied to Remote Node 2004

15 How to

05-DEC-


Generic PROBLEM

recreate ONS,GSD,VIP

285046.1 08-

deleted from ocr by crs_

OCT-2004

Enterprise Edition Oracle Server - Generic HOWTO

unregisterInstalling Oracle 10g RAC on IBM pSeries with AIX 5L page 161/ 161

18.4 ABOUT VIP ...

Subject Doc ID Modified Date Product Platform Type 1 Configuring the IBM 296856.1 17

AIX 5L Operating System for the Oracle 10g VIP

2 Changing the check 294336.1 23-FEB

-FEB-2005 Oracle Server - Enterprise Edition

-2005 Oracle Server -

Enterprise Edition

Generic BULLETINinterval for the Oracle 10g VIP

2 Modifying the VIP of 276434.1 13-JAN-2005 Oracle Server - Enterprise Edition

Generic BULLETINa Cluster Node

3 Modifying the default 298895.1 07-MAR-2005 Oracle Server - Enterprise Edition

Generic BULLETINgateway address used by the Oracle 10g VIP

4 How to Configure 264847.1 24-FEB-2005 Oracle Server - Generic HOWTO Virtual IPs for 10g Enterprise RAC Edition

18.5 ABOUT MANUAL DATABASE CRATION ...

1 10g Manual 240052.1 07-DEC-2004 Oracle Generic BULLETIN

Database Creation Diagnostics in Oracle (Single Pack Instance and RAC)

18.6 ABOUT GRID CONTROL ...

1 Enterprise Manager

Grid Control 284707.1 04-OCT-2004 Oracle

Enterprise Generic BULLETIN

10.1.0.3.0 Release Notes

Manager

2 EM 10G Grid Control Preinstall Steps for AIX 5.2

277420.1 06-JUL-2004 Enterprise Manager Core

IBM RS 6000 AIX 5L

18.7 ABOUT TAF ...

1 Troubleshooting TAF 271297. 11-MAY-2004 Oracle Net Generic BULLETIN

Issues in 10g RAC 1 Services



18.8 ABOUT ADDING/REMOVING NODE ...

1 HRemoving a Node

from a 10g RAC Cluster H

269320.1 06-OCT-2004

Oracle Server -Enterprise Edition

Generic BULLETIN

2 HAdding a Node to a 10g RAC Cluster H

270512.1 14-JAN-2005

Oracle Server -Enterprise Edition

Generic BULLETIN

18.9 ABOUT ASM ...

HSubject H HHHDoc ID H HModified Date H HProduct H HPlatformH HType H

1 H10G New Storage

Features and Enhancements H

243245.1 04-AUG-2004 Oracle Server - Enterprise Edition

Generic BULLETIN

2 HRe-creating ASM Instances and Diskgroups H

268481.1 24-FEB-2005 Oracle Server - Enterprise Edition

Generic BULLETIN

3 HSGA sizing for ASM instances and databases that use ASMH

282777.1 10-SEP-2004 Generic Generic BULLETIN

4 HCreating an ASM-enabled Database H


Generic HOWTO

5 HNew Feature on ASM (Automatic Storage Manager).H


Generic BULLETIN

6 HSteps To Migrate Database From Non-ASM to ASM And Vice-Versa H

252219.1 04-JUN-2004 Oracle Server - Enterprise Edition

Generic BULLETIN

7 HHow To Move Archive Files from ASMH

293234.1 07-DEC-2004 Oracle Server - Enterprise Edition

Generic HOWTO

8 HManage ASM instance-creating diskgroup,adding/dropping/resizing disks.H


Generic HOWTO

9 HHow To Delete Archive Log Files Out Of +Asm? H

300472.1 04-MAR-2005 Oracle Server - Enterprise Edition

Generic HOWTO

10 HASM Technical Best Practices H

265633.1 16-JUN-2004 Oracle Server - Enterprise Edition

Generic WHITE PAPER

Hhttp://metalink.oracle.com/metalink/plsql/docs/ASM.pdf H

For full article, download HAutomatic Storage ManagementH (154K/pdf) 11 HOracle ASM and Multi- 294869. 03-JAN- Oracle Server - AIX-Based WHITE


Pathing Technologies H 1 2005 Enterprise Edition Systems PAPER


18.10 NOTE #2064876.102 : HOW TO SETUP HIGH AVAILABILITY GROUP SERVICES (HAGS) ON IBM AIX/RS6000.

Article-ID: <Note:2064876.102> Circulation: PUBLISHED (EXTERNAL) Folder: server.OPS.Parallelserver Topic: - - - IBM RS6000 and SP Title: How to setup High Availability Group Services (HAGS) on IBM AIX/RS6000 Document-Type: BULLETIN Impact: LOW Skill-Level: CASUAL Server-Version: 08.01.XX.XX.XX Updated-Date: 04-FEB-2002 08:21:25 References: Shared-Refs: Authors: BLEVE.US Attachments: NONE Content-Type: TEXT/PLAIN Keywords: LMON; OPS; PARALLEL; SERVER; SUBCOMP-OPS; Products: 236/RDBMS (08.01.XX.XX.XX); Platforms: 319 (4.3); Purpose ======= This article gives quick reference instructions on how to configure High Availability Group Services (HAGS) on IBM AIX RS6000 for Oracle 8.1.X. Scope and Application ===================== These instructions are helpful to any customer using Oracle on IBM AIX/RS6000 on which HACMP is installed. How to Configure High Availability Group Services (HAGS) ======================================================== In order to configure High Availability Group Services (HAGS), you need to be connected as root. Do the following on all nodes that form the cluster: 1. Create the "hagsuser" group and place "oracle" into the "hagsuser" group: Verify the group does not exists: # grep hagsuser /etc/group If this returns nothing do the following: # smitty groups Select "Add a Group" and fill in the following: Group Name ----> hagsuser USER list ----> oracle You can take the defaults for the other settings. Also note that after the group is created you will have to log out and log back in as "oracle" to be sure "oracle" is part of the "hagsuser" group.


2. Change the permissions on the "cldomain" executable: # chmod a+x /usr/sbin/cluster/utilities/cldomain 3. Change the group to "hagsuser" for the "svcsdsocket.rac_cluster" socket: # chgrp hagsuser /var/ha/soc/grpsvcsdsocket.rac_cluster 4. Change the group permissions for the "grpsvcsdsocket.rac_cluster" socket: # chmod g+w /var/ha/soc/grpsvcsdsocket.rac_cluster The HAGS socket needs to be writeable by "oracle" and the "cldomain" executable needs to be executable by "oracle". By configuring the group and permissions for the "grpsvcsdsocket.rac_cluster" file the instance will be able to communicate with HAGS and the instance will mount. References ========== Oracle Installation Guide for AIX RS6000, release 8.1.5. Search Words ============ OPS HAGS RS6000 You should have something like the following : ls –l /var/ha/soc total 4 srw-rw-rw- 1 root haemrm 0 Jun 19 16:58 em.clsrv.rac92_cluster srw-rw---- 1 root haemrm 0 Jun 19 16:58 em.rmsrv.rac92_cluster drwxrwxrwx 2 root system 512 Jun 19 17:51 grpsvcs.clients.rac92_cluster srw-rw-rw- 1 root hagsuser 0 Jun 19 16:57 grpsvcsdsocket.rac92_cluster drwxrwx--- 2 root haemrm 512 Jun 19 16:57 haem drwxr-x--- 2 root system 512 Apr 02 23:30 hats drwxr-xr-x 2 root system 512 Jun 19 16:57 topsvcs


18.11 NOTE # 115792.1 , EXPLAINING HOW TO SETUP HACMP CLUSTER INTERCONNECT ADAPTER.

Article-ID: <Note:115792.1> Circulation: PENDING_DELETE (EXTERNAL) Folder: server.OPS.Parallelserver Topic: - - - IBM RS6000 and SP Title: AIX: ORA-600 [KCCSBCK_FIRST] starting up second OPS Instance Document-Type: BULLETIN Impact: LOW Skill-Level: NOVICE Server-Version: 08.01.06.0X to 08.01.06.0X Updated-Date: 05-FEB-2002 13:10:49 References: Shared-Refs: Authors: RKIRCHHE.DE Attachments: NONE Content-Type: TEXT/PLAIN Keywords: HACMP; OPS; Errors: ORA-600; [KCCSBCK_FIRST]; Products: 5/RDBMS; Platforms: 319; ************************************************************* This article is being delivered in Draft form and may contain errors. Please use the MetaLink "Feedback" button to advise Oracle of any issues related to this article. ************************************************************* PURPOSE ------- This article helps to resolve problems with Oracle Parallel Server startup related to HACMP configuration SCOPE & APPLICATION ------------------- How to setup HACMP cluster interconnect adapter ----------------------------- Oracle Parallel Server software is successfully installed. The first OPS instance starts without errors. Trying to start a second OPS instance on another cluster node fails with ORA-600 [KCCSBCK_FIRST]. $ORACLE_HOME/bin/lsnodes will list all cluster nodes. /usr/sbin/cluster/diag/clverify doesn't show any errors. Check HACMP interconnect network adapter configuration with /usr/sbin/cluster/utilities/cllsif cllsif on a working configuration should look like this: Adapter Type Network NetType Attribute Node IP Address InterfaceName interconnect_node1 service rac92_network ether private node1 10.10.11.141 en3 interconnect_node2 service rac92_network ether private node2 10.10.11.142 en5 RELATED DOCUMENTS ----------------- <List related manuals, articles and other documents.>


19 APPENDIX D : HACMP CLUSTER VERIFICATION OUTPUT All the following outputs come from an HACMP configuration for raw devices implementation which was including a parallel instance database (HACMP concurrent mode) and a single instance database (HACMP rotating mode). (Information about the concurrent mode configuration is in bold.)

Cluster description : output from a ‘cllscf’ command. Node node1 Interfaces to network net_ether_01 Communication Interface: Name node1_gpfs, Attribute private, IP address 10.10.12.81 Node node2 Interfaces to network net_ether_01 Communication Interface: Name node2_gpfs, Attribute private, IP address 10.10.12.82 Cluster Description of Cluster ora9irac There were 1 networks defined: net_ether_01 There are 3 nodes in this cluster NODE node1: This node has 1 service IP label(s): Service IP Label node1_gpfs: IP address: 10.10.12.81 Hardware Address: Network: net_ether_01 Attribute: private Aliased Address?: Not Supported Service IP Label node1_gpfs has no communication interfaces. Service IP Label node1_gpfs has no communication interfaces for recovery. NODE node8: This node has 2 service IP label(s): Service IP Label node8_gpfs: IP address: 10.10.12.88 Hardware Address: Network: net_ether_02 Attribute: private Aliased Address?: Not Supported Service IP Label node8_gpfs has no communication interfaces. Service IP Label node8_gpfs has no communication interfaces for recovery. Service IP Label node8: IP address: 10.2.12.88 Hardware Address: Network: net_ether_03 Attribute: private Aliased Address?: Not Supported Service IP Label node8 has no communication interfaces.


Service IP Label node8 has no communication interfaces for recovery. Breakdown of network connections: Connections to network net_ether_02 Node node5 is connected to network net_ether_02 by these interfaces: node5_gpfs Node node8 is connected to network net_ether_02 by these interfaces: node8_gpfs Connections to network net_ether_03 Node node5 is connected to network net_ether_03 by these interfaces: node5 Node node8 is connected to network net_ether_03 by these interfaces: node8

Cluster network interfaces : output from a ‘cllsif’ command. Adapter Type Network NetType Attribute Node IP Address InterfaceName interconnect_node1 service rac92_network ether private node1 10.10.12.141 en3 interconnect_node2 service rac92_network ether private node2 10.10.12.142 en5 Adapter Type Network Net Type Attribute Node IP Address Hardware Address Interface Name Global Name Netmask Alias for HB node5_gpfs service net_ether_02 ether private node5 10.10.12.85 en0 255.255.255.0 node5 service net_ether_03 ether private node5 10.2.12.85 en1 255.255.255.0 node8_gpfs service net_ether_02 ether private node8 10.10.12.88 en1 255.255.255.0 node8 service net_ether_03 ether private node8 10.2.12.88 en0 255.255.255.0


Cluster resource group : output from a ‘clshowres’ command. Resource Group Name racdisk Node Relationship concurrent Participating Node Name(s) node1 node2 Dynamic Node Priority Service IP Label Filesystems Filesystems Consistency Check fsck Filesystems Recovery Method sequential Filesystems/Directories to be exported Filesystems to be NFS mounted Network For NFS Mount Volume Groups Concurrent Volume Groups oradatavg Disks Connections Services Fast Connect Services Shared Tape Resources Application Servers Highly Available Communication Links Miscellaneous Data Automatically Import Volume Groups false Inactive Takeover false Cascading Without Fallback false 9333 Disk Fencing false SSA Disk Fencing false Filesystems mounted before IP configured false Run Time Parameters: Node Name node1 Debug Level high Host uses NIS or Name Server false Format for hacmp.out Standard Node Name node2 Debug Level high Host uses NIS or Name Server false Format for hacmp.out Standard


20 APPENDIX F : FILESETS TO BE INSTALLED ON THE MACHINES OF THE CLUSTER.

This appendix provides the list of filesets which must be installed for each kind of implementation.

20.1 RSCT 2.3.4 (PROVIDED WITH AIX 5.2) FOR ALL IMPLEMENTATION rsct.basic.rte 2.3.4.0 C F RSCT Basic Function rsct.compat.basic.rte 2.3.4.0 C F RSCT Event Management Basic rsct.compat.clients.rte 2.3.4.0 C F RSCT Event Management Client rsct.core.auditrm 2.3.4.0 C F RSCT Audit Log Resource rsct.core.errm 2.3.4.0 C F RSCT Event Response Resource rsct.core.fsrm 2.3.4.0 C F RSCT File System Resource rsct.core.gui 2.3.4.0 C F RSCT Graphical User Interface rsct.core.hostrm 2.3.4.0 C F RSCT Host Resource Manager rsct.core.rmc 2.3.4.0 C F RSCT Resource Monitoring and rsct.core.sec 2.3.4.0 C F RSCT Security rsct.core.sensorrm 2.3.4.0 C F RSCT Sensor Resource Manager rsct.core.sr 2.3.4.0 C F RSCT Registry rsct.core.utils 2.3.4.0 C F RSCT Utilities rsct.msg.EN_US.core.auditrm rsct.msg.EN_US.core.errm 2.3.0.0 C F RSCT Event Response RM Msgs - rsct.msg.EN_US.core.fsrm 2.3.0.0 C F RSCT File System RM Msgs - rsct.msg.EN_US.core.gui 2.3.0.0 C F RSCT GUI Msgs - U.S. English rsct.msg.EN_US.core.hostrm rsct.msg.EN_US.core.rmc 2.3.0.0 C F RSCT RMC Msgs - U.S. English rsct.msg.EN_US.core.sec 2.3.0.0 C F RSCT Security Msgs - U.S. rsct.msg.EN_US.core.sensorrm rsct.msg.EN_US.core.sr 2.3.0.0 C F RSCT Registry Msgs - U.S. rsct.msg.EN_US.core.utils 2.3.0.0 C F RSCT Utilities Msgs - U.S. rsct.msg.en_US.core.auditrm rsct.msg.en_US.core.errm 2.3.0.0 C F RSCT Event Response RM Msgs - rsct.msg.en_US.core.fsrm 2.3.0.0 C F RSCT File System RM Msgs - rsct.msg.en_US.core.gui 2.3.0.0 C F RSCT GUI Msgs - U.S. English rsct.msg.en_US.core.gui.com rsct.msg.en_US.core.hostrm rsct.msg.en_US.core.rmc 2.3.0.0 C F RSCT RMC Msgs - U.S. English rsct.msg.en_US.core.rmc.com rsct.msg.en_US.core.sec 2.3.0.0 C F RSCT Security Msgs - U.S.


rsct.msg.en_US.core.sensorrm rsct.msg.en_US.core.sr 2.3.0.0 C F RSCT Registry Msgs - U.S. rsct.msg.en_US.core.utils 2.3.0.0 C F RSCT Utilities Msgs - U.S.

20.2 WHEN IMPLEMENTING HACMP

20.2.1 HACMP 5.1 filesets

Result of the command : lslpp -L | grep cluster

cluster.adt.es.client.demos cluster.adt.es.client.include cluster.adt.es.client.samples.clinfo cluster.adt.es.client.samples.clstat cluster.adt.es.client.samples.demos cluster.adt.es.client.samples.libcl cluster.adt.es.java.demo.monitor cluster.adt.es.server.demos cluster.adt.es.server.samples.demos cluster.adt.es.server.samples.images cluster.es.client.lib 5.1.0.5 C F ES Client Libraries cluster.es.client.rte 5.1.0.3 C F ES Client Runtime cluster.es.client.utils 5.1.0.4 C F ES Client Utilities cluster.es.clvm.rte 5.1.0.0 C F ES for AIX Concurrent Access cluster.es.cspoc.cmds 5.1.0.5 C F ES CSPOC Commands cluster.es.cspoc.dsh 5.1.0.0 C F ES CSPOC dsh cluster.es.cspoc.rte 5.1.0.5 C F ES CSPOC Runtime Commands cluster.es.server.diag 5.1.0.4 C F ES Server Diags cluster.es.server.events 5.1.0.5 C F ES Server Events cluster.es.server.rte 5.1.0.5 C F ES Base Server Runtime cluster.es.server.utils 5.1.0.5 C F ES Server Utilities cluster.license 5.1.0.0 C F HACMP Electronic License

20.2.2 RSCT 2.3.4 filesets for HACMP Implementation

Add these three filesets to the ones mentinoned in 20.1 RSCT 2.3.4 (provided with AIX 5.2) for all implementation on page 171 rsct.basic.hacmp 2.3.4.0 C F RSCT Basic Function (HACMP/ES rsct.compat.basic.hacmp 2.3.4.0 C F RSCT Event Management Basic rsct.compat.clients.hacmp 2.3.4.0 C F RSCT Event Management Client

20.3 FOR GPFS IMPLEMENTATION mmfs.base.cmds 3.5.0.6 C F GPFS File Manager Commands mmfs.base.rte 3.5.0.9 C F GPFS File Manager mmfs.gpfs.rte 2.1.0.9 C F GPFS File Manager mmfs.gpfsdocs.data 3.5.0.0 C F GPFS Server Manpages and mmfs.msg.en_US 3.5.0.0 C F GPFS Server Messages - U.S.

Documents

CookBook V1.10 Oracle 10g RAC AIX V5L