3 Install Clusterware (1)

7/27/2019 3 Install Clusterware (1)

1/49

Author A.Kishorehttp:/www.appsdba.info

Build Your Own Oracle RAC 11gRelease 2 Cluster on Red Hat 4 and iSCSI

Here is my setup which I have used for Installing Oracle RAC 11gR2

OpenFiler

To access linux1 andlinux22 GB

500 GB HDDLinux1

4 GB250 GB

Oracle AppsR12.1.1

Node 1

Linux2

4 GB250 GB

Node 2
http://appsdba.info/http://appsdba.info/http://appsdba.info/http://appsdba.info/http://appsdba.info/http://appsdba.info/


2/49


High Level Steps

> Install openfiler on linux1 using VMWARE

> Install Grid Infrastructure on linux1 and linux2> Install Oracle 11gR2 software on linux1 and linux2

Download the software from the below site

http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linuxsoft-085393.html
http://appsdba.info/http://appsdba.info/http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linuxsoft-085393.htmlhttp://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linuxsoft-085393.htmlhttp://appsdba.info/http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linuxsoft-085393.htmlhttp://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linuxsoft-085393.html


3/49


1. Install Oracle 11gR2 Grid Infrastructure (Both the nodes should be ready)

-- clean the old installation (on both nodes)

rm -rf /etc/oraclerm -rf /etc/oraInst.locrm -rf /etc/oratabcd /d01/oracle/apprm -rf oraclerm -rf oraInventoryrm -rf /d01/oracle/app/*.*rm -rf /d01/oracle/app/11.2.0/grid/*.*

-- On one of the node Using only one diskgroup

/etc/init.d/oracleasm deletedisk VOL1 /dev/sdb1/etc/init.d/oracleasm createdisk VOL1 /dev/sdb1

/etc/init.d/oracleasm listdisks

Note I am using only one diskgroups

-- If there are 3 diskgroups then use the below commands

/etc/init.d/oracleasm deletedisk VOL1 /dev/sdc1/etc/init.d/oracleasm deletedisk VOL2 /dev/sdd1/etc/init.d/oracleasm deletedisk VOL3 /dev/sde1/etc/init.d/oracleasm createdisk VOL1 /dev/sdc1/etc/init.d/oracleasm createdisk VOL2 /dev/sdd1/etc/init.d/oracleasm createdisk VOL3 /dev/sde1/etc/init.d/oracleasm listdisks

Run cluvfy to verify all the prereqs are available

su - root

cd /tmprm -rf bootstrap/

su - oraclecd $SOFTWARE_LOCATION/11gR2/grid

sh runcluvfy.sh stage -pre crsinst -n linux1,linux2 verbose

See Appendix A for output

Only one package has failed and that is fine, red hat comes with older version of glibc,but the cluvfy script is looking for the latest script

Check: Package existence for "glibc-2.3.4-2.41"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------
http://appsdba.info/http://appsdba.info/http://appsdba.info/http://appsdba.info/


4/49


linux1 glibc-2.3.4-2.39 glibc-2.3.4-2.41 failedResult: Package existence check failed for "glibc-2.3.4-2.41"

cd $SOFTWARE_LOCATION/11gR2/grid./runInstaller


5/49

http://appsdba.info/http://appsdba.info/http://appsdba.info/


6/49


Check NetworkCheck NetworkCheck NetworkCheck Network Requirements:Requirements:Requirements:Requirements: In this release there are two new network related components

SCAN and GNS.

Single Client Access Name (SCAN) for the Cluster:Single Client Access Name (SCAN) for the Cluster:Single Client Access Name (SCAN) for the Cluster:Single Client Access Name (SCAN) for the Cluster:

During Typical installation, you are prompted to confirm the default Single Client Access

Name (SCAN), which is used to connect to databases within the cluster irrespective of which

nodes they are running on. By default, the name used as the SCAN is also the name of the

cluster. The default value for the SCAN is based on the local node name. If you change the

SCAN from the default, then the name that you use must be globally unique throughout your

enterprise.

In a Typical installation, the SCAN is also the name of the cluster. The SCAN and cluster

name must be at least one character long and no more than 15 characters in length, must be


7/49


alphanumeric, and may contain hyphens (-). If you require a SCAN that is longer than 15

characters, then select an Advanced installation.

Configure the following addresses:Configure the following addresses:Configure the following addresses:Configure the following addresses:

A public IP address for each node

A virtual IP address for each node

A single client access name (SCAN) configured on the domain name server (DNS)domain name server (DNS)domain name server (DNS)domain name server (DNS) or GridGridGridGrid

Naming Service (GNS)Naming Service (GNS)Naming Service (GNS)Naming Service (GNS) for Round Robin resolution to three addresses (recommended) or at

least one address (check documentation)

Some more information on Single Client Access Name

Single Client Access Name (SCAN) for the Cluster

If you have ever been tasked with extending an Oracle RAC cluster by adding a newnode (or shrinking a RAC cluster by removing a node), then you know the pain of goingthrough a list of all clients and updating their SQL*Net or JDBC configuration to reflectthe new or deleted node! To address this problem, Oracle 11gRelease 2 introduced a newfeature known as Single Client Access Name or SCAN for short. SCAN is a new featurethat provides a single host name for clients to access an Oracle Database running in acluster. Clients using SCAN do not need to change their TNS configuration if you add or

remove nodes in the cluster. The SCAN resource and its associated IP address(s) providea stable name for clients to use for connections, independent of the nodes that make upthe cluster. You will be asked to provide the host name and up to three IP addresses to beused for the SCAN resource during the interview phase of the Oracle grid infrastructureinstallation. For high availability and scalability, Oracle recommends that you configurethe SCAN name so that it resolves to three IP addresses. At a minimum, the SCAN mustresolve to at least one address.

The SCAN virtual IP name is similar to the names used for a node's virtual IP addresses,

such as racnode1-vip. However, unlike a virtual IP, the SCAN is associated with theentire cluster, rather than an individual node, and can be associated with multiple IP

addresses, not just one address. Note that SCAN addresses, virtual IP addresses, andpublic IP addresses must all be on the same subnet.

The SCAN should be configured so that it is resolvable either by using Grid NamingService (GNS) within the cluster, or by using Domain Name Service (DNS) resolution.

In this article, I will configure SCAN to resolve to only one, manually configured staticIP address using the DNS method (but not actually defining it in DNS)
http://appsdba.info/http://appsdba.info/http://appsdba.info/http://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi-3.html#Configure%20Grid%20Plug%20and%20Play%20Information%20-%20SCANhttp://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi-3.html#Configure%20Grid%20Plug%20and%20Play%20Information%20-%20SCANhttp://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi-3.html#Configuring%20SCAN%20without%20DNShttp://appsdba.info/http://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi-3.html#Configuring%20SCAN%20without%20DNShttp://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi-3.html#Configure%20Grid%20Plug%20and%20Play%20Information%20-%20SCANhttp://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi-3.html#Configure%20Grid%20Plug%20and%20Play%20Information%20-%20SCAN


8/49


192.168.1.187 racnode-cluster-scan make the entry in /etc/hosts


9/49



10/49



11/49


Enter Oracle password and click Setup


12/49



13/49



14/49



15/49



16/49



17/49



18/49



19/49



20/49


The above error can safely be ignored


21/49


22/49


[root@linux1 ~]# /d01/oracle/app/oraInventory/orainstRoot.shChanging permissions of /d01/oracle/app/oraInventory.Adding read,write permissions for group.Removing read,write,execute permissions for world.

Changing groupname of /d01/oracle/app/oraInventory to oinstall.The execution of the script is complete.

[root@linux2 ~]# /d01/oracle/app/oraInventory/orainstRoot.shChanging permissions of /d01/oracle/app/oraInventory.Adding read,write permissions for group.Removing read,write,execute permissions for world.

Changing groupname of /d01/oracle/app/oraInventory to oinstall.The execution of the script is complete.


23/49


Run root.sh on linux1 and linux2

cd /d01/oracle/app/11.2.0/grid/[root@linux1 grid]# sh root.sh

See Appendix B and C for outputs

Click OK


24/49


INFO: Checking Single Client Access Name (SCAN)...INFO: Checking name resolution setup for "racnode-cluster-scan"...INFO: ERROR:INFO: PRVF-4657 : Name resolution setup check for "racnode-cluster-scan" (IPaddress: 216.24.138.153) failedINFO: ERROR:INFO: PRVF-4657 : Name resolution setup check for "racnode-cluster-scan" (IPaddress: 192.168.1.187) failedINFO: ERROR:INFO: PRVF-4664 : Found inconsistent name resolution entries for SCAN name"racnode-cluster-scan"INFO: Verification of SCAN VIP and Listener setup failed

Provided this is the only error reported by the CVU, it would be safe to ignore this check

and continue by clicking the [Next] button in OUI and move forward with the Oraclegrid infrastructure installation. This is documented in Doc ID: 887471.1on the MyOracle Support web site.
http://appsdba.info/http://appsdba.info/https://support.oracle.com/CSP/ui/flash.html#tab=KBHome%28page=KBHome&id=%28%29%29%2C%28page=KBNavigator&id=%28viewingMode=1143&bmDocID=887471.1&from=BOOKMARK&bmDocType=PROBLEM&bmDocDsrc=DOCUMENT&bmDocTitle=PRVF-4664%20:%20Found%20inconsistent%20name%20resolution%20entries%20for%20SCAN%20name%29%29https://support.oracle.com/CSP/ui/flash.html#tab=KBHome%28page=KBHome&id=%28%29%29%2C%28page=KBNavigator&id=%28viewingMode=1143&bmDocID=887471.1&from=BOOKMARK&bmDocType=PROBLEM&bmDocDsrc=DOCUMENT&bmDocTitle=PRVF-4664%20:%20Found%20inconsistent%20name%20resolution%20entries%20for%20SCAN%20name%29%29http://www.myoraclesupport.com/http://www.myoraclesupport.com/http://appsdba.info/http://www.myoraclesupport.com/http://www.myoraclesupport.com/https://support.oracle.com/CSP/ui/flash.html#tab=KBHome%28page=KBHome&id=%28%29%29%2C%28page=KBNavigator&id=%28viewingMode=1143&bmDocID=887471.1&from=BOOKMARK&bmDocType=PROBLEM&bmDocDsrc=DOCUMENT&bmDocTitle=PRVF-4664%20:%20Found%20inconsistent%20name%20resolution%20entries%20for%20SCAN%20name%29%29


25/49


If on the other hand you want the CVU to complete successfully while still only defining

the SCAN in the hosts file, simply modify the nslookup utility as root on both Oracle

RAC nodes as follows.

First, rename the original nslookup binary to nslookup.original on both Oracle RACnodes:

[root@racnode1 ~]#mv /usr/bin/nslookup /usr/bin/nslookup.original

Next, create a new shell script named /usr/bin/nslookup as shown below while

replacing 24.154.1.34 with your primary DNS, racnode-cluster-scan with your

SCAN host name, and 192.168.1.187 with your SCAN IP address:

#!/bin/bash

HOSTNAME=${1}

if [[ $HOSTNAME = "racnode-cluster-scan" ]]; thenecho "Server: 24.154.1.34"echo "Address: 24.154.1.34#53"echo "Non-authoritative answer:"echo "Name: racnode-cluster-scan"echo "Address: 192.168.1.187"

else/usr/bin/nslookup.original $HOSTNAME

Fi

Finally, change the new nslookup shell script to executable:

[root@racnode1 ~]# chmod 755 /usr/bin/nslookup

Remember to perform these actions on both Oracle RAC nodes.

The new nslookup shell script simply echo's back your SCAN IP address whenever the

CVU calls nslookup with your SCAN host name; otherwise, it calls the original

nslookup binary.

The CVU will now pass during the Oracle grid infrastructure installation when it attempts

to verify your SCAN:

[grid@racnode1 ~]$ cluvfy comp scan -verbose

Verifying scan

Checking Single Client Access Name (SCAN)...SCAN VIP name Node Running? ListenerName Port

Running?


26/49


---------------- ------------ ------------ ------------ ------------ ------------

racnode-cluster-scan racnode1 true LISTENER 1521true

Checking name resolution setup for "racnode-cluster-scan"...SCAN Name IP Address Status Comment------------ ------------------------ ------------------------ ----------racnode-cluster-scan 192.168.1.187 passed

Verification of SCAN VIP and Listener setup passed

Verification of scan was successful.

===============================================================================

[grid@racnode2 ~]$ cluvfy comp scan -verbose

Verifying scan

Checking Single Client Access Name (SCAN)...SCAN VIP name Node Running? ListenerName Port

Running?---------------- ------------ ------------ ------------ ------------ ----

--------racnode-cluster-scan racnode1 true LISTENER 1521

true

Checking name resolution setup for "racnode-cluster-scan"...

SCAN Name IP Address Status Comment------------ ------------------------ ------------------------ ----------racnode-cluster-scan 192.168.1.187 passed

Verification of SCAN VIP and Listener setup passed

Verification of scan was successful.

Now retry


27/49



28/49



29/49


cat grid.envexport ORACLE_HOME=/d01/oracle/app/11.2.0/grid

export PATH=$PATH:$ORACLE_HOME/bin. ./grid.envcrs_stat t

Post-Install Actions

By default, the Global Services Daemon (GSD) is not started on the cluster. To start GSD, changedirectory to the and issue the following commands:

srvctl enable nodeapps gsrvctl start nodeapps


30/49


ora.oc4j is offine, that is fine

Successful Oracle Clusterware operation can also be verified using the following command:crsctl check crs

olsnodes -n


31/49


srvctl status asm -a

ocrcheck

crsctl query css votedisk


32/49


./crsctl check cluster -all


33/49


References:

Thanks to Jeff Hunter, without his help I couldnt have done this assignment

http://www.rachelp.nl/index_kb.php?menu=articles&actie=show&id=61

11gR2 Grid: root.sh fails to start the clusterware on the second node (Doc ID 981357.1)

1. short-term: disable the firewall on all nodes, on Linux this can be done by running thefollowing command(s) as the root user on each node of the cluster:

service iptables stopservice ip6tables stop

To permanently disable the firewall, use:

chkconfig iptables offchkconfig ip6tables off

2. long-term: exclude all traffic on the private network from the firewall configuration.http://coskan.wordpress.com/

http://coskan.wordpress.com/2009/12/07/root-sh-failed-after-asm-disk-creation-for-11gr2-grid-infrastructure/

http://oracle-base.com/forums/viewtopic.php?f=1&t=11307&start=0

1. How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation [ID942166.1]

Issues and solutions

cd /home/oracle/app/11.2.0/grid/log/linux2

cat alertlinux2.log

[ctssd(27935)]CRS-2409:The clock on host linux2 is not synchronous with the meancluster time. No action has been taken as the Cluster Time Synchronization Service isrunning in observer mode.
http://appsdba.info/http://appsdba.info/http://www.rachelp.nl/index_kb.php?menu=articles&actie=show&id=61http://coskan.wordpress.com/2009/12/07/root-sh-failed-after-asm-disk-creation-for-11gr2-grid-infrastructure/http://coskan.wordpress.com/2009/12/07/root-sh-failed-after-asm-disk-creation-for-11gr2-grid-infrastructure/http://appsdba.info/http://coskan.wordpress.com/2009/12/07/root-sh-failed-after-asm-disk-creation-for-11gr2-grid-infrastructure/http://coskan.wordpress.com/2009/12/07/root-sh-failed-after-asm-disk-creation-for-11gr2-grid-infrastructure/http://www.rachelp.nl/index_kb.php?menu=articles&actie=show&id=61


34/49


2010-07-24 05:26:20.471[ctssd(27935)]CRS-2412:The Cluster Time Synchronization Service detects that the local

time is significantly different from the mean cluster time. Details in/home/oracle/app/11.2.0/grid/log/linux2/ctssd/octssd.log.2010-07-24 05:26:20.471[ctssd(27935)]CRS-2409:The clock on host linux2 is not synchronous with the meancluster time. No action has been taken as the Cluster Time Synchronization Service isrunning in observer mode.http://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi.htmlhttp://arjudba.blogspot.com/2010/03/in-11gr2-grid-rootsh-fails-with-crs.htmlHow to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation [ID 942166.1]

-- If root.sh has failed, run the below commands to clean up

-- first nodecd /home/oracle/app/11.2.0/grid/crs/install./rootcrs.pl -verbose -deconfig -force-- second nodecd /home/oracle/app/11.2.0/grid/crs/install./rootcrs.pl -verbose -deconfig -force -lastnode

To add vip manually

[root@linux2 bin]# ./srvctl add vip -n linux2 -A 192.168.0.201/255.255.255.0/eth0 -k 1[root@linux2 bin]# ./srvctl config vip -n linux2

http://forums.oracle.com/forums/thread.jspa?messageID=4039293

rpm -q binutils-2.15.92.0.2 \compat-libstdc++-33.2.3 \elfutils-libelf-0.97 \

elfutils-libelf-devel-0.97 \gcc-3.4.6 \gcc-c++-3.4.6 \glibc-2.3.4-2.41 \glibc-common-2.3.4 \glibc-devel-2.3.4 \glibc-headers-2.3.4 \libaio-devel-0.3.105 \
http://appsdba.info/http://appsdba.info/http://forums.oracle.com/forums/thread.jspa?messageID=4039293http://appsdba.info/http://forums.oracle.com/forums/thread.jspa?messageID=4039293


35/49


libaio-0.3.105 \libgcc-3.4.6 \

libstdc++-3.4.6 \libstdc++-devel-3.4.6 \make-3.80 \pdksh-5.2.14 \sysstat-5.0.5 \unixODBC-2.2.11 \unixODBC-devel-2.2.11

http://forums.oracle.com/forums/thread.jspa?messageID=4345930

http://www.oracle.com/technology/pub/articles/wartak-rac-vm_3.htmlhttp://wiki.oracle.com/page/11gR2+RAC+on+a+Mac+-+Part+6 nice one

Appendix A

sh runcluvfy.sh stage -pre crsinst -n linux1,linux2 -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "linux1"Destination Node Reachable?

------------------------------------ ------------------------linux1 yeslinux2 yes

Result: Node reachability check passed from node "linux1"

Checking user equivalence...

Check: User equivalence for user "oracle"Node Name Comment------------------------------------ ------------------------linux2 failedlinux1 passed

Result: PRVF-4007 : User equivalence check failed for user "oracle"

WARNING:User equivalence is not set for nodes:linux2

Verification will proceed with nodes:linux1

Checking node connectivity...

Checking hosts config file...Node Name Status Comment------------ ------------------------ ------------------------
http://appsdba.info/http://appsdba.info/http://forums.oracle.com/forums/thread.jspa?messageID=4345930http://www.oracle.com/technology/pub/articles/wartak-rac-vm_3.htmlhttp://wiki.oracle.com/page/11gR2+RAC+on+a+Mac+-+Part+6http://appsdba.info/http://wiki.oracle.com/page/11gR2+RAC+on+a+Mac+-+Part+6http://www.oracle.com/technology/pub/articles/wartak-rac-vm_3.htmlhttp://forums.oracle.com/forums/thread.jspa?messageID=4345930


36/49


linux1 passed

Verification of the hosts config file successful

Interface information for node "linux1"Name IP Address Subnet Gateway Def. Gateway HWAddress MTU------ --------------- --------------- --------------- --------------- ----------------- ------eth0 192.168.1.3 192.168.1.0 0.0.0.0 192.168.1.100:08:A1:95:2F:61 1500eth1 192.168.0.101 192.168.0.0 0.0.0.0 192.168.1.100:13:72:22:4C:78 1500

Check: Node connectivity of subnet "192.168.1.0"Result: Node connectivity passed for subnet "192.168.1.0" with node(s) linux1

Check: TCP connectivity of subnet "192.168.1.0"Result: TCP connectivity check passed for subnet "192.168.1.0"

Check: Node connectivity of subnet "192.168.0.0"Result: Node connectivity passed for subnet "192.168.0.0" with node(s) linux1

Check: TCP connectivity of subnet "192.168.0.0"Result: TCP connectivity check passed for subnet "192.168.0.0"

Interfaces found on subnet "192.168.1.0" that are likely candidates for VIPare:linux1 eth0:192.168.1.3

Interfaces found on subnet "192.168.0.0" that are likely candidates for aprivate interconnect are:linux1 eth1:192.168.0.101

Result: Node connectivity check passed

Check: Total memoryNode Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 3.71GB (3895072.0KB) 1.5GB (1572864.0KB) passed

Result: Total memory check passed

Check: Available memoryNode Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 3.32GB (3479748.0KB) 50MB (51200.0KB) passed

Result: Available memory check passed

Check: Swap spaceNode Name Available Required Comment------------ ------------------------ ------------------------ ----------


37/49


linux1 5.86GB (6144820.0KB) 3.71GB (3895072.0KB) passedResult: Swap space check passed

Check: Free disk space for "linux1:/tmp"Path Node Name Mount point Available Required

Comment---------------- ------------ ------------ ------------ ------------ ---

---------/tmp linux1 / 6.82GB 1GB

passedResult: Free disk space check passed for "linux1:/tmp"

Check: User existence for "oracle"Node Name Status Comment------------ ------------------------ ------------------------linux1 exists passed

Result: User existence check passed for "oracle"

Check: Group existence for "oinstall"Node Name Status Comment------------ ------------------------ ------------------------linux1 exists passed

Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"Node Name Status Comment------------ ------------------------ ------------------------linux1 exists passed

Result: Group existence check passed for "dba"

Check: Membership of user "oracle" in group "oinstall" [as Primary]Node Name User Exists Group Exists User in Group Primary

Comment---------------- ------------ ------------ ------------ ------------ ---

---------linux1 yes yes yes yes

passedResult: Membership check for user "oracle" in group "oinstall" [as Primary]passed

Check: Membership of user "oracle" in group "dba"Node Name User Exists Group Exists User in Group Comment---------------- ------------ ------------ ------------ ----------------linux1 yes yes yes passed

Result: Membership check for user "oracle" in group "dba" passed

Check: Run levelNode Name run level Required Comment------------ ------------------------ ------------------------ ----------linux1 5 3,5 passed

Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"Node Name Type Available Required Comment---------------- ------------ ------------ ------------ ----------------linux1 hard 65536 65536 passed

Result: Hard limits check passed for "maximum open file descriptors"


38/49


Check: Soft limits for "maximum open file descriptors"Node Name Type Available Required Comment---------------- ------------ ------------ ------------ ----------------

linux1 soft 65536 1024 passedResult: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"Node Name Type Available Required Comment---------------- ------------ ------------ ------------ ----------------linux1 hard 61416 16384 passed

Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"Node Name Type Available Required Comment---------------- ------------ ------------ ------------ ----------------linux1 soft 61416 2047 passed

Result: Soft limits check passed for "maximum user processes"

Check: System architectureNode Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 i686 x86 passed

Result: System architecture check passed

Check: Kernel versionNode Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 2.6.9-67.EL 2.6.9 passed

Result: Kernel version check passed

Check: Kernel parameter for "semmsl"Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 256 250 passed

Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 32000 32000 passed

Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 100 100 passed

Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 142 128 passed

Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 4294967295 536870912 passed


39/49


Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"

Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 4096 4096 passed

Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 2097152 2097152 passed

Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 6815744 6815744 passed

Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 between 9000 & 65500 between 9000 & 65500 passed

Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 262144 262144 passed

Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 4194304 4194304 passed

Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 262144 262144 passed

Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 1048576 1048576 passed

Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"Node Name Configured Required Comment------------ ------------------------ ------------------------ ----------linux1 1048576 1048576 passed

Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make-3.80"Node Name Available Required Comment


40/49


------------ ------------------------ ------------------------ ----------linux1 make-3.80-6.EL4 make-3.80 passed

Result: Package existence check passed for "make-3.80"

Check: Package existence for "binutils-2.15.92.0.2"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 binutils-2.15.92.0.2-24 binutils-2.15.92.0.2 passed

Result: Package existence check passed for "binutils-2.15.92.0.2"

Check: Package existence for "gcc-3.4.6"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 gcc-3.4.6-9 gcc-3.4.6 passed

Result: Package existence check passed for "gcc-3.4.6"

Check: Package existence for "gcc-c++-3.4.6"Node Name Available Required Comment

------------ ------------------------ ------------------------ ----------linux1 gcc-c++-3.4.6-9 gcc-c++-3.4.6 passed

Result: Package existence check passed for "gcc-c++-3.4.6"

Check: Package existence for "libaio-0.3.105"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 libaio-0.3.105-2 libaio-0.3.105 passed

Result: Package existence check passed for "libaio-0.3.105"

Check: Package existence for "glibc-2.3.4-2.41"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 glibc-2.3.4-2.39 glibc-2.3.4-2.41 failed

Result: Package existence check failed for "glibc-2.3.4-2.41"

Check: Package existence for "compat-libstdc++-33-3.2.3"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 compat-libstdc++-33-3.2.3-47.3 compat-libstdc++-33-3.2.3

passedResult: Package existence check passed for "compat-libstdc++-33-3.2.3"

Check: Package existence for "elfutils-libelf-0.97"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 elfutils-libelf-0.97.1-5 elfutils-libelf-0.97 passed

Result: Package existence check passed for "elfutils-libelf-0.97"

Check: Package existence for "elfutils-libelf-devel-0.97"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 elfutils-libelf-devel-0.97.1-5 elfutils-libelf-devel-0.97

passedResult: Package existence check passed for "elfutils-libelf-devel-0.97"

Check: Package existence for "glibc-common-2.3.4"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 glibc-common-2.3.4-2.39 glibc-common-2.3.4 passed


41/49


Result: Package existence check passed for "glibc-common-2.3.4"

Check: Package existence for "glibc-devel-2.3.4"

Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 glibc-devel-2.3.4-2.39 glibc-devel-2.3.4 passed

Result: Package existence check passed for "glibc-devel-2.3.4"

Check: Package existence for "glibc-headers-2.3.4"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 glibc-headers-2.3.4-2.39 glibc-headers-2.3.4 passed

Result: Package existence check passed for "glibc-headers-2.3.4"

Check: Package existence for "libaio-devel-0.3.105"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 libaio-devel-0.3.105-2 libaio-devel-0.3.105 passed

Result: Package existence check passed for "libaio-devel-0.3.105"

Check: Package existence for "libgcc-3.4.6"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 libgcc-3.4.6-9 libgcc-3.4.6 passed

Result: Package existence check passed for "libgcc-3.4.6"

Check: Package existence for "libstdc++-3.4.6"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 libstdc++-3.4.6-9 libstdc++-3.4.6 passed

Result: Package existence check passed for "libstdc++-3.4.6"

Check: Package existence for "libstdc++-devel-3.4.6"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 libstdc++-devel-3.4.6-9 libstdc++-devel-3.4.6 passed

Result: Package existence check passed for "libstdc++-devel-3.4.6"

Check: Package existence for "sysstat-5.0.5"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 sysstat-5.0.5-16.rhel4 sysstat-5.0.5 passed

Result: Package existence check passed for "sysstat-5.0.5"

Check: Package existence for "unixODBC-2.2.11"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 unixODBC-2.2.11-1.RHEL4.1 unixODBC-2.2.11 passed

Result: Package existence check passed for "unixODBC-2.2.11"

Check: Package existence for "unixODBC-devel-2.2.11"Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 unixODBC-devel-2.2.11-1.RHEL4.1 unixODBC-devel-2.2.11

passedResult: Package existence check passed for "unixODBC-devel-2.2.11"

Check: Package existence for "pdksh-5.2.14"


42/49


Node Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 pdksh-5.2.14-30.6 pdksh-5.2.14 passed

Result: Package existence check passed for "pdksh-5.2.14"

Checking for multiple users with UID value 0Result: Check for multiple users with UID value 0 passed

Check: Current group IDResult: Current group ID check passedChecking Core file name pattern consistency...Core file name pattern consistency check passed.

Checking to make sure user "oracle" is not in "root" groupNode Name Status Comment------------ ------------------------ ------------------------linux1 does not exist passed

Result: User "oracle" is not part of "root" group. Check passed

Check default user file creation maskNode Name Available Required Comment------------ ------------------------ ------------------------ ----------linux1 0022 0022 passed

Result: Default user file creation mask check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...The NTP configuration file "/etc/ntp.conf" is available on all nodesNTP Configuration file check passed

Checking daemon liveness...

Check: Liveness for "ntpd"Node Name Running?------------------------------------ ------------------------linux1 yes

Result: Liveness check passed for "ntpd"

Checking NTP daemon command line for slewing option "-x"Check: NTP daemon command lineNode Name Slewing Option Set?------------------------------------ ------------------------linux1 yes

Result:NTP daemon slewing option check passed

Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd",for slewing option "-x"

Check: NTP daemon's boot time configurationNode Name Slewing Option Set?------------------------------------ ------------------------linux1 yes

Result:NTP daemon's boot time configuration check for slewing option passed

NTP common Time Server Check started...


43/49


NTP Time Server "202.71.140.36" is common to all nodes on which the NTP daemonis runningNTP Time Server "209.81.9.7" is common to all nodes on which the NTP daemon is

runningNTP Time Server ".INIT." is common to all nodes on which the NTP daemon isrunningNTP Time Server "218.45.21.199" is common to all nodes on which the NTP daemonis runningCheck of common NTP Time Server passed

Clock time offset check from NTP Time Server started...Checking on nodes "[linux1]"...Check: Clock time offset from NTP Time Server

Time Server: 202.71.140.36Time Offset Limit: 1000.0 msecsNode Name Time Offset Status------------ ------------------------ ------------------------

linux1 1.711 passedTime Server "202.71.140.36" has time offsets that are within permissible limitsfor nodes "[linux1]".

Time Server: 209.81.9.7Time Offset Limit: 1000.0 msecsNode Name Time Offset Status------------ ------------------------ ------------------------linux1 11.497 passed

Time Server "209.81.9.7" has time offsets that are within permissible limitsfor nodes "[linux1]".

Time Server: .INIT.Time Offset Limit: 1000.0 msecsNode Name Time Offset Status------------ ------------------------ ------------------------linux1 0.0 passed

Time Server ".INIT." has time offsets that are within permissible limits fornodes "[linux1]".

Time Server: 218.45.21.199Time Offset Limit: 1000.0 msecsNode Name Time Offset Status------------ ------------------------ ------------------------linux1 23.78 passed

Time Server "218.45.21.199" has time offsets that are within permissible limitsfor nodes "[linux1]".Clock time offset check passed

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Pre-check for cluster services setup was unsuccessful on all the nodes._]0;oracle@linux1:/d01/oracle/RAC/11gR2/grid_[oracle@linux1 grid]$Script done on Tue 27 Jul 2010 08:02:17 AM IST

Appendix B


44/49


cd /d01/oracle/app/11.2.0/grid/[root@linux1 grid]# script akr.txt

Script started, file is akr.txt[root@linux1 grid]# sh root.shRunning Oracle 11g root.sh script...

The following environment variables are set as:ORACLE_OWNER= oracleORACLE_HOME= /d01/oracle/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)[n]: y

Copying dbhome to /usr/local/bin ...The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)[n]: y

Copying oraenv to /usr/local/bin ...The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)[n]: y

Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is createdFinished running generic part of root.sh script.Now product-specific root actions will be performed.2010-07-27 08:46:51: Parsing the host name2010-07-27 08:46:51: Checking for super user privileges2010-07-27 08:46:51: User has super user privilegesUsing configuration parameter file: /d01/oracle/app/11.2.0/grid/crs/install/crsconfig_paramsCreating trace directoryLOCAL ADD MODECreating OCR keys for user 'root', privgrp 'root'..

Operation successful.root walletroot wallet certroot cert exportpeer walletprofile reader walletpa walletpeer wallet keys


45/49


pa wallet keyspeer cert request

pa cert requestpeer certpa certpeer root cert TPprofile reader root cert TPpa root cert TPpeer pa cert TPpa peer cert TPprofile reader pa cert TPprofile reader peer cert TPpeer user cert

pa user certAdding daemon to inittabCRS-4123: Oracle High Availability Services has been started.ohasd is startingADVM/ACFS is not supported on Redhat 4

CRS-2672: Attempting to start 'ora.gipcd' on 'linux1'CRS-2672: Attempting to start 'ora.mdnsd' on 'linux1'CRS-2676: Start of 'ora.gipcd' on 'linux1' succeededCRS-2676: Start of 'ora.mdnsd' on 'linux1' succeededCRS-2672: Attempting to start 'ora.gpnpd' on 'linux1'

CRS-2676: Start of 'ora.gpnpd' on 'linux1' succeededCRS-2672: Attempting to start 'ora.cssdmonitor' on 'linux1'CRS-2676: Start of 'ora.cssdmonitor' on 'linux1' succeededCRS-2672: Attempting to start 'ora.cssd' on 'linux1'CRS-2672: Attempting to start 'ora.diskmon' on 'linux1'CRS-2676: Start of 'ora.diskmon' on 'linux1' succeededCRS-2676: Start of 'ora.cssd' on 'linux1' succeededCRS-2672: Attempting to start 'ora.ctssd' on 'linux1'CRS-2676: Start of 'ora.ctssd' on 'linux1' succeeded

ASM created and started successfully.

DiskGroup DATA created successfully.

clscfg: -install mode specifiedSuccessfully accumulated necessary OCR keys.Creating OCR keys for user 'root', privgrp 'root'..Operation successful.CRS-2672: Attempting to start 'ora.crsd' on 'linux1'


46/49


CRS-2676: Start of 'ora.crsd' on 'linux1' succeededCRS-4256: Updating the profile

Successful addition of voting disk 0482638d4e5d4f0fbfd40e025d6ca900.Successful addition of voting disk 63878fa4923f4f5dbfadf6a8be49a18a.Successful addition of voting disk 433cdc1d1b7f4f1fbf7f126bc0452222.Successfully replaced voting disk group with +DATA.CRS-4256: Updating the profileCRS-4266: Voting file(s) successfully replaced## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 0482638d4e5d4f0fbfd40e025d6ca900 (ORCL:VOL1) [DATA]2. ONLINE 63878fa4923f4f5dbfadf6a8be49a18a (ORCL:VOL2) [DATA]3. ONLINE 433cdc1d1b7f4f1fbf7f126bc0452222 (ORCL:VOL3) [DATA]

Located 3 voting disk(s).CRS-2673: Attempting to stop 'ora.crsd' on 'linux1'CRS-2677: Stop of 'ora.crsd' on 'linux1' succeededCRS-2673: Attempting to stop 'ora.asm' on 'linux1'CRS-2677: Stop of 'ora.asm' on 'linux1' succeededCRS-2673: Attempting to stop 'ora.ctssd' on 'linux1'CRS-2677: Stop of 'ora.ctssd' on 'linux1' succeededCRS-2673: Attempting to stop 'ora.cssdmonitor' on 'linux1'CRS-2677: Stop of 'ora.cssdmonitor' on 'linux1' succeededCRS-2673: Attempting to stop 'ora.cssd' on 'linux1'CRS-2677: Stop of 'ora.cssd' on 'linux1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'linux1'CRS-2677: Stop of 'ora.gpnpd' on 'linux1' succeededCRS-2673: Attempting to stop 'ora.gipcd' on 'linux1'CRS-2677: Stop of 'ora.gipcd' on 'linux1' succeededCRS-2673: Attempting to stop 'ora.mdnsd' on 'linux1'CRS-2677: Stop of 'ora.mdnsd' on 'linux1' succeededCRS-2672: Attempting to start 'ora.mdnsd' on 'linux1'CRS-2676: Start of 'ora.mdnsd' on 'linux1' succeededCRS-2672: Attempting to start 'ora.gipcd' on 'linux1'CRS-2676: Start of 'ora.gipcd' on 'linux1' succeededCRS-2672: Attempting to start 'ora.gpnpd' on 'linux1'

CRS-2676: Start of 'ora.gpnpd' on 'linux1' succeededCRS-2672: Attempting to start 'ora.cssdmonitor' on 'linux1'CRS-2676: Start of 'ora.cssdmonitor' on 'linux1' succeededCRS-2672: Attempting to start 'ora.cssd' on 'linux1'CRS-2672: Attempting to start 'ora.diskmon' on 'linux1'CRS-2676: Start of 'ora.diskmon' on 'linux1' succeededCRS-2676: Start of 'ora.cssd' on 'linux1' succeededCRS-2672: Attempting to start 'ora.ctssd' on 'linux1'


47/49


CRS-2676: Start of 'ora.ctssd' on 'linux1' succeededCRS-2672: Attempting to start 'ora.asm' on 'linux1'

CRS-2676: Start of 'ora.asm' on 'linux1' succeededCRS-2672: Attempting to start 'ora.crsd' on 'linux1'CRS-2676: Start of 'ora.crsd' on 'linux1' succeededCRS-2672: Attempting to start 'ora.evmd' on 'linux1'CRS-2676: Start of 'ora.evmd' on 'linux1' succeededCRS-2672: Attempting to start 'ora.asm' on 'linux1'CRS-2676: Start of 'ora.asm' on 'linux1' succeededCRS-2672: Attempting to start 'ora.DATA.dg' on 'linux1'CRS-2676: Start of 'ora.DATA.dg' on 'linux1' succeeded

linux1 2010/07/27 08:54:48 /d01/oracle/app/11.2.0/grid/cdata/linux1/back

up_20100727_085448.olrConfigure Oracle Grid Infrastructure for a Cluster ... succeededUpdating inventory properties for clusterwareStarting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 6000 MB PassedThe inventory pointer is located at /etc/oraInst.locThe inventory is located at /d01/oracle/app/oraInventory'UpdateNodeList' was successful.

Appendix C

sh root.shRunning Oracle 11g root.sh script...

The following environment variables are set as:ORACLE_OWNER= oracleORACLE_HOME= /d01/oracle/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:Copying dbhome to /usr/local/bin ...Copying oraenv to /usr/local/bin ...

Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root.sh script.Now product-specific root actions will be performed.


48/49


2010-07-27 08:57:25: Parsing the host name2010-07-27 08:57:25: Checking for super user privileges

2010-07-27 08:57:25: User has super user privilegesUsing configuration parameter file: /d01/oracle/app/11.2.0/grid/crs/install/crsconfig_paramsCreating trace directoryLOCAL ADD MODECreating OCR keys for user 'root', privgrp 'root'..Operation successful.Adding daemon to inittabCRS-4123: Oracle High Availability Services has been started.ohasd is startingADVM/ACFS is not supported on Redhat 4

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node linux1, number 1, and is terminatingAn active cluster was found during exclusive startup, restarting to join the clusterCRS-2672: Attempting to start 'ora.mdnsd' on 'linux2'CRS-2676: Start of 'ora.mdnsd' on 'linux2' succeededCRS-2672: Attempting to start 'ora.gipcd' on 'linux2'CRS-2676: Start of 'ora.gipcd' on 'linux2' succeededCRS-2672: Attempting to start 'ora.gpnpd' on 'linux2'CRS-2676: Start of 'ora.gpnpd' on 'linux2' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'linux2'CRS-2676: Start of 'ora.cssdmonitor' on 'linux2' succeededCRS-2672: Attempting to start 'ora.cssd' on 'linux2'CRS-2672: Attempting to start 'ora.diskmon' on 'linux2'CRS-2676: Start of 'ora.diskmon' on 'linux2' succeededCRS-2676: Start of 'ora.cssd' on 'linux2' succeededCRS-2672: Attempting to start 'ora.ctssd' on 'linux2'CRS-2676: Start of 'ora.ctssd' on 'linux2' succeededCRS-2672: Attempting to start 'ora.asm' on 'linux2'CRS-2676: Start of 'ora.asm' on 'linux2' succeededCRS-2672: Attempting to start 'ora.crsd' on 'linux2'

CRS-2676: Start of 'ora.crsd' on 'linux2' succeededCRS-2672: Attempting to start 'ora.evmd' on 'linux2'CRS-2676: Start of 'ora.evmd' on 'linux2' succeeded

linux2 2010/07/27 09:01:36 /d01/oracle/app/11.2.0/grid/cdata/linux2/backup_20100727_090136.olrPreparing packages for installation...cvuqdisk-1.0.7-1


49/49


Configure Oracle Grid Infrastructure for a Cluster ... succeededUpdating inventory properties for clusterware

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 7994 MB PassedThe inventory pointer is located at /etc/oraInst.locThe inventory is located at /d01/oracle/app/oraInventory'UpdateNodeList' was successful.

Documents

3 Install Clusterware (1)