http://linuxadviser.com/showCat.php?id=MQ==         http://www.puschitz.com/InstallingOracle10gRAC.shtml#InstallingOracleDatabase10gWithRACDBCA                                                                                                                                                      

Pre-Installation Task:    

 Task List:

Minimum Hardware requiredTechnical Architecture of 2-node RACRedhat Packages RequiredConfiguring the Public, Private NetworkConfiguring Shared Storage Based on iSCSI technologyDownload Oracle 11g RDBMS software’s from OTNMemory and Swap SpaceSetting up Kernel ParameterCreating oracle User account.Creating required Directories for Oracle 10g R2 RAC software and setting up correct PermissionSetup shell Limits for the oracle userEnable SSH oracle user Equivalency on all the cluster nodes.Installing and Configuring OCFS2 (Oracle Cluster File System)Creating ASM disks using oracle asm (ASMLib IO) For the Clustered Database

Required Hardware:

To create 2-node RAC using SAN one would require 3 machines with the following hardware installed on it.

Per Node:

1 GB RAM, at least 8 GB of harddrive, 1 GHz CPU, 3 Network Interfaces: 1 for Public, 1 for Private, Cluster interconnect and 1 for storage network    

Per Cluster:

1 Storage Server hosting Storage Operating System1 Network HUB  (Public)1 Network HUB  (Private -- cluster interconnect)1 Network HUB  (Private -- Storage Server)

2 network cables for Public network (eth0)2 network cables for private network for internode communication (eth1)3 network cables for private network for storage Network (eth2)

I used the below hardware to build 2-node RAC cluster for this excercise.

RAC Node1 Dell Intel PIII 1.3 GHz, 256 MB RAM, 20 GB HD

$200 - Used one

RAC Node2 Dell Intel PIII 1.3 GHz, 256 MB RAM, 20 GB HD

$200 - Used one

Upgrade Memory to 512MB on RAC Nodes

256 MB x 2 for Both the Server $110

NFS Storage Server -- IP BASED iSCSI SAN

AMD athelon, 1.2 GHz, 1GB RAM, 60GB HD

$300 -- Used one

USB External HD 500GC USB MyBook WD harddrive $180

Added  2 extra NICs  on Each RAC nodes

D-Link Ethernet card x 4 $30 X 4

Network Hub "NETWORK Everywhere"10/100 5-Port Hub

$30 X 3

Ethernet cable                                                   $15 X 7

 Total Cost: $1305.00           Public Name: Dop-oradb1, dop-oradb2

Private Name: Dop-oradb1-prv, dop-oradb2-prvVirtual Name: Dop-oradb1-vip, dop-oradb1-vip  Operating System: Redhat Linux 4 Update 5 (2.6.9-89.EL)

ORACLE_BASE: /oracle/app/product/10.2/ora_dbASM_BASE: /oracle/app/product/10.2/ora_asmCRS_BASE: /oracle/app/product/10.2/ora_crsCRS file Location: /dev/raw/raw1/ mounted on /dev/sda1 (hpfs/ntfs)

Packages:

Oracle 11g R1 on CentOS EL 4 Update 5 requires below extra packages to be installed with the version same or higher than listed below.

binutils-2.15.92.0.2-18compat-libstdc++-33.2.3-47.3elfutils-libelf-0.97-5elfutils-libelf-devel-0.97.5glibc-2.3.9.4-2.19glibc-common-2.3.9.4-2.19glibc-devel-2.3.9.4-2.19gcc-3.4.5-2gcc-c++-3.4.5-2libaio-devel-0.3.105-2libaio-0.3.105-2libgcc-3.4.5libstdc++-3.4.5-2libstdc++-devel-3.4.5-2make-3.80-5sysstat-5.0.5unixODBC-2.2.11unixODBC-devel-2.2.11iscsi-initiator-utils-4.0.3.0-5 <-- For iscsi based SAN configuration

Please execute the below command as root to make sure that you have this rpms installed. If not installed, then download and install

          Using YUM. rpm -q binutils elfutils-libelf elfutils-libelf-devel glibc glibc-common glibc-devel \       gcc gcc-c++ libaio libaio-devel libgcc libstdc++ libstdc++-devel make \       compat-libstdc++ sysstat unixODBC unixODBC-devel iscsi-initiator-utils

I had to install the below extra rpms:

[root@node1-pub ~]# yum install libaio libaio-devel elfutils-libelf-devel unixODBC unixODBC-devel

Configuring Public and Private network for the Cluster Nodes:

Each node in the cluster must have 3 network adapter (eth0, eth1and eth2) one for the public, second one for the privatenetwork interface (internode communication, interconnect) and third one for the Network Storage System (Private). You make sure that if you configure eth1 as the private interface for node1 then, eth1 must be configured as private interface for the node2.Follow the below step to configre these networks:

(1) Change the hostname value by executing the below command:

For Node node1-pub:

[root@localhost ~] # hostname dop-oradb1

For Node node2-pub:

[root@localhost ~] # hostname dop-oradb2

For Storage Server:

[root@localhost ~] # hostname nas-server

(2) Edit the /etc/hosts file as shown below:

Definition: The IP address 127.0.0.1 is a special purpose address reserved for use on each computer. 127.0.0.1 Is conventionally a computer's loopback address.

Network software and utilities can use 127.0.0.1 to access a local computer's TCP/IP network resources. Messages sent to loopback IP addresses like 127.0.0.1 do not reach outside to the local area network (LAN) but instead are automatically re-routed by the computer's own network adapter back to the receiving end of the TCP/IP stack.

Typically all IP addresses in the range 127.0.0.1 - 127.255.255.255 are reserved for private use, but 127.0.0.1 is by convention the loopback address in almost all cases.

[root@node2-pub ~]# cat /etc/hosts# Do not remove the following line, or various programs# that require network functionality will fail.127.0.0.1               localhost.localdomain localhost 

####Public IPs for RAC#########

10.1.96.26 dop-oradb110.1.96.27 dop-oradb2

########## VIP ####################10.1.96.28 dop-oradb1-vip10.1.96.29 dop-oradb2-vip

########### Private ips ###########

192.168.1.7 dop-oradb1-priv192.168.1.9 dop-oradb2-priv

(3) Edit OR create the /etc/sysconfig/network-scripts/ifcfg-eth0 as shown below:

Create the same file on both the nodes as shown below.

[root@node2-pub ~]# Cat /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0BOOTPROTO=noneHWADDR=D8:D3:85:BA:74:20ONBOOT=yesTYPE=EthernetNETMASK=255.255.255.192IPADDR=10.1.96.26USERCTL=noIPV6INIT=noPEERDNS=yesGATEWAY=10.1.96.1 [root@node2-pub ~] #

(4) Edit OR create the /etc/sysconfig/network-scripts/ifcfg-eth1 as shown below: -<-- For Cluster interconnects

For Node node1-pub:

[root@node1-pub ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1DEVICE=eth1BOOTPROTO=noneHWADDR=D8:D3:85:BA:74:22ONBOOT=yesTYPE=EthernetNETMASK=255.255.255.0IPADDR=192.168.1.7USERCTL=noIPV6INIT=noPEERDNS=yes[root@node1-pub ~]#

(6) Edit the /etc/sysconfig/network file with the below contents:

For Node node1-pub:

[root@node1-pub ~]# cat  /etc/sysconfig/networkNETWORKING=yesHOSTNAME=dop-oradb1

For Node node2-pub:

[root@node2-pub ~]# cat  /etc/sysconfig/networkNETWORKING=yesHOSTNAME=dop-oradb1

(7) Restart the network service OR reboot the nodes:      After I rebooted both the nodes, I verified the network interface configurations by running the ifconfig command as shown below.

[root@node2-pub ~]# ifconfig -aeth0      Link encap:Ethernet  HWaddr 00:06:5B:AE:AE:7F            inet addr:216.160.37.156  Bcast:216.160.37.159  Mask:255.255.255.248          inet6 addr: fe80::206:5bff:feae:ae7f/64 Scope:Link          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1          RX packets:24093 errors:0 dropped:0 overruns:0 frame:0          TX packets:8506 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:1000           RX bytes:4694668 (4.4 MiB)  TX bytes:761581 (743.7 KiB)          Interrupt:11 Base address:0xe480  eth1      Link encap:Ethernet  HWaddr 00:13:46:6A:FC:6D            inet addr:192.168.0.22  Bcast:192.168.0.255  Mask:255.255.255.0          inet6 addr: fe80::213:46ff:fe6a:fc6d/64 Scope:Link          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1          RX packets:150 errors:0 dropped:0 overruns:0 frame:0          TX packets:1355 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:1000           RX bytes:9000 (8.7 KiB)  TX bytes:57078 (55.7 KiB)          Interrupt:10 Base address:0x6800  eth2      Link encap:Ethernet  HWaddr 00:18:F8:0F:0D:C1            inet addr:192.168.1.22  Bcast:192.168.1.255  Mask:255.255.255.0          inet6 addr: fe80::218:f8ff:fe0f:dc1/64 Scope:Link          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1          RX packets:32249 errors:0 dropped:0 overruns:0 frame:0          TX packets:64333 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:1000           RX bytes:3682359 (3.5 MiB)  TX bytes:5795687 (5.5 MiB)          Interrupt:9 Base address:0x6c00  lo        Link encap:Local Loopback            inet addr:127.0.0.1  Mask:255.0.0.0          inet6 addr: ::1/128 Scope:Host          UP LOOPBACK RUNNING  MTU:16436  Metric:1          RX packets:6354 errors:0 dropped:0 overruns:0 frame:0          TX packets:6354 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:0           RX bytes:8304597 (7.9 MiB)  TX bytes:8304597 (7.9 MiB) sit0      Link encap:IPv6-in-IPv4            NOARP  MTU:1480  Metric:1          RX packets:0 errors:0 dropped:0 overruns:0 frame:0          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:0           RX bytes:0 (0.0 b)  TX bytes: 0 (0.0 b) [root@node2-pub ~] #

Downloading Oracle 10g Software:

        Download the Oracle 10g from here.

Memory and swap Space:

Oracle 10g RAC requires to have 1GB of RAM available on each node to sucessfully install 11g RAC. Well, I havemanaged to install it with 512 MB RAM. You will get warning during checking of pre-requise step of installationstep which you can ignore. Please goto Adding an Extra Swapspace if you want to have an extra swapspace added.

Kernel Parameters:

Oracle recommends that you set shared memory segment attributes as well as semaphores to the following values.If not set, database instance creation will fail. I added the following lines to /etc/sysctl.conf file. Every OS process needs semaphore where it waits on for the resources. For

More on semaphore, please read the UNIX os documents.Oracle 10g RAC requires to have 1GB of RAM available on each node to sucessfully install 11g RAC. Well, I havemanaged to install it with 512 MB RAM. You will get warning during checking of pre-requise step of installationstep which you can ignore. Please goto adding an Extra Swapspace if you want to have an extra swapspace added.

 NOTE: If the current value for any parameter is higher than the value listed in this table, then do not change              the value of that parameter.

                  To see the current setting in the kernel, please use the below command.

/sbin/sysctl -a | grep sem      -- for semmsl, semmns, semopm, semmni/sbin/sysctl -a | grep shm      -- for shmall, shmmax, shmmni/sbin/sysctl -a | grep file-max/sbin/sysctl -a | grep ip_local_port_range/sbin/sysctl -a | grep rmem_default

Please add/change the appropriate variables value in the /etc/sysctl.conf file as shown below.[root@node2-pub ~]# cat /etc/sysctl.conf# Kernel sysctl configuration file for Red Hat Linux## For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and# sysctl.conf(5) for more details. # Controls IP packet forwardingnet.ipv4.ip_forward = 0 

# Controls source route verificationnet.ipv4.conf.default.rp_filter = 1 # Do not accept source routingnet.ipv4.conf.default.accept_source_route = 0 # Controls the System Request debugging functionality of the kernelkernel.sysrq = 0 # Controls whether core dumps will append the PID to the core filename.# Useful for debugging multi-threaded applications.

kernel.core_uses_pid = 1# Extra parameters For 11g RAC installation kernel.shmmax = 2147483648kernel.shmmni = 4096kernel.shmall = 2097152kernel.sem = 250 32000 100 128fs.file-max = 6553600net.ipv4.ip_local_port_range = 1024 65000net.core.rmem_default = 4194304net.core.wmem_default = 262144net.core.wmem_max= 262144net.core.rmem_max = 4194304 [root@node2-pub ~]# 

After adding these lines to /etc/sysctl.conf, please run the below command as root to make them enabled.

[root@node2-pub ~]#  sysctl -p

Creating oracle OS User Account:

groupadd -g 900 dba groupadd -g 901 oinstalluseradd -u 900 -g oinstall -G dba oraclepasswd oracle

Please verify that oracle user has same gid and uid on all the RAC nodes by executing the this command

[oracle@node2-pub ~]$ iduid=900(oracle) gid=901(oinstall) groups=901(oinstall),900(dba)

[oracle@node1-pub ~]$ iduid=900(oracle) gid=901(oinstall) groups=901(oinstall),900(dba)

bash profile of user (oracle)

export PATHunset USERNAMEexport ORACLE_CRS_HOME=/oracle/app/product/10.2/ora_crs

export ORACLE_ASM_HOME=/oracle/app/product/10.2/ora_asmexport ORACLE_HOME=/oracle/app/product/10.2/ora_dbexport PATH=$ORACLE_HOME/bin:$ORACLE_CRS_HOME/bin:$PATHexport ORACLE_SID=pli1

Enable SSH oracle user Equivalency on Both the Cluster Nodes:

To configure SSH user equivalency, you must create RSA and DSA keys on each cluster node and copy these keysfrom all the cluster node members into an authorized key file on each node. Follow the below steps to achieve this task.

From each node, logon as oracle$ mkdir ~/.ssh$ chmod 755 ~/.ssh$ /usr/bin/ssh-keygen -t rsaCut and paste the following line separately$ /usr/bin/ssh-keygen -t dsaFrom the first node (dop-oradb1) ONLY, logged in as oracle (copy the local account's keys so that ssh to the local node will work):$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keysNow copy the keys to the other node (dop-oradb2) so that we can ssh to the remote node without being prompted for a password.$ ssh oracle@d1sol6 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys(Run each of them separately. SSH will prompt for the oracle password each time and if the commands are pasted at the same time, the other commands will be lost when the first one flushes the input buffer prior to prompting for the password.)$ ssh oracle@d1sol6 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys$ chmod 644 ~/.ssh/authorized_keys$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys$ ssh oracle@d1sol5 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys$ ssh oracle@d1sol5 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys$ chmod 644 ~/.ssh/authorized_keysAs oracle on the node where the Oracle 10g Release 2 software will be installed (dop-oradb1)$ exec /usr/bin/ssh-agent $SHELL$ /usr/bin/ssh-addTo Test$ ssh d1sol6 dateNow, try executing the date (or any other command) on remote node to make sure that oracle is not asked for the password. Do the same on both the nodes in cluster. You should not recieve any error

message while you execute these commands on all the nodes. If you get any error, first fix them before you go further.

[oracle@node2-pub oracle]$ ssh node1-prv dateThu Sep 20 21:16:56 CDT 2007[oracle@node2-pub oracle]$ ssh node2-prv dateThu Sep 20 21:17:03 CDT 20075[oracle@node2-pub oracle]$ ssh node1-pub dateThu Sep 20 21:17:09 CDT 2007[oracle@node2-pub oracle]$ ssh node2-pub date

Thu Sep 20 21:17:19 CDT 2007

Errors / Warnings during the network configurations:

I got the below warning when I tried below command.

[oracle@node2-pub .ssh]$ ssh node1-pub dateWarning: No xauth data; using fake authentication data for X11 forwarding.Thu Sep 20 20:43:39 CDT 2007

To fix the above warning, create the /home/oracle/.ssh/config file (logged in as oracle user) and make the below entryin it. then run the same command again and the above warning would not show up.

[oracle@node2-pub oracle]$ cat .ssh/config Host *      Forwardx11 no

You may be prompted to enter 'yes' or 'no' for the first time performing ssh commands. Simply enter 'yes' and continue.After words, when oracle connects to the remote node, it won't be asked for the password.

[oracle@node2-pub oracle]$ ssh node1-prv dateThe authenticity of host 'node1-prv (192.168.203.1)' can't be established.RSA key fingerprint is <********************************************>Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added 'node1-prv,192.168.203.1' (RSA) to the list of known hosts.Sun Dec 18 20:01:09 CST 2005[oracle@node2-pub oracle]$ ssh node1-prv dateSun Dec 18 20:01:13 CST 2005[oracle@node2-pub oracle]$

[oracle@node2-pub oracle]$ ssh node2-prv dateWarning: Permanently added the RSA host key for IP address '192.168.203.2' to the list of known hosts.Sun Dec 18 20:14:16 CST 2005[oracle@node2-pub oracle]$ ssh node2-pub dateSun Dec 18 20:15:05 CST 2005

If you get then below erorr message when try to connect to remote node, please make sure that the firewall is disabled on theremote node.

[root@node2-pub root]# telnet node1-prvTrying 192.168.203.1...telnet: Unable to connect to remote host: No route to host

          Partitioning the Shared disk:

I am going to have 4 disks assigned to ASM and one for OCFS. So, I have created a single partition onthese 5 disks. Create partitions from ONLY one of the available RAC nodes. This can be any node in cluster.

[root@node1-pub ~]# fdisk -l   Disk /dev/hda: 160.0 GB, 160041885696 bytes 255 heads, 63 sectors/track, 19457 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes      Device Boot      Start         End      Blocks   Id  System /dev/hda1   *           1          13      104391   83  Linux /dev/hda2              14       19457   156183930   8e  Linux LVM   Disk /dev/sda: 80.6 GB, 80664854528 bytes 255 heads, 63 sectors/track, 9806 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes   Disk /dev/sda doesn't contain a valid partition table   Disk /dev/sdb: 104.8 GB, 104857600000 bytes 255 heads, 63 sectors/track, 12748 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes   Disk /dev/sdb doesn't contain a valid partition table   Disk /dev/sdc: 104.8 GB, 104857600000 bytes 255 heads, 63 sectors/track, 12748 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes   Disk /dev/sdc doesn't contain a valid partition table   Disk /dev/sdd: 104.8 GB, 104857600000 bytes 255 heads, 63 sectors/track, 12748 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes   Disk /dev/sdd doesn't contain a valid partition table   Disk /dev/sde: 104.8 GB, 104857600000 bytes 255 heads, 63 sectors/track, 12748 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes   Disk /dev/sde doesn't contain a valid partition table

----==========Create partitions================

[root@node1-pub ~]# fdisk /dev/sda Device contains neither a valid DOS partition table, nor Sun, SGI or OSF

disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.     The number of cylinders for this disk is set to 9806. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs    (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by

w(rite)   Command (m for help): n Command action    e   extended    p   primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-9806, default 1):

Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-9806, default 9806):

Using default value 9806   Command (m for help): w The partition table has been altered!   Calling ioctl() to re-read partition table. Syncing disks. [root@node1-pub ~]# fdisk /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF

disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.     The number of cylinders for this disk is set to 12748. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs    (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by

w(rite)   Command (m for help): n Command action    e   extended    p   primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-12748, default 1):

Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-12748, default 12748):

Using default value 12748   Command (m for help): w The partition table has been altered!   Calling ioctl() to re-read partition table. Syncing disks. [root@node1-pub ~]# fdisk /dev/sdc Device contains neither a valid DOS partition table, nor Sun, SGI or OSF

disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.     The number of cylinders for this disk is set to 12748. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs    (e.g., DOS FDISK, OS/2 FDISK)

Warning: invalid flag 0x0000 of partition table 4 will be corrected by

w(rite)   Command (m for help): n Command action    e   extended    p   primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-12748, default 1):

Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-12748, default 12748):

Using default value 12748   Command (m for help): p   Disk /dev/sdc: 104.8 GB, 104857600000 bytes 255 heads, 63 sectors/track, 12748 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes      Device Boot      Start         End      Blocks   Id  System /dev/sdc1               1       12748   102398278+  83  Linux   Command (m for help): w The partition table has been altered!   Calling ioctl() to re-read partition table. Syncing disks. [root@node1-pub ~]# fdisk /dev/sdd Device contains neither a valid DOS partition table, nor Sun, SGI or OSF

disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.     The number of cylinders for this disk is set to 12748. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs    (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by

w(rite)   Command (m for help): n

Command action    e   extended    p   primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-12748, default 1):

Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-12748, default 12748):

Using default value 12748   Command (m for help): p   Disk /dev/sdd: 104.8 GB, 104857600000 bytes 255 heads, 63 sectors/track, 12748 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes      Device Boot      Start         End      Blocks   Id  System /dev/sdd1               1       12748   102398278+  83  Linux   Command (m for help): w The partition table has been altered!   Calling ioctl() to re-read partition table. Syncing disks. [root@node1-pub ~]# fdisk /dev/sde Device contains neither a valid DOS partition table, nor Sun, SGI or OSF

disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.     The number of cylinders for this disk is set to 12748. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs    (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by

w(rite)   Command (m for help): n Command action    e   extended    p   primary partition (1-4) p

Partition number (1-4): 1 First cylinder (1-12748, default 1):

Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-12748, default 12748):

Using default value 12748   Command (m for help): p   Disk /dev/sde: 104.8 GB, 104857600000 bytes 255 heads, 63 sectors/track, 12748 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes      Device Boot      Start         End      Blocks   Id  System /dev/sde1               1       12748   102398278+  83  Linux   Command (m for help): w The partition table has been altered!   Calling ioctl() to re-read partition table. Syncing disks. [root@node1-pub ~]#

Listing the Partitions:   [root@node1-pub ~]# fdisk -l   Disk /dev/hda: 160.0 GB, 160041885696 bytes 255 heads, 63 sectors/track, 19457 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes      Device Boot      Start         End      Blocks   Id  System /dev/hda1   *           1          13      104391   83  Linux /dev/hda2              14       19457   156183930   8e  Linux LVM   Disk /dev/sda: 80.6 GB, 80664854528 bytes 255 heads, 63 sectors/track, 9806 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes      Device Boot      Start         End      Blocks   Id  System /dev/sda1               1        9806    78766663+  83  Linux   Disk /dev/sdb: 104.8 GB, 104857600000 bytes 255 heads, 63 sectors/track, 12748 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes      Device Boot      Start         End      Blocks   Id  System

/dev/sdb1               1       12748   102398278+  83  Linux   Disk /dev/sdc: 104.8 GB, 104857600000 bytes 255 heads, 63 sectors/track, 12748 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes      Device Boot      Start         End      Blocks   Id  System /dev/sdc1               1       12748   102398278+  83  Linux   Disk /dev/sdd: 104.8 GB, 104857600000 bytes 255 heads, 63 sectors/track, 12748 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes      Device Boot      Start         End      Blocks   Id  System /dev/sdd1               1       12748   102398278+  83  Linux   Disk /dev/sde: 104.8 GB, 104857600000 bytes 255 heads, 63 sectors/track, 12748 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes      Device Boot      Start         End      Blocks   Id  System /dev/sde1               1       12748   102398278+  83  Linux [root@node1-pub ~]#

/etc/sysconfig/rawdevices(both node)

# This file and interface are deprecated.# Applications needing raw device access should open regular# block devices with O_DIRECT.# raw device bindings# format: <rawdev> <major> <minor># <rawdev> <blockdev># example: /dev/raw/raw1 /dev/sda1# /dev/raw/raw2 8 5

#Vote disk for rac/dev/raw/raw1 /dev/sda1/dev/raw/raw2 /dev/sdc1/dev/raw/raw3 /dev/sdd1#ocr for rac size 214mb/dev/raw/raw4 /dev/sde1/dev/raw/raw5 /dev/sdf1

/etc/rc.local (both node)

touch /var/lock/subsys/local#Hangcheck timer for oracle racinsmod /lib/modules/2.6.9-55.ELsmp/kernel/drivers/char/hangcheck-timer.ko hangcheck_tick=30 hangcheck_margin=180

#Permisstion for deviceschown oracle:dba /dev/raw/raw1chown oracle:dba /dev/raw/raw2chown oracle:dba /dev/raw/raw3chown root:dba /dev/raw/raw4chown root:dba /dev/raw/raw5#permission for ocr and votediskchmod 660 /dev/raw/raw1chmod 660 /dev/raw/raw2chmod 660 /dev/raw/raw3chmod 640 /dev/raw/raw4chmod 640 /dev/raw/raw5

Installing CRS for Oracle 10g

Make sure all required software is under /xyz/xyz directoryCd /clusterware/

./runInstaller

Change the Name to CRS_HOME and path to /oracle/app/product/10.2/crs

Click Edit and modify Virtual hostnames to match the names in /etc/hosts

ipge0 is the public nic, so click Edit and change ipge0 interface type to Public

Click Edit and change OCR location to match the names in /etc/sysconfig/rawdeveics

Click Edit and change OCR location to match the names in /etc/sysconfig/rawdeveics

Don’t click the OK button now. Run the scripts one by one on each node, configure Virtual ip’s using vipca and then comeback and click OK

Logon as root on node1 (dop-oradb1) and run# /opt/oracle/oraInventory/orainstRoot.shand# /opt/oracle/product/10g/crs/root.sh

Logon as root on 2nd node (dop-oradb2) and run# /opt/oracle/oraInventory/orainstRoot.sh and# /opt/oracle/product/10g/crs/root.sh

Make sure CSS is active on both the nodesAt the end of the script, we might receive following error messages saying “ipge0 is not public”Running vipca(silent) for configuring nodeappsThe given interface(s), "ipge0" is not public. Public interfaces should be used to configure virtual IPs.

root@dop-oradb2 # cd /oracle/product/10g/crs/binroot@dop-oradb2 #./vipca

Enter Virtual IP alias name, d1sol5-virt.campd.local, press tab remaining column values will be filled automatically

Click OK after running the scripts and vipca

Click Yes

Installing the Oracle 10g BinariesLog on as oracle$ cd /database$ ./runInstaller

Change Name to ORACLE_HOME and Path to /opt/oracle/product/10g/db_1

Ignore the OUI finding in the area of “noexec_user_stack for the Solaris 10 OS (Refer Metalink 377679.1)Checking for noexec_user_stack=1; found no entry.Failed <<<<Check complete. The overall result of this check is: Failed <<<<

Don’t click OK now

Logon as root on node1 (d1sol5) and run# /opt/oracle/product/10g/db_1/root.sh

Logon as root on node1 (d1sol6) and run#/opt/oracle/product/10g/db_1/root.sh

Creating Automatic Storage Management (ASM) Disks for the Clustered Database:

          Download the below rpms and installed them as root user on both the nodes.

[oracle@node2-pub ~]$ uname -a

Linux node2-pub.hingu.net 2.6.9-55.0.2.EL #1 Tue Jun 26 14:08:18 EDT 2007 i686 i686 i386 GNU/Linux

oracleasm-2.6.9-55.0.2.EL-2.0.3-1.i686.rpmoracleasmlib-2.0.2-1.i386.rpmoracleasm-support-2.0.3-1.i386.rpm

[root@node1-pub rpms]# rpm -ivh oracleasm-2.6.9-55.0.2.EL-2.0.3-1.i686.rpm \ > oracleasmlib-2.0.2-1.i386.rpm \ > oracleasm-support-2.0.3-1.i386.rpm Preparing...                ###########################################

[100%]    1:oracleasm-support      ###########################################

[ 33%]    2:oracleasm-2.6.9-55.0.2.###########################################

[ 67%]    3:oracleasmlib           ###########################################

[100%] [root@node1-pub rpms]#  

Configure ASM on both the nodes

[root@node1-pub ~]# /etc/init.d/oracleasm configure Configuring the Oracle ASM library driver.   This will configure the on-boot properties of the Oracle ASM library driver.  The following questions will determine whether the driver is loaded on boot and what permissions it will have.  The current values will be shown in brackets ('[]').  Hitting <ENTER> without typing an answer will keep that current value.  Ctrl-C will abort.   Default user to own the driver interface []: oracle Default group to own the driver interface []: dba Start Oracle ASM library driver on boot (y/n) [n]: y Fix permissions of Oracle ASM disks on boot (y/n) [y]:

Writing Oracle ASM library driver configuration:           [  OK  ] Creating /dev/oracleasm mount point:                       [  OK  ] Loading module "oracleasm":                                [  OK  ] Mounting ASMlib driver filesystem:                         [  OK  ] Scanning system for ASM disks:                             [  OK  ] [root@node1-pub ~]#

Create ASM Disk Device(s) that will be used in ASM diskgroup (stamping devises as an ASM disks): from one node only

[root@node1-pub ~]# /etc/init.d/oracleasm createdisk DSK1 /dev/sdb1 Marking disk "/dev/sdb1" as an ASM disk:                   [  OK  ] [root@node1-pub ~]# /etc/init.d/oracleasm createdisk DSK2 /dev/sdc1 Marking disk "/dev/sdc1" as an ASM disk:                   [  OK  ]

[root@node1-pub ~]# /etc/init.d/oracleasm createdisk DSK3 /dev/sdd1 Marking disk "/dev/sdd1" as an ASM disk:                   [  OK  ] [root@node1-pub ~]# /etc/init.d/oracleasm createdisk DSK4 /dev/sde1 Marking disk "/dev/sde1" as an ASM disk:                   [  OK  ] [root@node1-pub ~]# /etc/init.d/oracleasm scandisks Scanning system for ASM disks:                             [  OK  ] [root@node1-pub ~]# /etc/init.d/oracleasm listdisks DSK1 DSK2 DSK3 DSK4 [root@node1-pub ~]#

[root@node1-pub ~]# /etc/init.d/oracleasm status Checking if ASM is loaded:                                 [  OK  ] Checking if /dev/oracleasm is mounted:                     [  OK  ] [root@node1-pub ~]#

On the other node, you only need to execute the below command to show these disks up there.

[root@node2-pub ~]# /etc/init.d/oracleasm scandisks[root@node2-pub ~]# /etc/init.d/oracleasm listdisks

DSK1 DSK2 DSK3 DSK4

http://www.idevelopment.info/data/Oracle/DBA_tips/Linux/LINUX_12.shtmlhttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle10gRAC/CLUSTER_18.shtmlhttp://oracleadvisor.com/documentation/oracle/database/11.2/server.112/e10880/prctmsg.htmhttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle10gRAC/CLUSTER_65.shtml#Administering the OCR Filehttp://www.fadalti.com/oracle/database/managed_errors_oracle.htm

http://download.oracle.com/docs/cd/B19306_01/relnotes.102/b15666/toc.htm

Voting disk: is a file that manages information about node membership,OCR: is a file that manages cluster and RAC database configuration information

6.4 List of Packages for Asianux 2, Oracle Linux 4, and Red Hat Enterprise Linux 4

In "Checking the Software Requirements" section of Oracle Database Quick Installation Guide for Linux x86-64 and Chapter 2 of Oracle Database Installation Guide for Linux x86-64, the following (or later versions) are the list of packages for Asianux 2, Oracle Linux 4, and Red Hat Enterprise Linux 4:

binutils-2.15.92.0.2-10.EL4compat-db-4.1.25-9compat-libstdc++-33-3.2.3-47.3compat-libstdc++-33-3.2.3-47.3(i386)compat-libstdc++-296.i386control-center-2.8.0-12gcc-3.4.3-22.1gcc-c++-3.4.3-22.1glibc-2.3.4-2glibc-2.3.4-2(i386)glibc-common-2.3.4-2glibc-devel-2.3.4-2glibc-devel-2.3.4-2(i386)gnome-libs-1.4.1.2.90-44.1libaio-0.3.96-3libgcc-3.4.3-9.EL4libstdc++-3.4.3-9.EL4libstdc++-devel-3.4.3-9.EL4make-3.80-5numactl-0.6.4.x86_64pdksh-5.2.14-30sysstat-5.0.5-1