HP Parallel Database Cluster installation and configuration guide for Oracle 10g RAC, OAS, SuSE Linux (SLES9SP3) and HP ProLiant server architectures (AMD64 & Intel32)

An HP Reference Architecture for Oracle

Overview............................................................................................................................................ 3 Intended audience ............................................................................................................................... 3 Architecture overview........................................................................................................................... 4 Supported hardware and software for RAC /OAS................................................................................... 4 Hardware requirements for Real Application Clusters (RAC)...................................................................... 4 Hardware requirements for Oracle Application Server (OAS) ................................................................... 5 1.1 Downloading software for RAC and or OAS installation..................................................................... 6

1.1.1 Download Support Matrix and environment variables ................................................................. 6 1.1.2 Download ASMLIB library RPMs [Not required for OAS] ............................................................. 6 1.1.3 Download Oracle Clusterware and Database binaries [Required for RAC install only] .................... 7 1.1.4 Download for OAS CD for J2EE [Required for OAS install only] ................................................... 7 1.1.5 Download ProLiant Support Pack .............................................................................................. 7 1.1.6 Scripts available with PDC kit ................................................................................................... 8

1.2 System setup information, MetaLink patches, OTN, and OS CDs......................................................... 8 1.2.1 Obtaining Oracle patchsets and patches ................................................................................... 8 1.2.2 Operating system CDs............................................................................................................. 8 1.2.3 Hostnames, IP addresses.......................................................................................................... 8

2 Servers/Storage hardware configuration update .................................................................................. 8 2.1 Update HBA BIOS ..................................................................................................................... 8 2.2 Shared storage firmware update.................................................................................................. 9 2.3 Configuring iLO on blade servers................................................................................................. 9 2.4 Configuring the Management Server for SLES9SP3 installation on blade servers................................ 9

3 Installation of SLES9SP3 operating system.......................................................................................... 10 3.1 Updating with SP3 ................................................................................................................... 13 3.2 Verify that OS packages and kernel versions are correct .............................................................. 14

4 Prerequisites for installation of RAC or OAS....................................................................................... 154.1 Network configuration.............................................................................................................. 15 4.2 Install ProLiant Support Pack on each node ................................................................................. 16 4.3 Copy and execute script to set kernel parameters......................................................................... 17 4.4 Configure shared storage [required for RAC installation only] ....................................................... 18 4.5 Create sub-partitions on the LUNs for Oracle............................................................................... 19 4.6 Execute script to configure CRS partitions for LUN persistency ....................................................... 19 4.7 Install and configure ASMLIB ..................................................................................................... 20 4.8 Configure SSH......................................................................................................................... 21

5 Installing 10g Oracle OAS using Oracle Universal Installer (OUI)......................................................... 23 5.1 Installing 10g Oracle Application Server .................................................................................... 23

5.1.1 Installing Oracle Application Server..................................................................................... 23 6 Installing 10g Oracle RAC using OUI ............................................................................................... 24

6.1 Installing 10g Oracle Real Application Cluster ............................................................................ 24 6.2 Extract Oracle binaries ............................................................................................................. 24 6.3 Notes on using Oracle documentation........................................................................................ 25 6.4 Verification of system readiness using CLUVFY ............................................................................ 25 6.5 Synchronizing time between the nodes ....................................................................................... 26 6.6 Installing Oracle Clusterware..................................................................................................... 27 6.7 Installing ASM binaries ............................................................................................................. 28 6.8 Install Oracle database binaries ................................................................................................ 28 6.9 Creating the Oracle RAC Database using DBCA ......................................................................... 29 6.10 Verification of the sample database ......................................................................................... 29

Appendix A: PDC Scripts.................................................................................................................... 31 Appendix B: Configuring MSA1000/1500 shared storage.................................................................... 32

Configuring MSA1000/1500 storage ............................................................................................. 33 Configuring Selective Storage Presentation ....................................................................................... 33

Appendix C: Configuring the HP Enterprise Virtual Array EVA4000, EVA6000 and EVA8000 .................. 35 Pre-configuration ............................................................................................................................ 35 Installation..................................................................................................................................... 35 Opening Command View EVA ........................................................................................................ 35 Configuring EVA host profiles .......................................................................................................... 36 Configuring and presenting LUNs .................................................................................................... 36

Appendix D: fdisk partitioning............................................................................................................. 38 For more information.......................................................................................................................... 40

Overview This document describes a set of steps and processes that will help the user complete the installation of Oracle® 10g Real Application Clusters (RAC) or Oracle Application Server (OAS) in a SuSE Linux Enterprise Server 9 SP3 (SLES9SP3), AMD™ 64-bit, and Intel® 32-bit based HP ProLiant servers and HP StorageWorks storage environment. This document also describes the installation tools and architecture that will allow rapid and repeatable deployments of Oracle 10g RAC and Oracle Application Server 10g in a Linux environment.

HP Parallel Database Cluster (PDC) kits and documents are designed to reduce the cost and complexity of rolling out Oracle 10g RAC and Application Server 10g deployments. This document describes how to use a variety of HP tools to prepare the hardware and SLES9SP3 OS environment so the Oracle database and Application Server can be quickly and easily installed.

Intended audience This document is intended for systems administrators and Oracle database administrators who want to improve the installation process for Oracle Real Application Clusters and Oracle Application servers, particularly using HP servers and storage.

3

Architecture overview

Figure 1. Architectural overview

Public network gateway to clients

2-to-8 application servers: HP BladeSystem p-Class 2 or 4 CPUs Up to 12 Gigabytes RAM . . .

. . .

#1

2 #1

#2

#n

#n

2-to-8 database servers: HP BladeSystem p-Class 2 or 4 CPUs Up to 12 Gigabytes RAM per server

HP SANs: HP StorageWorks EVA or MSA SAN

1 Gigabit Ethernet network Fibre channel storage area network Cluster interconnect 1 Gigabit Ethernet

Redundant connections for high availability

Management Server

Supported hardware and software for RAC /OAS Refer to the support matrix for the latest list of all servers supported. This can be obtained at the following URL: http://h18026.www1.hp.com/solutions/enterprise/highavailability/oracle/linux-rac/documentation.html.

Hardware requirements for Real Application Clusters (RAC) • RAC installation requires a minimum setup of two servers (or blade servers). Each of the servers

(nodes) in the cluster have a minimum of either four 18GB disk drives or two 36GB drives. If the user requires the archive logs to be stored locally, a minimum of four 36GB drives are required (two for the operating system and Oracle software and two for the archive logs).

• Servers must have at least two NICs, one each for public and private network interconnect. If you require network redundancy, then additional NIC cards, one each for Public or Private Bonding (Teaming), are required. If you are installing using HP ProLiant Essentials Rapid Deployment Pack (RDP), then each server node must have a NIC capable of PXE boot.

• All Ethernet adapters are supported by HP.

4

• The appropriate network switch, depending on the speed of interconnect (Ethernet or Gigabit Ethernet)

• QLogic Host Bus Adapters (HBAs). The server requires that at least one HBA connects to shared storage. If redundancy is required, then additional HBAs are necessary. For HP storage, the requirement is to have HP StorageWorks FCA2214 HBA installed in each server.

• Gigabit Interface Converter-Shortwave (SFP) modules • Fibre Channel cables • Shared Storage. RAC nodes must be attached to shared storage. This document explains the

configuration of HP StorageWorks Modular Smart Array (MSA) and Enterprise Virtual Array (EVA) storage subsystems in the Appendices. The following are the details of the supported EVA and MSA storage configurations:

– MSA1000 /MSA1500 Storage Solution: HP StorageWorks 1000 /1500 Modular Smart Array storage system MSA1000/1500 Controller installed in each MSA1000/1500 Modular Smart Array 1000 /1500 expansion tray or additional storage enclosure

– EVA4000, 6000, 8000 and EVA3000, 5000 Storage Solution: HP StorageWorks Enterprise Virtual Array Storage System

Hardware requirements for Oracle Application Server (OAS) • OAS requires a server with minimum of 36GB local disk storage. OAS requires at least one NIC

card. • The server must have a minimum of 2GB memory.

PDC hardware configurations Below is an example of the hardware components that could be used in the configurations described in this document. Unless noted, all listed items are per server node.

Quantity Description

4-8 HP ProLiant BL25p Blade Server with x86_64 architecture, 2 CPUs (single- or dual-core), 2GB of system memory, and 2 36GB hard disks

2 QLogic Fibre Channel Host Bus Adapter FCA2214 (per server)

4 10/100 iLO NIC dedicated to management if enhanced backplane is used. Otherwise, each server has 3 PCI-X Gigabit NICs and 10/100 iLO NIC

2 HP GbE2 Interconnect Ethernet switch (per blade enclosure)

2 HP ProCurve 2724 network switch

2 HP StorageWorks SAN Switch 2/16

5

Quantity Description

1 HP StorageWorks 8000 Enterprise Virtual Array (EVA8000)

as needed

Shortwave Fibre cables

as needed

Cat5 network cables

1 HP ProLiant DL360 server as a Management Server

1.1 Downloading software for RAC and or OAS installation Note: You need an account at otn.oracle.com to download Oracle software and accept Oracle License for download

1.1.1 Download Support Matrix and environment variables Obtain the support matrix for the latest list of all servers supported. This can be obtained at the following URL: http://h18026.www1.hp.com/solutions/enterprise/highavailability/oracle/linux-rac/documentation.html.

1.1.2 Download ASMLIB library RPMs [Not required for OAS] To download ASMLIB RPMs from the Oracle Technology Network website, perform the following. You need these RPMs only if you want to use ASM over ASMLIB.

1. Open a browser window and go to http://www.oracle.com/technology/software/tech/linux/asmlib/index.html

2. Locate the RPMs under OS type and OS architecture type. 3. Select the RPMs for your OS:

oracleasmlib-<Version>.< OS architecture>rpm [User space library]

oracleasm-support-<Version>.<OS architecture>.rpm [Driver support files]

4. Download the ASMLIB RPMs. 5. Select the ASM RPM

oracleasm-<kernel-version>.-<Version>.<OS architecture>.rpm

6. Download the ASM RPMs.

6

1.1.3 Download Oracle Clusterware and Database binaries [Required for RAC install only]

Note: You need an account at otn.oracle.com to download Oracle software and accept Oracle License for download.

Download the Oracle binaries from otn.oracle.com for Oracle 10g for the OS type you are installing.

1. Open a browser window and go to http://otn.oracle.com 2. Click on Downloads and select Database from the top menu of the screen. 3. Locate and select the appropriate version for x86_64 or x86 Linux as “Oracle Database 10g

Release 2 (10.2.0.1.0) for Linux” and accept license. Please check the support matrix for latest version supported.

4. Download the “<10gversion>_database_linux_cpio.gz>” file. 5. Download the “Oracle Clusterware Release <version> for x86_64.cpio.gz>” file. The size of the binary for x86_64 or x86 may exceed 670MB, so HP recommends that the binary be downloaded onto a DVD which can be copied into the first node of the RAC cluster.

1.1.4 Download for OAS CD for J2EE [Required for OAS install only]

Note: You need an account at otn.oracle.com to download Oracle software and to accept the Oracle License for download.

To download OAS from the Oracle Technology Network website:

1. Open a browser window and go to http://otn.oracle.com. 2. Click on Downloads on the left half of the screen. 3. Select Application Server under Downloads. 4. Locate the Latest supported version of Oracle Application Server for Linux. 5. Download the “as_linux_<10gVersion ..>.cpio.gz” file. 6. Download the Oracle binaries from otn.oracle.com for Oracle 10g for the OS type you are

installing. These are in cpio.gz format. Copy this to a CD or to an NFS server that is accessible from the node on which you are installing OAS.

1.1.5 Download ProLiant Support Pack

To download the ProLiant Support Pack from the HP website:

1. Open a browser and go to http://www.hp.com. 2. Click Support & Troubleshooting link. 3. Select Servers. 4. Select ProLiant /tc Series and HP BladeSystem Servers. 5. Select the correct server series. 6. If asked, select the correct server model. 7. Select Download drivers and software. 8. Select the correct OS.

7

9. Scroll to the Software-Support Pack. 10. Click ProLiant Support Pack for <OS type you are installing >. 11. When that page displays, select “download >>” to save the file. This will be a tar file that can be

used to install the agents and NIC driver.

1.1.6 Scripts available with PDC kit The PDC kit contains scripts as mentioned in Appendix A of this document. This has to be copied to each node of the cluster after the OS has been installed.

1.2 System setup information, MetaLink patches, OTN, and OS CDs

1.2.1 Obtaining Oracle patchsets and patches Patchsets and patches should be obtained on an as needed basis from the Oracle MetaLink website at http://metalink.oracle.com.

Note: You need an account at metalink.oracle.com to obtain patchsets and or patches for Oracle software.

1.2.2 Operating system CDs Obtain SLES9SP3 base OS install CDs (1-6) and SP3 patch CDs (1-3) for OS installation.

1.2.3 Hostnames, IP addresses In RAC installation, obtain two static IP addresses for each host, and hostname to be installed. In OAS, at least one static IP address is required along with hostname.

2 Servers/Storage hardware configuration update

2.1 Update HBA BIOS Verify that QLogic BIOS version 1.45 or later is installed at boot time; otherwise, update the BIOS to the latest version:

www.hp.com ->

->Select Large Enterprise Business dropdown menu and then select Storage.

-> Select San Infrastructure.

-> Select Fibre Channel Host Bus Adapters.

-> Select HP StorageWorks FCA2214 <or if different, the QLogic Card that you have>.

-> Click on Software & drivers

-> Select the operating system.

Download the BIOS.

8

Alternatively, you can also obtain it from the following link http://h18006.www1.hp.com/storage/saninfrastructure/hba.html.

Update the QLogic HBA adapter BIOS version. The readme file that comes with the new BIOS flash utility gives instructions for using a boot floppy with the proper parameters to perform the update.

2.2 Shared storage firmware update Verify that the shared storage firmware is greater than the version in the Support matrix document. If not, upgrade the storage firmware (EVA or MSA). Refer to the storage documentation CD that comes along with the storage to upgrade the firmware.

2.3 Configuring iLO on blade servers The HP Integrated Lights-Out (iLO) firmware level for the ProLiant servers must be at v1.84 or later. Please check www.hp.com for the latest supported version, download and upgrade if needed.

If the ProLiant server will be accessed remotely make sure that the iLO IP is configured properly.

The HP PDC installation requires that all servers in the cluster have public and private network connectivity during the installation.

The user must be adept in the use of iLO technology before proceeding with the installation. This document does not discuss configuration or use of iLO. Refer to the server documentation describing the iLO configuration and usage.

IMPORTANT: The Java™ Run-time Environment (RTE) software version 1.4 or later is needed for the iLO screens to work properly.

2.4 Configuring the Management Server for SLES9SP3 installation on blade servers

Important: If HBA ports are connected to the fibre channel switches, disconnect cables now from the servers or switches side. This will allow recognizing the server’s local disk drive as a place to install the OS

Note: A management server (with Microsoft® Windows® Server 2003 operating system installed and the network interface connected to the public subnet of the blade servers

9

under OS deployment) is required to install SLES9SP3 on blade servers.

Execute the following steps: • Insert the SLES9 first CD into the CD-ROM of the Windows server. • Open the browser on the Windows-based server and login to the iLO port of the blade server

where the OS will be installed

Note: For instructions on how to use iLO refer to the HP ProLiant server documentation

• Select Virtual Media. • Respond Yes to the security warnings. • Select Local CD-ROM Drive and click Connect. • Connect monitor, mouse and keyboard to the first blade server or use iLO-based Remote

Access or use iLO-based Remote Console. • Power on or restart the blade server. • The Novell SLES installation screen appears. • Go to Section 3 and continue the installation.

3 Installation of SLES9SP3 operating system Verify that the fibre channel cables are disconnected from the server or shared storage. This will prevent the OS from putting the master boot record on the shared disk during OS install.

When installing a 32-bit OS on an Intel x86-64 server, disregard the warning: “You are about to install 32-bit Software on a 64-bit computer” and continue.

1. Boot the server, using the SLES9 CD. – You may need to change your BIOS settings to allow booting from the CD. 2. The Novell SLES installation screen appears. 3. Select Installation 4. The installer scans your hardware and presents the YaST interface 5. Language Selection. 6. Accept the license agreement. 7. Accept the default, English (US). 8. Installation Settings. 9. Installation Settings. Select New Installation.

– This leads to Installation Settings. Verify the settings.

9.1 Select Partitioning

10

Note: The partitioning scheme is done using ext3 filesystem and requires a swap partition up to twice the size of memory if the memory in the server is less than or equal to 4GB. If the memory is greater than 4GB, then the swap size should be equal to memory size. For Oracle binaries, a partition of up to 8GB is suitable. The table below provides partitioning schema guidelines, used for a 36GB (RAID 1+0) local drive.

Under partitioning

9.1.1 Select Create custom partition setup and click on Next.

9.1.2 Select Custom partitioning – for experts and click on Next.

9.1.3 Select device /dev/cciss/c0d0 and click on Create

9.1.4 Select Primary partition and click OK.

This brings up a panel or dialog box. Repeat steps 9.1.3 - 9.1.4 above and create the following three partitions as primary partitions. Select the information in the panel box as shown below.

Mount Point File System End (Size) Format Partition Type

/ ext3 +6GB Yes Primary

/usr ext3 +5GB Yes Primary

/opt ext3 +8GB Yes Primary

9.1.5 Select device /dev/cciss/c0d0 and click on Create. 9.1.6 Select extended partition and click OK to create an extended partition of the

remaining disk. 9.1.7 Select the extended partition which you just created and click Create.

This brings up a panel. Create the following extended partitions using the information below to fill in the panel and by repeating step 9.1.7 for each partition. For swap partitions, select the File system as swap then the mount point as swap.

Mount Point File System End (Size) Format Partition Type (extended)

/tmp ext3 +2GB Yes

swap swap +2GB Yes

swap swap +2GB Yes

swap swap +2GB Yes

swap swap +2GB Yes

/var ext3 +2GB Yes

/home ext3 +2GB Yes

Click Next. This brings the screen back to installation settings.

11

9.2 Select Software

– Click on the link for Software. – The Software Selection screen appears. – Click on Detailed Selection. – The left-hand window displays a list of software selections. Click on the box next to each selection

to select/deselect it. – Select the following software (this is the recommended set; all others should be deselected):

- Basis Runtime System - YaST - Linux Tools - Help & Support Documentation Under this selection on the right screen, deselect susehelp & susehelp-en. - Graphical Base System - GNOME System - Basis Sound Library and Tools

– Libraries and Tools - C/C++ Compiler and Tools - Analyzing Tools

– Select search tab in the filter and enter gnome-libs as the search and click on search. Select the gnome-libs, gnome-libs-32bit, and gnome-libs-devel packages.

Note: It is recommended that the following items not be installed, as they may conflict with Oracle products providing the same service: - Simple Webserver - Authentication Server (NIS, LDAP, Kerberos)

– Select Accept. This will display some additional packages to resolve dependences, Select Continue.

10. Time Zone, Set your time zone. Click on Accept. 11. Accept the installation now that all the required OS settings are selected. 12. A warning box appears. Click on Yes, install when ready to proceed. 13. Change CDs as prompted by the installer. 14. Password for "root," the system administrator. Enter the root password and repeat to confirm. 15. Configure your network interface(s):

– Delete all the configured networks if any, and configure two networks at minimum for RAC as recommended below.

– Static IP addresses are recommended for servers – one for public network and one for private network – and are required for Oracle RAC configuration. Also enter hostname of the server. This setting should configure at least two networks with static IP addresses for public and private networks.

– The following can be ignored: configure printers, modems, ISDN adapters, or sound in order to build a database.

– Click on Next when ready to proceed. 16. Test Internet Connection

– There is no need to connect to the Internet to download release notes or Linux updates. Select No, Skip This Test and click Next to continue.

17. Service Configuration – There is no need to configure CA Management or OpenLDAP Server. Select Skip Configuration.

12

18. User Authentication Method – Select Local (/etc/passwd).

19. Add a New Local User – Create an account for yourself. Do not create the oracle account at this time, as this will be done

by the script later. 20. Release Notes

– Click on Next 21. Hardware Configuration

– Confirm Hardware Detection – Graphics Cards o Click on the discovered Graphic Card. o Select the desired monitor type and click Finish. o Set the desired monitor Resolution.

– There is no need to configure sound cards for a database – click on Skip. – If the graphics card and monitor have been successfully detected, click on Next. Otherwise, click

on the Graphics Cards link and enter the correct information. 22. Installation Completed

– Click on Finish.

3.1 Updating with SP3 The SLES9 kernel has to be updated to SP3; to do this execute the following steps:

1. Login to root and open up a terminal window and enter command as follows: # yast2 2. Insert CD #1 of SP3 SLES9 patchset CDs. 3. Select Software in the yast2 control centre and then select Patch CD Update. 4. This invokes a screen for update. Select Next, and this searches through the RPM database and

displays a yast2 patchset screen. 5. Select Package Groups in the filter and select Accept after reviewing the packages. 6. Select Install Patch in the dialog box. This starts patch update for SP3 and requests for additional

CDs. Insert CDs as requested and select OK. 7. Select Finish to complete the SP3 update. 8. Exit the yast2 screen and remove the SP3 CD and reboot the node. When the system comes back

up, login and open up a terminal window. 9. The patch update does not load the kernel-source rpm. This has to be installed separately, as it is

required for QLogic driver installation later. To do the kernel-source RPM installation, mount the CD 1 of SP3, if not already mounted, and install the RPM as follows:

– #mount /media/cdrom For 32bit, use the following: – #rpm –Uvh /media/cdrom/i386/update/SUSE-CORE/9/rpm/i586/kernel-

source<kernel_version>.rpm – #rpm –Uvh /media/cdrom/i386/update/SUSE-CORE/9/rpm/i586/openmotif21-libs-32bit

<version_*.rpm> For 64bit, use the following: – #rpm –Uvh /media/cdrom/x86_64/update/SUSE-CORE/9/rpm/x86_64/kernel-

source<kernel_version>.rpm – #rpm –Uvh /media/cdrom/x86_64/update/SUSE-CORE/9/rpm/x86_64/openmotif21-libs-

32bit <version_*.rpm>

This installs the kernel source RPM and the openmotif21-libs-32bit RPMs.

13

3.2 Verify that OS packages and kernel versions are correct As user root, execute the following command:

# rpm -q binutils gcc gcc-c++ glibc gnome-libs libstdc++ libstdc++-devel make pdksh sysstat xscreensaver libaio openmotif21-libs gnome-libs-devel

The version numbers should be greater than or equal to following versions; if not, re-install those packages from the Service Pack CD.

binutils-2.15.90.0.1.1-32.5

gcc-3.3.3-43.24

gcc-c++-3.3.3-43.24

glibc-2.3.3-98.28

gnome-libs-1.4.1.7-671.1

libstdc++-3.3.3-43.24

libstdc++-devel-3.3.3-43.24

make-3.80-184.1

pdksh-5.2.14-780.1

sysstat-5.0.1-35.1

xscreensaver-4.16-2.6

openmotif21-libs

gnome-libs-devel-1.4.1.7

libaio-0.3.102-1.2

To verify the kernel version and architecture, use the following commands; the values should be as displayed below:

#uname –r

will list the supported kernel version; for SLES9 SP3. This should be 2.6.5-7.244.smp

#uname –p

x86_64 or i686

#uname -m

x86_64 or i686

#rpm –qa |grep kernel-source

kernel-source-2.6.5-7.244

This completes OS installation and verification. Repeat the OS installation on each of the nodes in the cluster before you proceed to the next step.

14

Important: Repeat the above OS Installation and verification on each of the nodes in the cluster for RAC or OAS nodes.

4 Prerequisites for installation of RAC or OAS This section describes the steps required to be done for RAC or OAS installation. This involves setting up network configuration, installing ProLiant Support Pack drivers, and configuring storage for RAC. Each step has a description, whether it has to be executed on all nodes or not. If you are doing RAC installation, then all the steps in this section have to be executed, and, for OAS only, steps 4.1 to 4.3 need to be executed.

4.1 Network configuration The networks are configured during OS installation. Verify that the network configuration on each of the nodes displays the public and private IP address.

As root, execute the following command:

#/sbin/ifconfig Make entries in the /etc/hosts files for public, private and virtual IP addresses for each of the nodes and verify that you are able to ping on both private networks from each node to every other node. For OAS installation, a single public IP address entry is sufficient.

A sample /etc/hosts of a 2 node RAC configuration is shown below.

127.0.0.1 localhost.localdomain localhost #Public IP addresses 100.100.100.1 racnode1 racnode1 100.100.100.2 racnode2 racnode2 #Private IP addresses 192.168.0.1 racnode1-priv racnode1-priv 192.168.0.2 racnode2-priv racnode2-priv #Virtual IP addresses 100.100.100.41 racnode1-vip racnode1-vip 100.100.100.42 racnode2-vip racnode2-vip

A sample /etc/hosts of a 2 node OAS configuration is shown below.

127.0.0.1 localhost.localdomain localhost #Public IP addresses 100.100.100.8 app1 app1 100.100.100.9 app2 app2

15

If any of the network configurations needs to be changed, then use the yast2 tool and select network devices and change the configuration as required.

If bonding needs to be configured, then refer to MetaLink note 291962,1 to set up bonding configuration on SLES9.

Important: Repeat the above network configuration steps in each of the nodes in the cluster.

4.2 Install ProLiant Support Pack on each node As root, execute the following commands:

1. Copy the HP ProLiant Support Pack (PSP) to the directory /tmp. #cp /media/cdrom/psp-<Version>.<Os_Type>.linux.en.tar.gz /tmp 2. Extract the PSP located in the /tmp directory. #cd /tmp

#tar –xzvf ./psp-<Version>.<Os_Type>.linux.en.tar.gz

3. Change to the PSP install directory. #cd ./compaq/csp/linux

4. Execute the install script in the ProLiant Support Pack. #./install<Version>.sh

where <Version> is the version number of the PSP release.

5. The install script will ask a question “Do you wish to continue”, answer Y and then press Enter. 6. This will bring up a GUI screen. 7. Locate the QLogic driver “HP StorageWorks FCA2214 FC HBA Driver for 2.x kernel.”

Right click on the above component and choose Configure component. This brings up a configuration screen. Select Failover mode to be “Single Path” or “QLogic Failover“ depending on whether or not you want redundancy. Select Save and OK to close the window.

8. Select the “HP ProLiant Smart Array Control (AMD64/EMT64) or (x86/AMD32) Driver for SuSE Enterprise Linux“, right click the mouse button and select the option “Do Not Install Component”.

Important: Please note that step 8 is important. Do not Install HP ProLiant Smart Array Control driver as this will change the kernel signature.

9. This completes all the drivers that are required for Oracle installation. Select Install. 10. Select Yes when asked to save parameters. This should start the installation. 11. Exit the PSP installations utility once the installation is complete. 12. Reboot the node to complete the PSP driver installation.

16

Important: Repeat the above steps in each of the nodes in the cluster.

4.3 Copy and execute script to set kernel parameters Create a directory for scripts – for example the “HPScripts” directory. The scripts are available along with the PDC kit download. Copy the set of scripts to a directory on each node of the cluster.

As root, execute the following commands:

#mkdir /tmp/HPScripts #cp /media/floppy/* /tmp/HPScripts #chmod +x /tmp/HPScripts/* As root, execute the script as follows.

#cd /tmp/HPScripts #./oracle_pre_req.sh

#sysctl –p

This will set the modified system parameters. Some of the parameters will take effect when the system is rebooted later during storage configuration for RAC. If you are doing OAS installation, reboot the nodes and proceed to OAS installation. Refer to Appendix A for details on scripts.

This is the end of OAS pre-requisites, proceed to Section 5 on Installing 10g Oracle OAS.

Important: Repeat the above setting-kernel-parameters step in each of the nodes in the cluster.

17

4.4 Configure shared storage [required for RAC installation only]

Note: If you have disconnected the fibre channel cables from the shared storage to the server, then reconnect them now to each of the nodes in the cluster.

The shared storage has to be configured at this point. To plan for the shared storage configuration, here are some tips. Oracle Clusterware requires 2 partitions for CRS files: namely voting disk and OCR registry. The Oracle database requires at least one partition for data files.

Based on your database size, carve out a LUN on the shared storage. The sample database created requires a partition of 10GB or so. In addition, also take note of the flash recovery area size if you plan on configuring flash recovery area and archive log.

Refer to the Oracle documentation Administrator's Guide for a detailed discussion on database creation and planning:

http://download-west.oracle.com/docs/cd/B14117_01/server.101/b10739/toc.htm

Example configurations:

Stripe and mirror required disk space and configure one single LUN on the shared storage. Carve out required sub-partitions for CRS and ASM using ASMLIB.

Example /dev/sda (40GB and above as required)

Sub partition /dev/sda1 – 200MB for CSS

Sub partition /dev/sda2 – 200MB for OCR

Sub-partition /dev/sda3 (remaining) for database on ASM using ASMLIB (If you need flash recovery area, then create another partition for the flash recovery area.)

You must have your shared storage planning done before carving out the LUNs.

At the end of this process, you should have the LUNs visible and accessible to the OS on each of the nodes in the cluster. This section describes, in detail, configuring HP shared storage, namely MSA1000, MSA1500, EVA3000/5000 and EVA4000/6000/8000 storage solutions.

For configuring MSA1000 or MSA1500 shared storage, you need the Modular Smart Array 1500cs CD or Modular Smart Array 1000 CD and documentation.

Refer to Appendix B to configure MSA1000 and MSA1500 shared storage configuration.

Refer to Appendix C to configure EVA3000/5000 and EVA4000/6000/8000 shared storage.

18

4.5 Create sub-partitions on the LUNs for Oracle

Note: After configuring shared storage, reboot the first node only and create sub-partitions for each LUN created. For example if you have created LUN /dev/sda, as in the example above, then create the sda1, sda2, and sda3 sub-partitions as shown below. This needs to be created only on the first node.

If there is more than one LUN created, such as /dev/sdb etc, then create a sub-partition /dev/sdb1 comprising the entire LUN. These sub-partitions are required best practice for Oracle RAC installation.

Create sub-partitions using fdisk. Refer to Appendix D for detailed fdisk command usage. Reboot the first node so that the sub-partitions are visible correctly. Reboot other nodes in the cluster so that each of the sub-partitions is visible to all the nodes in the cluster. When the system comes up, login to the system, open a terminal window, and verify if the partitions are shown correctly.

#cat /proc/partitions

This should list all the storage partitions.

4.6 Execute script to configure CRS partitions for LUN persistency As root, execute the following command:

#cd /tmp/HPScripts

Important: Execute this script on each node in the cluster.

Invoke the script and pass the two partitions: one for voting disk and the other for OCR.

#./HPConfigureUdevForCRS.sh /dev/sda1 /dev/sda2 This will create two raw devices and bind it to the udev symbolic link.

#raw -qa The above command will show the raw devices bound.

#ls –l /dev/hpdev The above command will list the udev links created.

19

4.7 Install and configure ASMLIB As root, execute the following commands:

Important: Execute all the steps below on the first node. On all other nodes, skip Step 4 to createdisk.

1. Copy the downloaded ASMLIB rpms to /tmp directory and install the RPMs as follows:

rpm -Uvh oracleasm-<kernel_version-asmlib_version.cpu_type>.rpm \ oracleasmlib-asmlib_<version.cpu_type>.rpm \ oracleasm-support-asmlib_<version.cpu_type>.rpm

2. To configure the ASMLIB on the first node, create the ASMDISK using the following command. # /etc/init.d/oracleasm configure

3. Configure the Oracle ASM library driver: This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ([ ]). Hitting ENTER without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface [ ]: oracle Default group to own the driver interface [ ]: dba Start Oracle ASM library driver on boot (y/n) [n]: y Fix permissions of Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: [ OK ] Creating /dev/oracleasm mount point: [ OK ] Loading module "oracleasm": [ OK ] Mounting ASMlib driver filesystem: [ OK ] Scanning system for ASM disks: [ OK ]

4. Execute the ASM following createdisk to add the partition as an ASM disk. Repeat this addition until you have added all the disks. Execute this command only on the first node in the cluster.

Example:

# /etc/init.d/oracleasm createdisk VOL1 /dev/sda3 Marking disk "/dev/sda3" as an ASM disk: [ OK ]

5. Execute the following on each of the nodes, including the first node: Verify the oracleasm is loaded

20

#/etc/init.d/oracleasm status The ASM should be loaded and /dev/oracleasm should be mounted 6. Scan the ASM disks. This has to be run on other nodes in the cluster. #/etc/init.d/oracleasm scandisks

7. List the ASM disk to verify, using the following command.

#/etc/init.d/oracleasm listdisks VOL1

4.8 Configure SSH As root, execute the following command:

# ulimit –u 16384 –n 65536

Set the password for Oracle that you want to use.

#passwd oracle

and then login to Oracle.

1. Login as oracle user.

#su - oracle

Generate ssh key.

#ssh-keygen –t dsa –b 1024 –P ““

Note that there is no space between the quotation marks after “P” above in the ssh-keygen command. Accept the default location and name for key.

2. Copy the id_dsa.pub to a file as <hostname>.pub where hostname indicates the name of the host on which the key is generated.

Example: if racnode1 is the hostname of the node

On racnode1,

#cp /home/oracle/.ssh/id_dsa.pub racnode1.pub

Repeat steps 1-2 above in each of the nodes, to create key and copy the id_dsa.pub onto its respective hostnames.pub.

3. Copy all the hostname.pub files to first node oracle home directory ( /home/oracle) using secure-copy as follows

#scp ./<hostname.pub> oracle@<first_hostname>:/home/oracle/

Accept RSA key and enter password when requested.

Example

21

#scp ./racnode2.pub oracle@racnode1:/home/oracle

4. Create an authorized_keys file by combining the public key of each of the nodes.

# cat /home/oracle/<hostname.pub> >> /home/oracle/.ssh/authorized_keys

Repeat the above command for each of the hostnames.pub, where hostname.pub represents the key for different nodes.

For example:

#cat /home/oracle/racnode1.pub >>/home/oracle/.ssh/authorized_keys

#cat /home/oracle/racnode2.pub >>/home/oracle/.ssh/authorized_keys

5. Now, copy the authorized_keys from the first node to each of the other nodes’ .ssh directory to complete the authorization. Accept RSA key and password when requested to complete the scp copy.

#scp /home/oracle/.ssh/authorized_keys oracle@<other_host_names>:/home/oracle/.ssh

6. To complete the ssh configuration, known_hosts list has to be created. This is done by initiating a ssh login to each node in the cluster including the local node and then accepting the login to add this host permanently to the list of known_hosts.

Example: To do this from each node, initiate a ssh login, say from racnode1 to itself and to racnode2:

#ssh racnode1

Enter yes and press enter to log in and then exit out of the shell just logged in.

# exit

Retry the ssh login again, and this time you should be able to log in without any question. Repeat for other nodes in the cluster.

#ssh racnode2

Repeat for racnode2 and for each node in the cluster.

Repeat this on private IP also

# ssh racnode1-priv

Enter yes and press enter to log in and then exit out of the just-logged-in ssh shell.

#exit

Repeat step 6 above to create known hosts on each node in the cluster by initiating a ssh login to every other node on public and private hostnames.

At this stage, you have configured your server to deploy Oracle. For RAC installation, skip section 5 and continue to section 6.

22

5 Installing 10g Oracle OAS using Oracle Universal Installer (OUI) If you are installing Real Application Cluster, then skip this section and continue to the next section, section 6.

5.1 Installing 10g Oracle Application Server Copy the Oracle OAS binary, which you have previously downloaded to the node on which you are installing OAS. Unzip and extract the files onto a local directory.

If the binary is in zip format, then unzip the binaries and extract using cpio command. If the download is at /home/oracle, then extract the binary and execute the following commands:

#cd /home/oracle #cpio –idmv < as_ linux_<10gVersion ..>.cpio

5.1.1 Installing Oracle Application Server

In a root terminal window,

#xhost +

# ulimit –u 16384 –n 65536

Login as Oracle :

#su – oracle

Set the ORACLE_BASE as follows:

#export ORACLE_BASE=/opt/oracle If the Oracle Application Server is being installed on x86_64 architecture, then the bash environment is set to 32-bit emulation. An entry is made in the /home/oracle/.bash_profile as shown below:

#linux32 bash

If not, execute the above command.

Invoke the OUI installer, as follows:

#/home/oracle/runInstaller Follow the steps to do the OAS with J2EE installation. The Installation document is available at http://otn.oracle.com. Some additional tips are described below.

• Enter the instance name and administration pwd in step1

• Select Install

• Select OK for all warning prompts.

5.1.2 Verification of the OAS installation

To verify the OAS installation use the following command from the app1:

23

#cd /opt/oracle/product/10.1.3/OracleAS_1/opmn/bin

#./opmnctl status

6 Installing 10g Oracle RAC using OUI

6.1 Installing 10g Oracle Real Application Cluster At this point, the HP PDC solution has completed the setup and configuration and validated that the requirements have been met for installing Oracle 10g Enterprise Database with Real Application Cluster software. Some important configuration that has been configured by the kit, which may be required for installing Oracle, is listed in the following table:

Configuration chosen during PDC installation Variables that are used during OUI

CRS on raw partitions

/dev/raw/raw1

/dev/raw/raw2

ASMLIB The disk alias name is the one you provided during the create disk, such as VOL1

ASM on RAW

The Oracle installer will pick all the raw devices. The raw –qa will list all the bindings to the shared storage partitions. The actual raw device configured is stored in /etc/sysconfig/rawdevices.

6.2 Extract Oracle binaries 1. These are in cpio.gz format. Copy the files to the first node of the server either from CD or NFS

server. Extract and unzip the Clusterware and database files as follows as oracle user. 2. Create a directory to stage:

#mkdir -p /opt/oracle/stage

#cd /opt/oracle/stage/

Download or copy the CRS CD to the above directory:

#gunzip <10201_clusterware_.cpio.gz> or unzip <10201_clusterware.zip>

#cpio –idmv < 10201_clusterware.cpio

You should see “clusterware” directory being created here for the CRS files.

3. Create another stage directory for database CDs and repeat the above procedure for database binaries. You will see “database” directory created for the database.

Next follow the documentation provided by Oracle to install CRS and Database. See below for key steps and other tips during CRS and Database installation.

24

6.3 Notes on using Oracle documentation This section details notes on following the Oracle OUI documentation for installation. Download the appropriate documentation for Oracle RAC installation.

For example, if you are installing Oracle 10g Release 2, the document part number refers to b14203.pdf part number B14203-05. This document is available at the otn.oracle.com under documentation for 10g Release 2.

Key sections with respect to Part III of the above document are as follows:

• For Clusterware installation: Section 4: Installing Oracle Clusterware, under the heading titled Installing Oracle Cluster using OUI

• For Oracle binary installation: Section 5: Installing Oracle Database 10g with Oracle Real Application Clusters

• For Database installation, using dbca: Section 6: Creating Oracle Real Application Clusters Database with Database Configuration Assistant

6.4 Verification of system readiness using CLUVFY Use Oracle Cluvfy to verify that all the pre-requisites are met before proceeding to Oracle installation. The documentation for executing the script is available at otn.oracle.com in the Oracle 10g installation document for Linux B14203-02.

If you have unzipped the Clusterware in the /opt/oracle/stage directory, then as root user:

#cd /opt/oracle/stage/clusterware/cluvfy

#unzip cvupack.zip

#cd /opt/oracle/stage/clusterware/rpm

#rpm –ivh cvuqdisk-<version>.rpm

#cd /opt/oracle/stage/clusterware/cluvfy

To verify, for example, if it is a cluster of two nodes and that racnode1 and racnode2 are the hostnames:

As oracle user:

#./runcluvfy.sh stage –pre crsinst –n racnode1, racnode2

#./runcluvfy.sh comp sys –n racnode1, racnode2 –p database

#./runcluvfy.sh comp nodereach –n racnode1, racnode2 –srcnode racnode1

To obtain help on various command options:

#./runcluvfy.sh –stage -help

25

#./runcluvfy.sh comp –help

Note: The cluvfy utility might fail on the following: compat-gcc-c++-, compat-libstdc++-, compat-libstdc++-devel, compat-gcc, openmotif-, and Setarch-

Refer to the Oracle document B14203-05 for troubleshooting the cluvfy errors. All the tests should pass before proceeding to Oracle installation. Once all tests pass, you can proceed to the installation using Oracle documentation.

6.5 Synchronizing time between the nodes It is necessary that the time difference between two nodes be less than 30 seconds, or else the installation will give an error during Oracle installation indicating that the time is in the future.

It is important to synchronize the time with an external NTP server. The following instructions detail how to synchronize between two nodes with the Universal Time clock.

Example: to sync between racnode1 and racnode2 with its UTC clock

On Racnode1:

1. Start yast2 from a terminal window. 2. Select Network Services; this displays a set of icons. 3. Scroll down and select NTP Client. This displays the NTP Client Configuration. 4. Select the radio button for When booting System. 5. Enter the local IP address (example 100.100.100.1) for the NTP server. 6. Select Finish. This adds the local node as the NTP server.

On Racnode2:

1. Start yast2 from a terminal window. 2. Select Network Services; this displays a set of icons. 3. Scroll down and select NTP Client. This displays the NTP Client Configuration. 4. Select the radio button for When booting System. 5. Under NTP_Server, select the Lookup button. This will scan for NTP server and should come back

with IP address of first node (100.100.100.1). 6. Select Finish. This adds the local node as the NTP Client.

Note: If the cluster consists of more than two nodes, repeat the procedure for node Racnode2 on other nodes.

To verify, use the following command from racnode1:

#date

#ssh racnode2 date

26

This displays the date and time on each of the nodes.

6.6 Installing Oracle Clusterware In a terminal window, type in as a root user:

#xhost +

Execute a rootpre.sh script before you start the installation.

#/opt/oracle/stage/clusterware/rootpre/rootpre.sh (Not required for 32bit)

Ignore the message displayed by the script, “No oracm is running and continue.”

# ulimit –u 16384 –n 65536

Login as Oracle user

#su – oracle

Set the ORACLE_BASE as:

#export ORACLE_BASE=/opt/oracle

The Clusterware default install location points to the ORACLE_HOME directory. The /opt/oracle is the Oracle mount point for Oracle binaries installation. As an example,

Set the ORACLE_HOME as follows for 10g R2:

#export ORACLE_HOME=/opt/oracle/product/10.2.0/crs

Invoke the runInstaller from the directory where unzipped Clusterware binaries were placed:

#/opt/oracle/stage/clusterware/runInstaller

This starts up the installer to install Clusterware.

Some key points are listed below as inputs for clusterware installation:

• Enter the appropriate hostname, private hostname and virtual IP name. These entries are already made in the /etc/hosts filename during the network configuration.

• If you have configured bonding, then instead of eth0 or eth1 for public and private networks, select the corresponding configured bond0 or bond1 for the networks.

• On the Specify Oracle Cluster Registry (OCR) location and Voting disk CSS location, choose external redundancy.

• For CRS configuration, the raw devices are mapped as follows: /dev/raw/raw2 for configuration registry (OCR)

/dev/raw/raw1 for voting disk. (CSS)

Refer to Oracle documentation for description of each OUI screen and steps. Some additional tips are described below:

• This will start up the database configuration assistant. • Select Next in step 1 of OUI. • Select Defaults in step 2, 3 and 4 of the OUI. • Add Node names in step 5 of OUI. • Select External redundancy in step 6 of OUI.

27

• Give the CSS path as: /dev/raw/raw1 and click Next • Select External redundancy in step 7 of OUI • Give the OCR path as: /dev/raw/raw2 and click Next • Select Install

6.7 Installing ASM binaries In Oracle 10g R2, the recommended installation is to have two Oracle homes: one for ASM, and the other for database binaries.

Continue as user oracle, and install ASM:

Set the Oracle home for ASM:

#export ORACLE_HOME=/opt/oracle/product/10.2.0/db_1

Invoke the runInstaller from the directory where you have unzipped the database binaries.

#/opt/oracle/stage/database/runInstaller

Refer to Oracle documentation for a detailed description of OUI steps and screens. Some additional tips for installation are provided below.

• Select the Enterprise Installation option. • Accept the ORACLE_HOME shown. • Select all the RAC cluster nodes with cluster installation. • Choose Configure Automatic Storage Management(ASM) on the “Select Configuration option screen

of the OUI” and enter password. • Enter Disk group name and select external for the redundancy option. Select the ASM disks to

install the binaries. • This will install the binaries for ASM and will prompt the user to execute root.sh on each of the

nodes. • This completes install of ASM binaries and ASM instance configuration.

After the OUI install is completed for ASM, follow the steps in the next sections to install database software and configure database.

6.8 Install Oracle database binaries Set the Oracle home for database binaries:

#export ORACLE_HOME=/opt/oracle/product/10.2.0/db_2

Invoke the runInstaller from the directory where you have unzipped the database binaries.

#/opt/oracle/stage/database/runInstaller This invokes the OUI GUI.

• Select Enterprise Installation option. • Accept the ORACLE_HOME shown. • Select all the RAC cluster nodes with cluster installation. • Select the Install database Software only option. • This will start installation of database binaries in the database home specified.

28

• If prompted to overwrite dbhome, answer “Yes”

Refer to Oracle documentation for a detailed description of OUI steps and screen.

6.9 Creating the Oracle RAC Database using DBCA

The database can be created separately once the binaries are installed using the DBCA utility.

Login as oracle user:

#su - oracle

As oracle user, set the following variables to the correct values; for example:

#export ORACLE_HOME=/opt/oracle/product/10.2.0/db_2 #export PATH=$PATH:$ORACLE_HOME/bin

Invoke the DBCA utility as follows:

#dbca

Follow the Oracle documentation for description of OUI screens using dbca. Some additional tips are described below:

• This will start up the database configuration assistant. • Select RAC. • Select Create Database in step 1 of OUI. • Select All nodes in step 2 of OUI. • Select General Purpose in step 3 of OUI. • Enter Global Database Name and SID Prefix in step 4 of OUI. • Select Configure db with Enterprise Manager in step 5 of OUI. • Select Automatic Storage Management (ASM) in step 7 of OUI. • During the installation of the Oracle 10g software, the user may choose to install a sample

database, which is recommended. • If you see error “ORA-27123 :Unable to attach shared memory,” then you have allocated more

memory than 2.7GB. Reduce the SGA memory, note metalink :307323.1, and create the database.

If this is done, then the Oracle 10g RAC cluster database instances can be verified as running by following the steps in the next section “Verification of the sample database.”

6.10 Verification of the sample database

To verify that the sample database was created and is running:

1. Enter the following commands on all the nodes of the cluster:

#su – oracle

Set the ORACLE_SID and ORACLE_HOME values, for example. Note that the exact path and SID value will vary depending on your installation. As an example,

#export ORACLE_HOME= /opt/oracle/10g/product/10.2.0/db_2

#export ORACLE_SID= orcl1

29

Note: ORACLE_SID is the database ID you entered during installation.

#export PATH=$ORACLE_HOME/bin:$PATH;

Type the following at the prompt:

#sqlplus /nolog

At the SQL> prompt, enter:

#connect / as sysdba

2. When connected, enter:

select * from v$thread;

This command displays all the threads in the database. You should see as many lines as the number of nodes in your cluster. Be sure the status of each thread is OPEN.

3. Enter the following command:

select * from v$active_instances;

This command displays all the active instances in the database. You should see as many lines as the number of nodes in your cluster.

4. Enter the following command to exit the SQL prompt:

SQL> exit;

The installation is complete.

30

Appendix A: PDC Scripts Script : oracle_pre_req.sh

This is a standalone script which sets oracle-required kernel parameters and creates oracle user.

Script : HPConfigureUdevForCrs.sh

Requires the following files to be present in the same directory

1. hp_gen_rules.sh 2. config_udev_for_storage 3. disk_test_response 4. udev.conf

Takes input of 2 partitions – one for CSS voting disk and other for registry.

31

Appendix B: Configuring MSA1000/1500 shared storage

Note: You should use the instructions contained in this Appendix if you are utilizing MSA1000/1500 shared storage. Procedures for configuring these storage subsystems are similar.

IMPORTANT: If you are implementing a redundant path to your MSA1000/1500 shared storage, ensure that only one controller is connected during these procedures.

Use the cpqacuxe utility to configure your MSA1000/1500 shared storage. This utility is part of the PSP installation in the SLES9 environment and could be run as follows:

1. cp cpqacuxe-<version>.linux.rpm.tar.gz /tmp

2. cd /tmp

3. tar –xzvf cpqacuxe <version>.linux.rpm.tar.gz

4. rpm –ivh cpqacuxe <version>.linux.rpm

5. cd /usr/sbin

6. cpqacuxe –R

The message Remote connection enabled! is displayed. Follow this procedure:

1. Open an Internet browser and type the following in the address field: https://127.0.0.1:2381 2. When the server authentication message appears, click OK to continue. 3. Accept the certificate for Current Sessions Only. 4. Log in using your credentials for the host operating system. 5. Click OK in response to the browser message, and Yes to continue loading. This opens the

System Home Management page. 6. Select the HP Array Configuration Utility to display the ACU main screen with a list of

available storage controllers. 7. Configure shared storage according to the steps below.

32

Configuring MSA1000/1500 storage 1. Select MSA1000 or MSA1500 Controller in the Select Controller pane. 2. Select the controller displayed in the Configuration View pane. 3. Clear any pre-existing array configuration by performing the following:

Note: Clearing the configuration will remove all data from the shared storage. If this installation is performed on an existing configuration, then do not perform this step. This is only for the first time installation of the cluster.

a. Under Common Tasks, click Clear Configuration to display a confirmation window. b. Click OK. c. Click Save, displaying another confirmation window. d. Click OK.

4. Under Common Tasks, click Create Array to display a list of available drives. 5. Select all the disks and click OK. 6. Select the newly-created array and click Create Logical Drive under Common Tasks. 7. In the Create Logical Drive window, enter the following values:

Fault Tolerance RAID 1+0

Stripe Size 128 KB

Size <size of partition to be configured> MB

Max Boot Disable

Array Accelerator Enable

8. Click OK. 9. Repeat the steps above to configure any other planned LUNs with their appropriate sizes. 10. Click Save to display a confirmation window. 11. Click OK to save the configuration.

Configuring Selective Storage Presentation 1. Select the MSA1000 Controller icon from the Configuration View pane. 2. Click Selective Storage Presentation from the Common Tasks window. 3. Select Enable and click OK. 4. Replace the word Unknown in the Connection Name column with a Friendly connection name

for all connections associated with the WWPN of the HBA ports in the servers. 5. Select Linux from the Host Mode drop box. 6. Select all the logical drives for each adapter ID that needs access to this server. 7. Click OK. A warning window is displayed. Select OK. 8. Click Save. A confirmation window is displayed. Select OK to save the configuration. 9. Click Exit ACU to close ACU.

33

10. Press Alt F4. The browser with the open HP Storage Home Management window closes. 11. Type in cpqacuxe –stop at the root prompt and press Enter.

IMPORTANT: If you disconnected a redundant MSA controller at the beginning of this procedure, you should now reconnect it.

34

Appendix C: Configuring the HP Enterprise Virtual Array EVA4000, EVA6000 and EVA8000

Pre-configuration In order to present LUNs from an HP StorageWorks Enterprise Virtual Array (EVA) storage subsystem to a host server, there must be a host profile for that server. To support the cluster described in this paper, you must create host profiles for every node, allowing the LUNs used for shared storage to be presented to each node.

To create a host profile for a server, you will need to obtain the following information:

• Hostname • IP address • HBA World Wide Port Names (WWPNs)

From each node in the cluster, perform the following actions to collect the required information:

1. Login to the node as root user and open a terminal window. 2. To obtain the server’s hostname enter the following command: #hostname

3. To obtain the HBA WWPN, enter the following command: #cat /proc/scsi/qla2xxx/0 | grep scsi-qla0

An output similar to the following is displayed:

scsi-qla0-adapter-node=200000e008b09eded

scsi-qla0-adapter-port=210000e008b09eded

scsi-qla0-adapter-target-0=50001fe10013a129

The value of scsi-qla0-adapter-port is the WWPN of the HBA.

4. If the cluster is configured with redundant HBAs, obtain the WWPN for the second HBA by entering the following command:

#cat /proc/scsi/qla2xxx/1 | grep scsi-qla1

Installation For instructions on initializing and setting up an EVA storage subsystem, refer to the documentation you received with this product.

For instructions on installing and using HP StorageWorks Command View EVA, refer to the appropriate guide.

The following sections provide instructions for opening Command View EVA, configuring EVA host profiles, and configuring and presenting LUNs.

Opening Command View EVA On the node where Command View EVA is installed, open a browser and type the following in the address field:

https://localhost:2381

The System Management Homepage opens. Select the appropriate link to open Command View EVA.

35

Configuring EVA host profiles In order to present EVA LUNs to the nodes in the cluster, an EVA host profile must be created for every node. To create an EVA host profile, perform the following steps from one of the cluster nodes:

1. Expand the sub-tree for the EVA subsystem that will be used by the cluster and click the Hosts folder. The Host Folder Properties page for the Hosts folder is displayed in the right pane.

2. Click the Create folder button in the content pane. The Create a Folder page is displayed. 3. Enter a name for the new folder to hold the host profiles for the cluster in the text field in Step 1:

Enter a name. 4. Enter any comments in the text box in Step 2: Enter comments. 5. Click the Finish button. The Operation succeeded page will appear in the content pane. Click OK. 6. The Host folder properties page for the folder you just created is displayed in the right pane. Click

the Add host button. The Add a Host page is displayed in the right pane. 7. Enter the hostname of the first node gathered in the “Preconfiguration” stage of this Appendix into

the Host name text field. 8. Select the WWPN of one of the HBAs installed in the first node from the Port WW Name

dropdown list. If the server is configured with redundant HBAs, the other HBA will be entered later. For an EVA3000/5000, select Sun Solaris from the Host OS dropdown list. For an EVA4000/6000/8000, select Custom and enter the following 16 hexadecimal characters in the Custom node number field:

00000020220008AA

9. Click the Add Host button. The Operation succeeded page will appear. Click OK. 10. The Host Properties page for the host profile is displayed in the right pane. If the server is

configured with redundant HBAs, click on the Ports tab and perform next step, otherwise skip to the last step of this procedure.

11. Click the Add port button. Select the WWPN of the second HBA from the dropdown list and click the Add port button. The Operation succeeded page is displayed in the content pane. Click OK.

12. Click on the host folder you created in Steps 3 – 6 of this procedure and repeat the steps above to configure hosts for remaining nodes in the cluster.

Configuring and presenting LUNs After the host profiles for each node in the cluster have been created, you can create the virtual disks that will be presented to the nodes in the cluster.

1. Expand the sub-tree for the EVA subsystem that will be used by the cluster and click the folder named Virtual Disks. The Virtual Disks Folder Properties page for the Virtual Disks folder is displayed in the right panel.

2. Click the Create folder button in the right pane. The Create a Folder page is displayed. 3. Enter a name for the new folder in the text field in Step 1: Enter a Name. 4. Enter any comments in the text box in Step 2: Enter Comments. 5. Click the Finish button. The Operation succeeded page will appear in the right pane. Click OK. 6. The Virtual Disks Folder Properties page for the folder is displayed in the right pane. Click the

Create Vdisk button. The Create a Vdisk Family page is displayed in the right page. 7. Enter the desired Vdisk name in the Vdisk name text field. 8. Select the desired disk group from the Disk group name dropdown list. The disks can be created in

any disk group so long as there is sufficient capacity available in that disk group.

36

9. Select the Vraid1 option in the Redundancy section. 10. Enter <size_of_partition>GB partition size you want to configure in the Size text field. 11. Click the Finish button. The Operation succeeded page will appear in the right pane. Click the

Create more Vdisks button. 12. Repeat Steps 7 – 11 to create more Vdisks to configure all LUNs planned for the shared storage. 13. Expand the Vdisk folder you created in Steps 2 – 5 of this procedure. 14. Expand the first Vdisk you created in Steps 7 – 11 of this procedure. Click the Active icon in the

tree. 15. The Vdisk Active Member Properties page is displayed in the content pane. Click the Presentation

tab. 16. Click the Present button. The Present Vdisk page is displayed in the content pane. 17. Select the host folder you created in the “Configuring EVA host profiles” stage of this Appendix. 18. Click the Present Vdisk button. The Operation succeeded page is displayed in the content pane.

Click OK. 19. Repeat Steps 15 – 18 for the other Vdisks you created in this procedure. After the Vdisks have successfully been created and presented, the EVA storage is configured and is ready for usage by the Oracle installation process.

37

Appendix D: fdisk partitioning fdisk /dev/sda The number of cylinders for this disk is set to 1274. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): p Disk /dev/sda: 10.4 GB, 10486579200 bytes 255 heads, 63 sectors/track, 1274 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System Command (m for help): n Command action e extended p p primary partition (1-4)

Partition number (1-4): 1 First cylinder (1-1274, default 1): <Press Enter> Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-1274, default 1274): +200M Command (m for help): n Command action e extended p p primary partition (1-4)

Partition number (1-4): 2 First cylinder (26-1274, default 26): <Press Enter> Using default value 26 Last cylinder or +size or +sizeM or +sizeK (26-1274, default 1274): +200M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (51-1274, default 51): <Press Enter> Using default value 51 Last cylinder or +size or +sizeM or +sizeK (51-1274, default 1274): <Press Enter> Using default value 1274 Command (m for help): p Disk /dev/sda: 10.4 GB, 10486579200 bytes 255 heads, 63 sectors/track, 1274 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

38

Device Boot Start End Blocks Id System /dev/sda1 1 25 200781 83 Linux /dev/sda2 26 50 200812+ 83 Linux /dev/sda3 51 1274 9831780 83 Linux Command (m for help):w <Press Enter>

The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.

39

For more information HP Parallel Database Cluster: http://h18026.www1.hp.com/solutions/enterprise/highavailability/oracle/index.html

HP.com - ProLiant High Availability, http://h18022.www1.hp.com/solutions/enterprise/highavailability/index.html

HP StorageWorks enterprise storage, http://h18006.www1.hp.com/storage/enterprisestorage.html

Oracle 10g downloads, http://www.oracle.com/technology/documentation/database10g.html

The base OS installation is also detailed in the following link at OTN, and this document adds some partitioning details. http://www.oracle.com/technology/pub/articles/smiley_10gdb_install.html#sles9

To help us improve our documents, please provide feedback at www.hp.com/solutions/feedback

© 2006 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

AMD is a trademark of Advanced Micro Devices, Inc. Intel is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States and other countries. Java is a US trademark of Sun Microsystems, Inc. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. Oracle is a registered US trademark of Oracle Corporation, Redwood City, California.

4AA1-0086ENW, Rev. 2, December 2006