Setting up IBM Power Systems 10 Gigabit Ethernet ports and ... · PDF fileSetting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity

© Copyright IBM Corporation, 2010. All Rights Reserved. All trademarks or registered trademarks mentioned herein are the property of their respective holders

. . . . . . . .

Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1

EtherChannel for Oracle RAC private interconnectivity

Ravisankar Shanmugam IBM Oracle International Competency Center

August 2010

Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity http://www.ibm.com/support/techdocs © Copyright 2010, IBM Corporation

Table of contents Abstract........................................................................................................................................1 Prerequisites ...............................................................................................................................1 Introduction .................................................................................................................................1 10 Gigabit Ethernet in Power System servers..........................................................................1 AIX EtherChannel on 10 Gigabit Ethernet interfaces ..............................................................2 Setting up Oracle 11g Database RAC nodes with 10 Gigabit Ethernet ports and switches 2 Software version used for the test ................................................................................................................ 3 Set up and configuration of the 10 Gigabit Ethernet switch...................................................3 Assigning HEA ports to the Oracle RAC nodes in AIX LPARs using the HMC.....................5 Creating the EtherChannel interface in AIX 6.1......................................................................11 Jumbo frames ............................................................................................................................................. 14 Summary....................................................................................................................................20 Appendix A ................................................................................................................................21 About the author .......................................................................................................................22 Trademarks and special notices..............................................................................................23

Setting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity http://www.ibm.com/support/techdocs © Copyright 2010, IBM Corporation 1

Abstract The purpose of this white paper is to document the setup and configuration of IBM Power Systems™ 10 Gigabit Ethernet ports. The environment used for the tests documented in this paper consist of the Oracle Database 11g with Real Application Clusters (RAC) software to configure nodes with private network intercommunication in the AIX® operating system environment. Tests were run on a POWER6™ processor-based Power® 570 server. The 10 Gigabit Ethernet cards are an option for the Power 570.

This document additionally covers the setup and configuration of AIX EtherChannel for Oracle RAC interconnectivity and the Ethernet switches.

This document does not cover the installation of the Virtual IO server (VIOS), Logical Partition (LPAR) creation, installation of the AIX operating system and various Oracle RAC components. Refer the Appendix A of this paper for document links covering these specific topics.

Prerequisites Knowledge of AIX administration, Virtual IO Server and Power Systems POWER6 features. Knowledge of configuring Force10 S2410 switch. Knowledge of Oracle RAC concepts.

Introduction Oracle RAC is a “shared everything” database architecture with two or more nodes clustered together with shared storage. These nodes are connected together with a high speed interconnect that facilitates fast communication between the nodes. The nodes use this interconnect at various times, such as exchanging information during database startup, resolution of data lock situations, exchanging cluster node information, exchanging transaction information and exchanging data between the Oracle RAC nodes.

While the database transaction load spreads across multiple Oracle RAC nodes, the Oracle Cache Fusion architecture can take advantage of a high speed network with high bandwidth and low latency to move requests from one Oracle RAC node to another. When a high number of Oracle RAC nodes are created and a high number of database blocks are exchanged between nodes, a high speed, high bandwidth, low latency private network is beneficial. The 10 Gigabit Ethernet option gives the ideal solution for high bandwidth and low latency needs in a clustered environment.

10 Gigabit Ethernet in Power System servers The POWER6 processor-based Power 570 server is a mid range server with various advanced hardware components and features including an optional dual port 10 Gigabit Ethernet card per building block for a high speed network and high bandwidth.

For more information about the Power 570 server, refer the Redbook: IBM Power Systems p570 Technical Overview and Introduction.


AIX EtherChannel on 10 Gigabit Ethernet interfaces IBM POWER6 processor-based servers and above offer Integrated Virtual Ethernet adapters (IVE), which give integrated high-speed Ethernet adapter ports (Host Ethernet Adapter (HEA)) with hardware assisted virtualization capabilities.

The POWER6 570 servers used for the Oracle RAC setup in this testing use the 10 Gigabit dual port Ethernet cards for Oracle cluster node interconnectivity.

The Host Ethernet Adapter can be used for external Ethernet connectivity for LPARs using dedicated ports without the need of a Virtual IO Server. The HEA adapter directly connects to the internal GX+ bus of POWER6 processor-based server instead of connecting to the PCIe or PCI-X bus.

IVE adapters also include special hardware features that provide the logical Ethernet adapter, otherwise called Logical Host Ethernet Adapter (LHEA). These LHEA adapters can be directly assigned to the LPARs without configuring through the POWER Hypervisor (PHYP). This eliminates the need to move the packets between LPARs through the Shared Ethernet Adapter (SEA). IVE replaces the need of virtual Ethernet and SEA in a Virtual IO Server (VIOS) environment and LPARs can share the HEA ports with improved performance.

Each physical port in HEA can provide 16 logical ports. The setup mentioned in this document uses logical ports from the HEA and assigns them to the LPARs. Refer to Figure 1 in this document for more details. From the operating system standpoint in the LPAR, a logical port appears as a standard physical Ethernet port with its attribute values.

The LHEA ports can be used for setting up an EtherChannel in AIX. EtherChannel is port link aggregation for grouping one or more Ethernet links with backup ports to create a single logical port for the purpose of providing high availability and load balancing to the connection between servers, switches and routers.

In this document, the EtherChannel port in the LPAR is used for the Oracle RAC private network connection.

For more information on the IVE, HEA and LHEA, refer the Redbook: Integrated Virtual Ethernet Adapter - Technical Overview and Introduction

Setting up Oracle 11g RAC nodes with 10 Gigabit Ethernet ports and switches Figure 1 shows the four node Oracle RAC cluster with 10 Gigabit Ethernet interfaces for interconnectivity.


Figure 1 – Oracle RAC node interconnectivity diagram

The Oracle RAC nodes RAC_1 and RAC_2 share 2 x 10 Gigabit Ethernet cards. For high availability and failover purposes, one card is always active and shared for EtherChannels and other card is in standby. The Oracle RAC nodes RAC_3 and RAC_4 are setup with 2 x 10 Gigabit Ethernet cards. Only one port is used from each card and the other port is left unused for testing purposes. There are 2 logical ports (LHEA) used from each physical port (HEA). The 10 Gigabit Ethernet switches are connected to each other. All of the 10 Gigabit Ethernet connections from the AIX server to the switches and between the switches are made using Fibre Channel cables (LC-LC type).

Software version used for the test

AIX 6.1 TL4 SP03

Oracle 11g Database with RAC version 11.1.0.7

Set up and configuration of the 10 Gigabit Ethernet switch For initial basic setup of the switch, refer to the following documents: Installing the S2410 system and S2410 Quick reference from the Force10 CD or the Force10networks web site.


After finishing the basic setup of the switch, such as network setup to access the switch, activate the ‘telnet’ and the ‘web interface’ to access it remotely.

To activate the telnet and Web access, use the following commands:

Force10 S2410 # enable Password: Force10 S2410 # config Force10 S2410 # ip telnet server enable Force10 S2410 # ip http server enable

In a brand new Force10 switch, all of the ports are in a disabled mode by default. The ports should be enabled by using either the command line or Web GUI interface.

The following command enables all of the ports in a switch. Enable the ports on the backup switch.

Force10 S2410 # no shutdown all Force10 S2410 #show port all Admin Physical Physical Link Link LACP Interface Type Mode Mode Status Status Trap Mode ---------- ------ ------- ---------- ---------- ------ ------- ------- 0/1 Enable 10G Full Down Enable Enable 0/2 Enable 10G Full Down Enable Enable 0/3 Enable 10G Full Down Enable Enable 0/4 Enable 10G Full Down Enable Enable 0/5 Enable 10G Full Down Enable Enable 0/6 Enable 10G Full Down Enable Enable 0/7 Enable 10G Full Down Enable Enable <Remaining output is eliminated>

The private network setup for the Oracle11g Database with RAC mentioned in Figure 1 of this document is created as follows. The following output shows the switch ports status on the switch.

Force10 S2410 #show port all Admin Physical Physical Link Link LACP Interface Type Mode Mode Status Status Trap Mode ---------- ------ ------- ---------- ---------- ------ ------- ------- 0/1 Enable 10G Full 10G Full Up Enable Enable 0/2 Enable 10G Full 10G Full Up Enable Enable 0/3 Enable 10G Full 10G Full Up Enable Enable 0/4 Enable 10G Full 10G Full Up Enable Enable 0/5 Enable 10G Full Down Enable Enable 0/6 Enable 10G Full Down Enable Enable 0/7 Enable 10G Full Down Enable Enable 0/8 Enable 10G Full Down Enable Enable 0/9 Enable 10G Full Down Enable Enable 0/10 Enable 10G Full Down Enable Enable <Remaining output is eliminated>

Set the “Flow Control” to enabled in the switch. After logging into the switch,

Force10 S2410 #enable Password: Force10 S2410 (Config)#storm-control flowcontrol Force10 S2410 (Config)#exit Force10 S2410 # write This operation may take a few minutes. Management interfaces will not be available during this time.


Are you sure you want to save? (y/n) y Configuration Saved!

Assigning HEA ports to the Oracle RAC nodes in AIX LPARs using the HMC The Host Ethernet Adapter (HEA) ports of the Power 570 server are shown in the Hardware Management Console (HMC)

The following screen shot shows the list of 10 Gigabit ports in the server. Each port can be configured for different purposes. For example, a port can be assigned to an LPAR as dedicated (promiscuous) or shared.


Select a port (port1) in the list and press “configure” button.

If the LPAR is selected in the “Promiscuous LPAR” list, the port is used only for that LPAR. By default, the ports are assigned to the system LPAR as dedicated (Promiscuous LPAR). In this server, by default the system LPAR “10-68F72” was Promiscuous LPAR for all of the 10 Gigabit Ethernet ports.

For the Oracle Database with RAC setup, the ports were released from the Promiscuous LPAR and selected to “None”, so that the ports can be shared with other LPARs.

Select “Flow Control Enabled” for each port.

Repeat the same assignment “None” to all of the HEA ports as needed. Setting up Jumbo Fames in this configuration can be seen in a later part of this paper.


To assign the logical port for the LPARs, select the LPAR and go to Task Configuration Manage Profiles.

Select the Profile and “Actions” Edit


The following window shows the list of Host Ethernet Adapters (Physical).

There are 2 x 2 ports Physical adapters are in the Power 570 machine. Select the appropriate port which is physically connected to the 10 Gigabit Ethernet switch.

For this setup, the port “DQD64A9-P1-C10-T1” and “DQD64A6-P1-C10-T1” are used for the LPAR “AIX_RAC1_209”.


Select “DQD64A9-P1-C10-T1” port and press “Configure” button

The following window shows the LHEA ports. Select the logical port1 and press OK and this port will be assigned to the LPAR “AIX_RAC1_209”.


Select the second port (backup port) for the LPAR “AIX_RAC1_209” and click “Configure”.

Select the logical port from the list.


Repeat the above procedures for other LPARs in the RAC configuration.

Shutdown (“shutdown –Fr”) the LPARs and bring them back online.

After the LPARs come online, check the list of Logical Host Ethernet Adapter (LHEA) ports (lhea0 and hea1) assigned to the LPAR as shown below.

The interfaces “ent1” and ‘ent2” are the corresponding interfaces for “lhea0” and “lhea1”.

# lsdev -Cc adapter ent0 Available Virtual I/O Ethernet Adapter (l-lan) ent1 Available Logical Host Ethernet Port (lp-hea) ent2 Available Logical Host Ethernet Port (lp-hea) fcs0 Available 00-08 FC Adapter fcs1 Available 00-09 FC Adapter lhea0 Available Logical Host Ethernet Adapter (l-hea) lhea1 Available Logical Host Ethernet Adapter (l-hea) vsa0 Available LPAR Virtual Serial Adapter vscsi0 Available Virtual SCSI Client Adapter vscsi1 Available Virtual SCSI Client Adapter

For verifying the mapping of the adapter name “ent1” displayed in AIX for the assigned port, execute the command “lscfg –vl ent1” in the AIX LPAR.

# lscfg -vl ent1 ent1 U789D.001.DQD64A9-P1-C10-T1 Logical Host Ethernet Port (lp-hea) IBM Host Ethernet Adapter: Network Address.............001A64283FB0

In the above output of the interface ent1, “U789D.001.DQD64A9-P1-C10-T1” is the physical adapter name and location in the Power 570 server.

Creating the EtherChannel interface in AIX 6.1 EhterChannel is a network port aggregation technique that aggregates more than one Ethernet adapter together to form a single logical Ethernet interface. Then the logical interface is configured with an IP address for communications. The Ethernet adapters which are part of the EtherChannel should be connected to the same switch.

EtherChannel can be used for load balancing and high availability of the network connections. This provides fault-tolerant high speed communications between servers and switches.

The participating Ethernet interfaces for the EtherChannel should not be configured with IP addresses and should be in “detach” mode before adding to them to the EtherChannel.

Use this command to detach the participating Ethernet interfaces.

chdev –l en3 –a state=detach ## where en3 is the Etherchannel port.

In AIX, EtherChannel provides a backup port option to connect an Ethernet interface to another switch if the primary switch or the primary EtherChannel ports are down or faulty.


The setup in this document used one primary port in the EtherChannel and a backup port for high availability. The backup port is in standby mode while the primary port is online.

Refer the Figure 1 for the detailed network connection of the Oracle RAC nodes EtherChannel setup.

The EtherChannel port mentioned in this document is created with one primary Ethernet port (ent1) and a backup port (ent2). The EtherChannel port (ent3) is created using ent1 and ent2 interfaces. Do not configure IP or set any attributes for the participating candidate Ethernet ports (ent1 and ent2). If you have them configured previously, reset the IP address and reset the previously assigned attributes.

The following procedure explains the creation of the EtherChannel interface using the “smitty” tool: # smitty etherchannel.

Select “Add an EtherChannel / Link Aggregation” in the list.


Select the primary Ethernet ports for the EtherChannel. Do not select the backup port now.

After selecting the primary Ethernet adapters, the following page is displayed. Select the Backup port from the list now. For “Internet Address to Ping”, enter the pingable private IP address, for example the other private IP address of other Oracle RAC nodes. Enter “3” for “Number of Retries” and “1” for the “Retry Timeout”.


Press “Enter” after filling in the screen above. It creates a new interface “ent3” for EtherChannel.

# lsdev -Cc adapter ent0 Available Virtual I/O Ethernet Adapter (l-lan) ent1 Available Logical Host Ethernet Port (lp-hea) ent2 Available Logical Host Ethernet Port (lp-hea) ent3 Available EtherChannel / IEEE 802.3ad Link Aggregation fcs0 Available 00-08 FC Adapter fcs1 Available 00-09 FC Adapter lhea0 Available Logical Host Ethernet Adapter (l-hea) lhea1 Available Logical Host Ethernet Adapter (l-hea)

The newly created EtherChannel interface ent3 can be configured with the IP address using: smitty mktcpip en3 <Setup the IP address>.

Repeat the above procedures to the remaining Oracle RAC nodes.

For more information on the AIX EtherChannel, refer the AIX 6.1 document: Etherchannel and IEEE 802.3ad Link Aggregation

Make sure the EtherChannel interface is properly setup for the Oracle RAC nodes and that they can communicate with each other by using “ping” command. Also, a small test would confirm the basic capacity of the network bandwidth between the Oracle RAC nodes.

The Oracle RAC node1 is configured with the private network IP address 10.0.0.1 on the EtherChannel interface “en3” and the Oracle RAC node2 is configured with 10.0.0.2 as same.

The test can be done by using “ftp” command as follows: from Oracle RAC node1, as a “root” user do the “ftp” and monitor the network traffic using the “nmon” tool.

The private IP address 10.0.0.,2 belongs to another Oracle RAC node’s 10 Gigabit Ethernet port

# ftp 10.0.0.2 Connected to 10.0.0.2. 220 dyn9038158218 FTP server (Version 4.2 Wed Oct 1 09:40:05 CDT 2008) ready. Name (10.0.0.2:root): 331 Password required for root. Password: 230-Last login: Wed Apr 8 19:35:59 2009 on /dev/pts/1 from sig-9-65-45-188.mts.ibm.com 230 User root logged in. ftp> put "|dd if=/dev/zero bs=1M count=200000" /dev/null 200 PORT command successful. 150 Opening data connection for /dev/null.

The output of network interfaces can be seen by using ‘nmon” tool.

Jumbo frames

Setting up jumbo frames on the private network interface for the Oracle RAC nodes would give an advantage for sending and receiving large size data frames between the nodes. This could better utilize the available bandwidth of the 10 Gigabit Ethernet connections between the Oracle RAC nodes.

Enabling jumbo frames on the network interfaces or EtherChannel interface requires certain steps to be successful in an Oracle RAC environment.


The general rule is to enable the jumbo frames option on both the network interface in the host side as well as the port in the network switch and the port(s) of the switch interconnection.

Before enabling Jumbo Frames, bring down the CRS and DB resources and “detach” the EtherChannel interfaces in the Oracle RAC nodes.

As a root user, issue the following commands.

#$CRS_HOME/bin/crsctl stop crs #ifconfig en3 down detach

In the network setup mentioned in this document for HEA ports the jumbo frames option was set to “Yes” on the EtherChannel interface “ent3”.

Use the “smitty” tool, #smitty etherchannel then select “Change / Show Characteristics of an EtherChannel / Link Aggregation” for “ent3” to verify that the jumbo frames setting is set to “yes”.


To enable jumbo frames on the physical ports in the Power 570 server, use HMC menus as follows. In HMC, select “Servers” in left side of the page, select the server listed on the right side where the Oracle RAC node is created, click on (>>) pop up menu of the server, navigate to Hardware Information >Adapters>Host Ethernet. This will bring up a pop-up window.

Select the HEA port used for the Oracle RAC interconnectivity as well as the EtherChannel. Click the “Configure” button and the following window will be popped up, select “9000 jumbo frame” option in the list. Select “Flow control enabled” to keep the flow control enabled and select “OK”. Repeat the same procedure on all of the ports which are involved in the Oracle RAC interconnectivity.


After setting up the jumbo frames option on the host and the HEA ports as mentioned above, enable jumbo frames on the network switch ports.

In the Force10 S2410 XFP switch, enabling jumbo frames on the ports can be done either with the Command line or Web GUI.


Using web GUI, login to the switch and enable the Jumbo frame as follows:

Enable jumbo frames by editing the field “maximum frame size” to “9022” and click the “Submit” button. Follow the same procedure for all of the ports involved in the Oracle RAC interconnect as well as the switch interconnect ports and “save” the changes made.


After enabling jumbo frames on the EtherChannel port, HEA ports and switch ports as mentioned above check to be sure the EtherChannel port on the Oracle RAC nodes accept jumbo frames.

Bring up the EtherChannel interface on the Oracle RAC nodes.

#ifconfig en3 up #chdev –l en3 –a state=up

Execute ‘ping” command with big size blocks from one Oracle RAC node to another and vice versa.

# ping –s8972 10.0.0.2 PING 10.0.0.2 (10.0.0.2): 8972 data bytes 8980 bytes from 10.0.0.2: icmp_seq=0 ttl=255 time=0 ms 8980 bytes from 10.0.0.2: icmp_seq=1 ttl=255 time=0 ms 8980 bytes from 10.0.0.2: icmp_seq=2 ttl=255 time=0 ms 8980 bytes from 10.0.0.2: icmp_seq=3 ttl=255 time=0 ms 8980 bytes from 10.0.0.2: icmp_seq=4 ttl=255 time=0 ms 8980 bytes from 10.0.0.2: icmp_seq=5 ttl=255 time=0 ms

Execute the “entstat” command to make sure the EtherChannel candidate ports accept jumbo frames.

# entstat –d ent3 |grep –i jumbo RX Q3 Jumbo Packets: 6 / 1 RX Q3 Jumbo Packets: 6 RX Q3 Jumbo Packets: 0 RX Q3 Jumbo Packets: 0 RX Q3 Jumbo Packets: 0 TX and RX Jumbo Frames: On Jumbo Frame Action: Accept RX Q3 Jumbo Packets: 0 / 0 RX Q3 Jumbo Packets: 0 RX Q3 Jumbo Packets: 0 RX Q3 Jumbo Packets: 0 RX Q3 Jumbo Packets: 0 TX and RX Jumbo Frames: On Jumbo Frame Action: Accept

Some issues might be seen while installing Oracle11g CRS version 11.1.0.6 on AIX 6.1.4.3 with root.sh. On the last node of the cluster, root.sh was timed out with the message: Waiting for the Oracle CRSD and EVMD to star”. This didn’t bring up the “ons” and “gsd” resources, thus “root.sh” failed. To avoid this failure, install the CRS without the jumbo frames set to ON in the Oracle RAC interconnect interfaces and enable them after the install and then apply patches or patch set for CRS.

Another issue while executing root.sh on an Oracle RAC node with AIX 6.1.4.3 with or without jumbo frames is that the VIP resource won’t come up ONLINE on the Oracle RAC nodes. To fix this issue, $CRS_HOME/bin/racgvip should be edited as shown below.

The debug trace should be set to capture more details of the VIP failure.

As a root user, #$CRS_HOME/bin/crsctl debug log res “ora.dyn9038158218.vip:1”

Part of $CRS_HOME/log/dyn9038158209/racg/ora.dyn9038158218.vip.log looks like:

2010-06-21 12:57:23.664: [ RACG][1] [299130][1][ora.dyn9038158218.vip]: Tue Jun 21 12:57:22 CDT 2010 [ 483342 ] About to execute command: /usr/sbin/ping -S 9.38.158.218 -c 1 -w 3 9.38.158.129 Tue Jun 21 12:57:23 CDT 2010 [ 483342 ] IsIfAlive: RX packets checked if=en0 failed


2010-06-21 12:57:23.664: [ RACG][1] [299130][1][ora.dyn9038158218.vip]: Tue Jun 21 12:57:23 CDT 2010 [ 483342 ] Interface en0 checked failed (host=dyn9038158218) Tue Jun 21 12:57:23 CDT 2010 [ 483342 ] IsIfAlive: end for if=en0 Tue Jun 21 12:57:23 CDT 2010 [ 483342 ] checkIf: end for if=en0 2010-06-21 12:57:23.664: [ RACG][1] [299130][1][ora.dyn9038158218.vip]: Invalid parameters, or failed to bring up VIP (host=dyn9038158218)

To fix this error, edit the $CRS_HOME/bin/racgvip script as below, look for lines starting with _O1 and _O2 and replace $5 instead of $6.

$CRS_HOME/bin/racgvip: # get RX packets numbers _O1=`$NETSTAT -n -I $_IF | $AWK "{ if (/^$_IF/) {print \\$5; exit}}"` #### the $5 should be replaced with $6 ### x=$CHECK_TIMES while [ $x -gt 0 ] .............. fi _O2=`$NETSTAT -n -I $_IF | $AWK "{ if (/^$_IF/) {print \\$5; exit}}"` #### the $5 should be replaced with $6 ### if [ "$_O1" != "$_O2" ] then # RX packets numbers changed

After editing $CRS_HOME/crs/racgvip script,

#$CRS_HOME/bin/crs_stop –all #$CRS_HOME/bin/crs_start –all

This will bring up the VIP resources on the nodes.

Summary This document provides the detailed AIX EtherChannel setup procedure when used for a private network of Oracle RAC nodes. The Oracle RAC nodes’ private network interconnectivity can take the advantage of 10 Gigabit Ethernet cards as part of the Integrated Virtual Ethernet (IVE) capability available on IBM POWER6 (or higher) processor-based Power System servers along with the AIX operating system EtherChannel feature for high availability, load balancing, higher bandwidth and lower latency.

The physical 10 Gigabit Ethernet ports can be shared by the LPARs using the Logical Host Ethernet Adapter feature, also available in POWER6 (or higher) class Power Systems servers.


Appendix A The following links refer the documents for installing Virtual I/O Server, AIX and Oracle RAC products.

PowerVM Virtualization on IBM System p: Introduction and configuration http://www.redbooks.ibm.com/redbooks/pdfs/sg247940.pdf

AIX 6.1 Installation and Migration http://publib.boulder.ibm.com/infocenter/systems/index.jsp?topic=/com.ibm.aix.install/doc/insgdrf/bos_install_intro.htm

Integrated Virtual Ethernet Adapter –Technical Overview and Introduction http://www.redbooks.ibm.com/redpapers/pdfs/redp4340.pdf

Oracle Real Application Clusters Installation Guide - 11g Release 1 (11.1) for Linux and UNIX http://download.oracle.com/docs/cd/B28359_01/install.111/b28264/toc.htm

Force10 Network S2410 – SFTOS Configuration guide http://www.force10networks.com/CSPortal20/KnowledgeBase/DOCUMENTATION/CLIConfig/SFTOS/SFTOS_Config_2.5.3.0_27-Aug-08.pdf

Force10 Networks S2410 10 Gigabit switch Command Line reference document http://www.force10networks.com/CSPortal20/KnowledgeBase/DOCUMENTATION/CLIConfig/SFTOS/SFTOS_CLI_2.5.3.0_27-Aug-08.pdf


About the author Ravisankar Shanmugam is a Senior IT Specialist with IBM Advanced Technical Skills Solutions team and works at the IBM Oracle International Competency Center based in San Mateo, CA. He provides Power Systems and System x® platform support for projects at the Competency Center and for enablement activities at Oracle Corporation in Redwood Shores and Pleasanton, CA.


Trademarks and special notices © Copyright. IBM Corporation 1994-2010. All rights reserved.

References in this document to IBM products or services do not imply that IBM intends to make them available in every country.

IBM. the IBM logo, ibm.com, AIX, POWER, System Storage, Power Systems and System x are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both:

Other company, product, or service names may be trademarks or service marks of others.

The information provided in this document is distributed “AS IS” without any warranty, either express or implied.

The information in this document may include technical inaccuracies or typographical errors.

Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction.

Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here.

Photographs shown are of engineering prototypes. Changes may be incorporated in production models.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

Documents

Setting up IBM Power Systems 10 Gigabit Ethernet ports and ... · PDF fileSetting up IBM Power Systems 10 Gigabit Ethernet ports and AIX 6.1 EtherChannel for Oracle RAC private interconnectivity