Upload
hoangtu
View
216
Download
0
Embed Size (px)
Citation preview
NetXen, Inc.
Intelligent NIC® Installation and Tuning GuideFor IBM Servers: NetXen Software Release 3.4.216
Version 3.5November 15, 2007
NetXen, Inc.3965 Freedom CircleFourth FloorSanta Clara, CA 95054Telephone: (408) 330-0010Web Site: http://www.netxen.com
Copyright© Copyright 2007. NetXen, Inc. — All Rights Reserved.
DisclaimerThis is a preliminary document and is subject to change without prior notification and does not represent a commitment on the part of NetXen, Inc. The information contained herein is the proprietary and confidential information of NetXen or its licensors, and is supplied subject to, and may be used only by NetXen’s customer in accordance with, a written agreement between NetXen and its customer. Except as may be explicitly set forth in such agreement, NetXen does not make, and expressly disclaims, any representations or warranties as to the completeness, accuracy or usefulness of the information contained in this document.
Please contact NetXen, Inc. to obtain the latest revision of this document.
TrademarksNetXen, the NetXen logo, and Intelligent NIC are trademarks or registered trademarks of NetXen, Inc. Third Party brands and names are the property of their respective owners.
Revision History
Version Date Editor Notes
3.0 6/20/2007 SR First release of document.
3.1 6/272007 SR Added Section 9, Teaming Driver, and all related subsections.Added Appendix C, PXE Boot with the NetXen CardAdded Appendix D, iSCSI Boot with the NetXen Card (place holder)
3.2 7/24/2007 SR Added Section B.2, PXE Configuration.Added Section 8.4, LSA Statistics.Added Appendix C, Using Accelerated iSCSI.
3.3 8/14/20078/20/20078/22/20078/23/2007
SR Added Section 6.5, Additional Support.Added Section 9.2, Linux Bonding/Teaming Mode and all related subsections.Added Section B.2, PXE Configuration for BIOS Setup Changes.Added Section B.3, PXE Configuration, Setting up Windows Server 2003.Added Section B.4, PXE Configuration, Setting up a Linux Server.Added iSCSI mode information at the beginning of Appendix C.
NetXen, Inc
3.4 10/18/07 SR Added Section 2.2.2, XFP Module.Added Table 7, “Quick Start Instructions — Enabling WIndows Chimney and Linux LSA”.Added TCPv4 Stateful Offload entry to Table 8.Added Section 4.5.3, “Uninstalling the NetXen NIC Driver in Linux”.Added Section 4.5.4, “Uninstalling the LSA Driver in Linux”.Removed Appendix B, PXE Boot.Added Section B.2.2, “Windows Remote Installation Services (RIS) Installation”.Added Section 8, NIC VLAN in Windows
3.5 11/13/07 SR Rewrote installation procedure for Section 4.2, “Installing and Connecting the NetXen Mezzanine Card for BladeCenter”.Added Section 4.4.4, Linux Kudzu.Updated Section 6.1, Linux Driver Installation.Updated steps 2 and 3 of Section 7.1.1, Windows Installation.Updated text in Section 10, User Diagnostics.
Version Date Editor Notes
NetXen, Inc
Table of Contents
1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Related Documents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 NIC Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Currently Supported Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2 System Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.1 Software Release Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.2 Hardware and Software Installation Requirements . . . . . . . . . . . . . . . . . . . 2
2.2.1 Cable Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.2.2 XFP Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2.3 System and Software Requirements . . . . . . . . . . . . . . . . . . . . . . 3
3 Quick-Start Installation Instructions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Detailed Installation and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.1 Installing and Connecting the NetXen Intelligent NIC Adapter . . . . . . . . . . 74.2 Installing and Connecting the NetXen Mezzanine Card for BladeCenter . . 94.3 Windows Driver Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.3.1 NIC Driver Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.3.2 Configuring the Network Interface Card . . . . . . . . . . . . . . . . . . . 18
4.4 Linux Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.4.1 NIC Driver Installation (TAR format) . . . . . . . . . . . . . . . . . . . . . . 204.4.2 Linux NIC Driver Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.4.3 Configuring the NIC on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.4.4 Configuring Under Linux Kudzu . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.5 Uninstalling the Software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.5.1 Uninstalling Windows Drivers for Both NIC and Chimney . . . . . 224.5.2 Uninstalling the NetXen NIC Driver in Linux. . . . . . . . . . . . . . . . 234.5.3 Uninstalling the LSA Driver in Linux . . . . . . . . . . . . . . . . . . . . . . 23
5 Firmware Updates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.1 Using the NetXen Flash Utility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.2 Windows Firmware Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255.3 Linux Firmware Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265.4 Updating the Flash on Multiple Standalone Cards in Windows. . . . . . . . . 26
6 Linux Socket Accelerator (LSA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276.1 LSA Driver Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
6.1.1 TAR Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276.2 LSA Offload Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
6.2.1 LSA Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.2.2 LSA Configuration Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.2.3 nxoffload Caveats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
6.3 Running LSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296.4 LSA Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
6.4.1 Example Read Out . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 Teaming Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
7.1 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337.1.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337.1.2 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357.1.3 Fault Tolerance Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
7.2 Linux Teaming/Bonding Mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
NetXen, Inc. i
Table of Contents
7.2.1 Mode 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457.2.2 Mode 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467.2.3 Bonding Driver Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477.2.4 Bonding Driver Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
8 NIC VLAN in Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498.1 Optional Configuration on the NIC and Switch Side . . . . . . . . . . . . . . . . . 498.2 NIC VLAN as Send Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498.3 NIC VLAN as Receive Side. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
9 Troubleshooting Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529.1 Unable to Boot the Host System with the NIC Card Installed . . . . . . . . . . 529.2 Unable to ping the NIC Installed on the Host . . . . . . . . . . . . . . . . . . . . . . 529.3 Gather System Information and Check for Errors on Linux Host . . . . . . . 539.4 Caveats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
9.4.1 MSI Interrupts for Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549.4.2 PCI-Express Slot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
9.5 Additional Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5510 User Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
10.1 Windows Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5610.2 Linux Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Appendix A Tuning Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59A.1 Windows Performance Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59A.1.1 NTttcp (Windows) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59A.1.2 nttcps and ntttcpr Command Line Options . . . . . . . . . . . . . . . . . . . 60A.2 Windows Optimization — Tcp1323Opts . . . . . . . . . . . . . . . . . . . . . . 61A.3 Linux Performance Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62A.3.1 IPERF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62A.4 Linux Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Appendix B Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
ii NetXen, Inc.
1. Overview
1 OverviewThis guide describes the system requirements, installation, and setup required for the NetXen Intelligent NIC® network interface card product release.
1.1 Related DocumentsRelease notes included in the software release.
Check NetXen web site for updates.
1.2 NIC VersionsThe NIC products support Windows, Linux, and other operating systems. See the release notes for an updated list.
1.3 Currently Supported Features
Note: Check with NetXen periodically for software updates and additional features.
Product Number Product Definition
43W4453 Dual 10-GE ports, IBM-MEZZNXB-10GSXR Single 10-GE port (optical fiber)NXB-10CX4 Single 10-GE port (copper)
Feature Windows Linux
TCP, UDP, and IP checksum offload Yes YesTCP segmentation offload (TSO / LSO) Yes YesInterrupt coalescing Yes YesInterrupt distribution No YesVLAN Yes YesSNMP Ethernet MIB Yes YesJumbo frames Yes YesTeaming and Failover Yes YesLinux Socket Accelerator (LSA) N/A YesChimney (TCP/IP Offlead) Yes N/A
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 1
2. System Requirements
2 System RequirementsBefore installing the product, verify the minimum system configuration requirements outlined below.
2.1 Software Release Package Contents
2.2 Hardware and Software Installation Requirements
2.2.1 Cable Specifications
Contents Format Directory Tree
Documentation IBM Installation and Tuning GuideBest Practices Guide
Release NotesNIC Linux Driver [version].tar.gz or [version].rpm
NIC Windows Driver [version].zipNX Intercept Driver [version].-lsa.tar.gz or
[version]-lsa.rpmLSA Driver [version].-lsa.tar.gz or
[version]-lsa.rpmFirmware imez_romimage or xfp_romimage
Diagnostics nxudiagTools nxflash
Table 1 NXB-10GXSR Cable Specifications — Optical1
1. Check with NetXen for information on the NXB-10GXLR long range optical cable specifications.
ParameterOptical Core Thickness
50 μm 62.5 μm
Wavelength 850 nm 850 nmBandwidth (min) 500 MHz 500 MHzOperating Range 300 m 300 m
Table 2 NXB-10CX4 Cable Specifications — Copper
802.3 Physical Layout Network Type Cable Length
10GBASE-CX4 LAN 15m (max)
Windows3.4.216
imx64x86
SNPx64x86
win2k3x64x86
utils
3.4.216
bindocsdriverinclude
Linux3.4.216-lsa
bindriverincludenxoffload
Linux LSA
2 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
2. System Requirements
2.2.2 XFP Module
2.2.3 System and Software Requirements
Table 3 NXB-10GXSR XFP Module Specifications
Manufacturer Name Part Number
JDS Uniphase Multi-rate 10 Gb/s Short Wave Optical XFP Transceiver
64P0194
PicoLight Multi-rate 10 Gb/s 850 nm Optical XFP Transceiver Module
PL-XXL-SC-S45-22
Table 4 System and Software Requirements
Hardware Systems Validated Platforms
Validated OS (32bit and x64) Firmware1
1. NIC and LSA use the same firmware.
Driver
PCIe: (x8)
Memory: Minimum
2GB
AMD, Intel Windows Server: 2003 SP1,SP2,R2
Linux Distros: RHEL3 U8,
RHEL4 U3,U4 SLES9 SP3,
SLES10, RHEL5
NICLSA
Chimney2
2. To enable Chimney, a license must be obtained from NetXen.
Linux:3 NIC, LSA
Windows:4NIC
Chimney2
3. For Linux NIC and LSA, install the kernel source and header files
4. For Windows NIC, SP1 must be installed. SP2 is recommended.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 3
3. Quick-Start Installation Instructions
3 Quick-Start Installation InstructionsThis section is for users already familiar with the Windows or Linux operating systems. For those user’s requiring more detailed information, refer to Section 4.
Table 5 Quick Start Instructions — New NIC Driver Installation
NetXen NIC Hardware Installation Windows Driver Installation Linux Driver Installation
Pre-Installation1. Verify the system meets the requirements
described in Section 2.2. Power down the system.
Installing the NetXen NIC Adapter1. Insert the NetXen NIC adapter card into an
available PCI Express slot.2. Connect the appropriate cables to the
board.3. Power on the system.
Note that the NIC card comes with the firmware already installed.
Driver Installation1. Copy all of the provided NIC driver files
to a directory.2. From the Device Manager, scan for new
hardware.3. When prompted to connect to Windows
Update, Click No, not at this time, then click Next.
4. In the next window, click Install from a list of specific location and then click Next.
5. Select Search for the best driver in these locations. Select Include this location in the search.
6. Provide the path to the directory where the driver NXP2NIC.sys is located. For 32-bit operating systems, choose the x86 directory. For 64-bit operating systems, choose the x64 directory.
NIC Card Configuration1. Open the Network Connections window
from the Control Panel.2. Verify that NetXen NIC adapter icon says
Connected. If not, right-click on the NIC Adapter icon and select Properties, then assign the IP and subnet addresses and click OK.
Driver Installation (tar.gz)1. Untar the file, build, install, and load the
driver: tar xzf [version].tar.gz cd [version] make make install insmod ./driver/nx_nic.ko
Driver Installation (source rpm)1. Install the RPM as follows1
rpm -iv nx_nic-[version].src.rpm2. Create the binary RPM as follows:
rpmbuild -bb /usr/src/redhat/SPECS/ nx_nic-[version].spec, (Redhat) or
rpmbuild -bb /usr/src/packages/SPECS nx_nic-[version].spec (SuSE)
3. Load the nx_nic driver. modprobe nx_nic
NIC Card Configuration1. Identify the NetXen Ethernet interface:
ethtool -i eth[n]2. Assign an IP address:
ifconfig eth[n] [IP address]
1. x.yy.zz is an example of [version]
4 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
3. Quick-Start Installation Instructions
If a NetXen Intelligent NIC card is already installed on the system and a driver and/or firmware update is required, refer to the procedure in Table 6 below.Table 6 Quick Start Instructions — Updating from an Existing Driver and Firmware
Windows Driver/Firmware Update Linux Driver/Firmware Update
Verify Link Status1. With the existing driver and firmware installed,
verify that the interface is up before continuing with the firmware update.
Update to New Firmware1. Open a Command window.2. Go to the utils directory in the driver package
provided.3. The following binaries are required to be in the
same directory from where the Flash command will be executed: nxflash.exe xfp_ibm_romimage imezz_romimage
4. Run the following commands: nxflash -i NX_NIC --all1
5. Verify the firmware has been updated using the following command. nxflash -i NX_NIC --info
Update to the New Driver1. Install the new driver version.2. Verify the new driver version.
1. Writing of the Flash is a critical operation. Please do not interrupt or reboot during this time.
Verify Link Status1. With the existing driver and firmware installed, verify
that the interface is up using ifconfig before continuing with the firmware update.
Update to New Firmware1. Run the following command from the bin directory of
the driver: ./nxflash -i eth[n] --all1
Unload and Remove the Existing Driver1. Unload the existing driver.
ifconfig eth[n] down rmmod nx_nic2. Remove the existing driver module. rm -f /lib/modules/[kernel version]/kernel/ drivers/net/nx_nic.ko
Reboot the System1. Once the existing driver has been removed, reboot
the system.
Update to the New Driver2. Update to the new driver as follows:
tar xzf [version].tar.gz cd [version] make make install insmod driver ./nx_nic.ko
3. Ensure that the Ethernet interface is up. ifconfig eth[n] <ip addr> up
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 5
3. Quick-Start Installation Instructions
Once the WIndows or Linux NIC installation is complete as described in the previous tables, the following procedures in Table 7 can be used to enable Windows Chimney or Linux LSA. Table 7 Quick Start Instructions — Enabling WIndows Chimney and Linux LSA
Enabling Windows Chimney Enabling Linux LSA
The following steps can be used to enable Windows Chimney on the driver. Note that the Windows NIC driver must be installed before executing the procedure below.1
1. Open the Network Connections window.
2. Right click on the icon for the NetXen adapter and select Properties.
3. Click on the Configure button.4. Click on the Advanced tab.5. Change the “TCPv4 Stateful offload” from disabled
to enabled.6. To verify the offload has been enabled and is
operating: C:\>netsh int ip show offload
1.A key is required to be loaded onto the card before Chimney can be enabled. Please refer to the Re-lease Notes for more information on how to obtain and verify this key.
The following steps can be used to enable Linux LSA on the driver. Note that the Linux NIC driver must be installed before executing the procedure below.
1. Untar the file, build, and load the drivers.tar xzf [version]-lsa.tar.gzcd [version]-lsamakecd driverinsmod ./nx_intercept.koinsmod ./nx_lsa.ko
2. To verify the LSA driver is operating, execute the following command:cat /proc/net/nx_nic/lsa_1/stats
3. Example for offloading iperf.cd ..\nxoffload./nxoffload –a –n iperf –z
4. To display the current offload rules execute the following:./nxoffload -s
6 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
4 Detailed Installation and Configuration
4.1 Installing and Connecting the NetXen Intelligent NIC Adapter1 Verify that the system is powered off.
2 Remove the system cover.
3 Locate the x8 PCIe slot if one is available.
4 Once the bracket is removed, insert the adapter into the PCI Express slot1. Apply uniform pressure on top of the board until it is firmly seated in the bus connector.
1. Due to space constraints in some systems, when installing an XFP board, it may be necessary to first remove theXPF module from the board, insert the board into the PCI slot, then reconnect the XFP module to the board.
Identify x8 slots for optimal performance
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 7
4. Detailed Installation and Configuration
5 For optical installations, once the board is inserted into the slot, connect the XFP module. Make sure the optical module clicks when inserted into the cage.
6 Secure the bracket.
7 Connect the appropriate cables to the NIC and replace the cover.
8 Power on the system.
XFP Cable connection
CX4 Cable connection
XFP Module
CX4Connector
8 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
4.2 Installing and Connecting the NetXen Mezzanine Card for BladeCenter1 Verify that the Blade is powered off.
2 Remove the Blade server from the chassis.
3 Remove the Blade cover.
4 Locate the Combo Form Factor Horizontal (CFFh) PCI-Express connector on the Blade server.
5 Remove the PCI-express connector cover if needed.
6 Insert the NetXen Mezzanine card firmly into the CFFh connector. A picture of the mezzanine card is shown below.
7 The NetXen Mezzanine card should be inserted at an angle into the mounting posts. Align the holes in the back of the card with the mounting posts at an angle of approximately 30 - 45 degrees. The card should then be rotated down onto the Blade, and pressed into the CFFh PCI-e connector. Refer to the figure on the next page.
NOTE: If the card is not inserted properly, first into the mounting posts and then into the connector as described above, damage to the card could occur.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 9
4. Detailed Installation and Configuration
8 Replace the Blade cover.
9 Insert the Blade server back into the chassis slot.
10 Power on the Blade.
CFFh PCI-eConnectors
NetXen Mezzanine Card
Mounting Posts
10 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
4.3 Windows Driver InstallationThis section describes the Windows installation procedure. For the Linux driver installation procedure, refer to Section 4.4.
4.3.1 NIC Driver InstallationNote that the following procedure is for a new installation.
1 Copy the NIC release package to a directory on the host.
2 After the server system is powered on, Windows detects the new card and displays the Found New Hardware Wizard window.
When this window appears, select No, not at this time, and click Next.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 11
4. Detailed Installation and Configuration
3 Select Install from a list or specific location (Advanced) and click Next, as follows:
4 Select Search for the best driver in these locations.Select Include this location in the search.
Type the appropriate path to the NIC driver directory in the accompanying field and click Next. An example is shown below:
If the exact directory path is not known, click Browse and select the NIC driver directory.
C:\3.4.216\windows\win2k3\x86
12 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
The NIC card supports both 32-bit and 64-bit operating systems. The x86 directory contains 32-bit driver and the x64 directory contains the 64-bit driver. Select the appropriate directory (x86 or x64) and click OK. Then click Next in the Hardware Update Wizard window to install the driver.
5 The following screen is displayed while the driver is installed.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 13
4. Detailed Installation and Configuration
6 Once the driver installation is complete, the following window will appear. Click Finish to exit.
7 To verify that the driver was properly installed, Right-click on My Computer, then select Device Manager. A listing of devices appears in right side of the window. Under Network Adapters, the NetXen 10Gb Ethernet Adapter should be listed as shown below:
14 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
8 In the above window, right-click on the NetXen 10Gb Ethernet adapter and select Properties. The General tab will be displayed as shown.
9 There are four tabs — General, Advanced, Driver, and Resources. Click on the Advanced tab to display the driver properties as shown below.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 15
4. Detailed Installation and Configuration
The Advanced tab selections displayed above are defined as follows:Table 8 Description of Advanced Properties Parameters
Parameter Description
Flow Control Sets flow control parameters.Registry Key: FlowControlDefault: Enabled.Values: Enabled/Disabled
Large Send Offload TCP Large send offload – Enable/Disable options.Registry Key: LargeSendOffloadDefault: Enabled.Values: Enabled/DisabledSize of LSO: 64K
Locally Administered Address This is the Locally Administered Address (LAA) that users/administrators can set on this interface and this address overrides the permanent address of the adapter (that may have been flashed in to the hardware).Registry Key: NetworkAddressFormat: HexOptions: Value (a value is required, enter the 12 hex bytes of the MAC address to be used), or not present.
Max Ethernet Frame Size Ethernet frame size for packet transmission and receive (does not include MAC Header).Reg Key: MaxFrameSize)Default: 1500 (corresponds to 1514 bytes on the wire + 4 bytes of CRC)Max: 8000 Min: 1500
Max Jumbo Buffers Number of buffers per Jumbo frame.Reg Key: RxJumboRingSizeDefault: 4096 Values: 1024, 2048, 4096, 8192.
Maximum Receive buffers (in K) This is packet buffers used by the driver for receive (DMA memory and indicated to stack). Size of each buffer is 1536 bytes.Registry Key: MaxRxBuffersMin: 1KMax: 16KDefault: 8192
802.1P/Q Packet Tagging For supporting 802.1p/q priority tagging. Should be enabled to set VLAN ID.Registry Key: PQTaggingDefault: EnabledValues: Enable/Disabled
16 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
Receive IP Checksum Offload This parameter is for enabling/disabling the receive IP checksum offload.Registry Key: RxIPChecksumOffloadDefault: EnabledValues: Enabled/Disabled
Receive TCP/UDP Checksum Offload This parameter is for enabling/disabling the receive TCP and receive UDP checksum offload.Registry Key: RxTCPChecksumOffloadDefault: EnabledValues: Enabled/Disabled
Status Ring Size Receive status ring size from hardware.Registry Key: StsRingSizeDefault: 8192Values: 1024, 2048, 4096, 8192, 16384, 32768.
TCPv4 Stateful Offload TCPv4 stateful offload.Registry Key: EnableTcpOffloadDefault: DisabledValues: Enabled/Disabled
Transmit IP Checksum Offload This parameter is for enabling/disabling the transmit IP checksum offload.Registry Key: TxIPChecksumOffloadDefault: EnabledValues: Enabled/Disabled
Transmit Ring Size Transmit command ring size provided to hardware.Registry Key: TxRingSizeDefault: 1024Values: 1024, 2048.
Transmit TCP/UDP Checksum Offload
This parameter is for enabling/disabling the transmit TCP and transmit UDP checksum offload.Registry Key: TxTCPChecksumOffloadDefault: EnabledValues: Enabled/Disabled
VLAN ID For setting the VLAN ID for this interface (also exposed through the standard OID).Registry Key: VlanIdDefault: 0 (no VLAN)Range: 0 - 4094
Table 8 Description of Advanced Properties Parameters (Continued)
Parameter Description
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 17
4. Detailed Installation and Configuration
10 Click on the Driver tab in the Advanced Properties window.
The driver can be updated, rolled back, or uninstalled from this window.
This completes the installation of the NetXen NIC software. Refer to the next section for information on configuring the NIC.
4.3.2 Configuring the Network Interface Card
1 Configure the network connections by opening the Network Connections window in the control panel, or directly from the desktop if the icon exists. The following window is displayed.
18 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
Note that if an IP address has not been assigned, the NetXen 10Gb Ethernet adapter entry above will say “Limited or no connectivity”.
If a red ‘X’ appears in the NetXen 10Gb Ethernet10 Adapter icon, this indicates that the link is down. If this occurs, check the following:
Verify that the XFP module is installed correctly.
Verify that the card is seated in the PCI slot properly.
Make sure the network cable is connected properly at both ends.
Is a known good cable being used?
Check the XFP module and replace if necessary.
Has an IP address been assigned to the interface?
To check the status of the connection, right-click on the NetXen 10 Gb NIC adapter icon above and select properties. This causes the Local Area Connection Properties window to be displayed.
2 To assign an IP address manually, select the Internet Protocol (TCP/IP) entry below and click the Properties tab.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 19
4. Detailed Installation and Configuration
3 In the Internet Protocol (TCP/IP) Properties window, assign the appropriate IP, subnet mask, and default gateway values.
4 Using the ping utility, verify connectivity with another card or switch.
4.4 Linux InstallationThe Linux sources can be installed in either TAR of RPM format depending on the release package you received as described below.
Note that the following procedure is for a new installation.
4.4.1 NIC Driver Installation (TAR format)
Note: Root level permissions are required for the driver installation.
1 Copy the file [version]1 tar.gz to the host system and follow the installation command sequence:
To build, install, and load the driver:
tar xvzf [version].tar.gz
cd [version]
make
make install
(driver module is installed in: /lib/modules/<kernel version>/kernel/drivers/net)
cd driver
1. Where [version] is the numbered release version of the software.
20 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
insmod ./nx_nic.ko
Verify that nx_nic driver is loaded:
lsmod | grep -i nx_nic
2 Identify the NetXen Ethernet interface:ethtool –i eth[n]
3 Assign an IP address and bring up the Ethernet interfaceifconfig eth[n] [IPaddress] up
4.4.2 Linux NIC Driver OptionsStep 1 above uses the insmod command. The following optional parameters can be passed to the driver with the modprobe or insmod commands:
insmod nx_nic <parameter> modprobe nx_nic <parameter>
The optional parameters are listed below.
4.4.3 Configuring the NIC on LinuxAssign an IP address to the interface from the command line. For example:
1 Identify the Ethernet interface (eth1, eth2, etc.) and the nx_nic driver:
ethtool -i eth[n]ifconfig -eth[n] -a
2 Assign an IP Address and activate the interface:ifconfig eth[n] downifconfig eth[n] <IP Address> <netmask> up
Table 9 insmod and modprobe Command Optional Parameters
Parameter Description Default Range
tx_desc1
1.Example usage: modprobe nx_nic tx_desc=1024 or insmod nx_nic tx_desc=1024
Sets the size of the command descriptor rings (Tx). 1024 256 - 4096 (must be power or two)
jumbo_desc2
2.Example usage: modprobe nx_nic jumbo_desc=1024 or insmod nx_nic jumbo_desc=1024
Sets the size of the receive ring for jumbo frames. 1024 64 - 4096(must be power or two)
rdesc_1g3
3.Example usage: modprobe nx_nic rdesc_1g=8192 or insmod nx_nic rdesc_1g=8192
Sets the size of the receive ring for normal packets (1 GbE ports).
8192 64 - 32,768
rdesc_10g4
4.Example usage: modprobe nx_nic rdesc_10g=32768 or insmod nx_nic rdesc_10g=32768
Sets the size of the receive ring for normal packets (10 GbE port).
32,768 64 - 32,768
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 21
4. Detailed Installation and Configuration
3 Verify network services are running, and ping:service network restartping <IP address>
4.4.4 Configuring Under Linux KudzuAfter powering on the system the Linux Kudzu may report that the NetXen XG Mgmt hardware has been added to the system. If so, please use the Ignore option to continue with the booting of the Linux OS as shown below.
4.5 Uninstalling the SoftwareThis section describes how to uninstall the NetXen NIC software from Windows and Linux systems.
4.5.1 Uninstalling Windows Drivers for Both NIC and ChimneyTo uninstall the Windows driver, perform the following steps:
1 Right-click on My Computer and click on Manage. The Computer Management window is displayed.
2 Double click on Device Manager. The list of devices in the computer is displayed on the right side of the window.
3 Locate the NetXen Dual 10G Ethernet Server Adapter under the Network adapters section.
4 Right-click on NetXen 10Gb Ethernet Server Adapter.
5 Click Uninstall to perform the operation.
22 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
4. Detailed Installation and Configuration
4.5.2 Uninstalling the NetXen NIC Driver in LinuxNote that if the LSA driver is installed, it must be uninstalled before the NetXen Linux NIC driver is uninstalled. Refer to Section 4.5.3 below for information on how to uninstall the LSA driver.
TAR FormatIn the Linux system, the TAR archive may be removed as follows:
rm -rf [version]
rm -f [version].tar.gz
ifconfig eth[n] downrmmod nx_nic.ko
The driver may be removed from /lib/modules as follows:
rm -f /lib/modules/$(uname -r)/kernel/drivers/net/nx_nic.ko
4.5.3 Uninstalling the LSA Driver in Linux
TAR FormatNote that the LSA driver must be removed before the Linux NIC driver can be uninstalled as described in Section 4.5.2 above. The removal procedure for the LSA driver is shown below.
rm -rf [version]-lsa
rm -f [version]-lsa.tar.gz
rmmod nx_lsa.koThe driver may be removed from /lib/modules as follows:
rm -f /lib/modules/$(uname -r)/kernel/drivers/net/nx_lsa.ko
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 23
5. Firmware Updates
5 Firmware UpdatesThe NetXen card is shipped with a default firmware version. When a newer version of driver and firmware is available, the card is updated to the latest GA version of the firmware as described in the Release Notes.
Typically the firmware and the driver version should match unless it is stated in the Release Notes that the supported firmware and driver versions in the specific release do not match.
Note that the firmware must be properly Flashed in order for the adapter to function properly. Check with NetXen periodically for firmware updates.
5.1 Using the NetXen Flash UtilityThe NetXen Flash utility can be used to update the firmware. When using the nxflash.exe utility, the following syntax should be used.
nxflash -i NX_NIC [command] (for Windows OS)1
./nxflash -i eth[n] [command] (for Linux OS)
In this command line, the -i indicates the interface command and the NX_NIC specifies the actual interface for Windows. The [n] indicates the number of the Ethernet interface for Linux.
Along with the interface, the available commands are listed in Table 10.
The following are some examples of how to use the nxflash utility:
Update firmware image
Get firmware information
Each of these functions is described in detail below.
1 Update firmware image
1. For Windows, the interface type MUST BE ALL CAPITALS to be recognized as a valid interface.
Table 10 NetXen Flash Utility Commands
Command Definition Description
-i--interface
interface Specifies the interface type [NX_NIC, NX_NIC1, etc.]
--info information Prints firmware version and MAC address.-a
--allFirmware update Complete update of BIOS and firmware.
-r--restore
restore Restores the existing firmware version on the card.
-h--help
help Displays the help menu.
-v--version
version Displays the version information.
24 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
5. Firmware Updates
The following files must be present in the directory from where nxflash is executed:
imez_romimage - This is the firmware image for the IBM IMEZZ card.
xfp_ibm_romimage - This is the firmware image for the IBM XFP card.
These files are supplied as part of the software release. To update the firmware, change directory where the rom_image files are located, and type the following:
nxflash -i NX_NIC --all (for Windows)
./nxflash -i eth[n] --all (for Linux)
This above update command displays the current firmware version on the card and the new firmware version to which the card will be updated upon completion. The Flash update takes a few minutes to complete and MUST NOT BE INTERRUPTED.
2 Get Flash related information
The --info option may be used to get information from the Flash. e.g.
nxflash -i NX_NIC --info (for Windows)
./nxflash -i eth[n] --info (for Linux)
The following is an example of a Windows output:
C:\3.4.216\utils>nxflash.exe -i NX_NIC --infoVerifying flash...Done.Board Type : XGb IMEZBoard Chip Rev : C1Serial Id : MZ76MK0238Firmware version : 3.4.216BIOS version : 1.0.2MAC Addr 0 : 00:0E:1E:00:1C:F6MAC Addr 1 : 00:0E:1E:00:1C:F7Subsystem ID : 0x00044040
5.2 Windows Firmware UpdateThe following steps are used for updating the Windows OS firmware.
1 Open Command window.
2 Go to the utils directory in the driver package provided.
3 The following binaries are required to be in the same directory from where the Flash command will be executed:
nxflash.exe imez_romimage xfp_ibm_romimage
4 Run the following command:nxflash -i NX_NIC --all
Please do not interrupt or reboot during this time.
5 Reboot the system.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 25
5. Firmware Updates
Verify the firmware has been updated using the following command.
nxflash -i NX_NIC --info
5.3 Linux Firmware UpdatesThe following steps are used for updating the Linux OS firmware.
1 Using the existing firmware and driver on the host, bring the Ethernet interface up using the following command:
Ifconfig eth[n] <IP addr> netmask <netmask> up
2 Update to the new firmware from the bin directory:cd <new release version directory>/bin./nxflash -i eth[n] --all
3 Bring down the Ethernet interface.ifconfig eth[n] down
4 Remove the existing or old driver module.rmmod nx_nic
5 Build and install the new driver.cd <new release version directory>makemake install
6 Reboot the host system.
7 Load the new driver.cd <new release version directory>/driverinsmod ./nx_nic.ko
5.4 Updating the Flash on Multiple Standalone Cards in WindowsThe Flash can be updated on multiple cards on the same host using one of the following procedures.
1 If the updates are being done from the same directory tree, the following commands must be run sequentially after the Flash update has completed.
nxflash.exe -i NX_NIC0 --allnxflash.exe -i NX_NIC1 --all
2 Updates can be done simultaneously from two different directory trees. Note that the following two commands are being executed from two separate directory trees.
c:\[version]-a\utils\ nxflash.exe -i NX_NIC0 --allc:\[version]-b\utils\ nxflash.exe -i NX_NIC1 --all
26 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
6. Linux Socket Accelerator (LSA)
6 Linux Socket Accelerator (LSA)The purpose of LSA is to allow the user to offload a significant amount of TCP network processing on to the Netxen NIC, significantly reducing host CPU utilization. Note that the LSA does not require the user to rebuild the Linux kernel, as simple loadable modules are used.
As described below, the user has very fine-grained control over what can be offloaded. Offload decision can be based on application name, local TCP port number, or remote TCP port number.
6.1 LSA Driver InstallationThis section describes the LSA installation instructions for the tar and RPM formats.
NOTE: The LSA Intercept driver must be running at all times for the LSA driver to work. Whenever the server is rebooted, verify the NIC and intercept drivers are running (by using the lsmod command) before using the LSA driver. The LSA intercept driver cannot be removed with the command rmmod; Remove the driver file from /lib/modules/<kernel>/drivers/net/, and reboot the host.
6.1.1 TAR FormatThis section describes the LSA driver installation from the tar package.
1 First Load the NIC driver (nx_nic.ko)
2 Build and load the Intercept and LSA driver
% tar xvzf <version>-lsa.tar.gz
% cd <version>-lsa
% make
% make install
% insmod./driver/nx_intercept.ko
% insmod./driver/nx_lsa.ko
3 Use offload rules to offload the applications
4 Use rmmod nx_lsa to remove the LSA driver
6.2 LSA Offload ConfigurationLSA can be configured via the nxoffload utility. This utility works by maintaining a list of rules that dictate which applications to offload. Each rule can take one of the following parameters:
Application name
Local TCP port number
Local and Remote TCP port number
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 27
6. Linux Socket Accelerator (LSA)
6.2.1 LSA Configuration OptionsA list of parameters to this utility can be obtained by typing ./nxoffload -help
usage: ./nxoffload <options>
The options are listed below.
6.2.2 LSA Configuration Examples
To offload the iperf application, type the following: ./nxoffload -a -n iperf
To offload the application that uses local TCP port 80, type the following: ./nxoffload -a -p80
To remove the application that uses local TCP port 80, type the following: ./nxoffload -r -p80
To list the current offload rules, type the following: ./nxoffload -s
To offload an application with zero copy enabled, type the following: ./nxoffload -a -z -n <application-name>.
Zero copy is a way to significantly reduce CPU utilization in the send path for some applications. Not all applications may benefit from using zero copy, and some reduction in throughput may be observed. Applications that do well with zero copy are those that have large write sizes and a large number of individual connections.
6.2.3 nxoffload Caveatsnxoffload has the following caveats:
To add or remove offload rules, nxoffload requires root access
Rules may be added or removed at any time. The addition or removal of a rule, however, will not affect connections that have already been established. For example,
Table 11 LSA Configuration Options
<option> Definition Comments
-p Port number to be offloaded.-n Application name to be offloaded.-t TCP tuple to be offloaded. Format: local IP, local port, remote
OP, remove port. Example: ./nxoffload -a -t'0, 0, 0, 5001'./nxoffload -a -t'0, 2000, 0, 5001'
-a Add an offload rule.-r Remove and offload rule.-z Enable zero copy.-s List all ports and applications offloaded.
28 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
6. Linux Socket Accelerator (LSA)
if the user starts a non-offloaded iperf application and then types ./nxoffload -a -n iperf, this instantiation of iperf will NOT be subsequently offloaded. Future instantiations, however, will be offloaded.
6.3 Running LSAOnce the appropriate rule has been entered into nxoffload, no further user intervention is required. The offload occurs automatically. The offload statistics can be monitored by reading the file /proc/net/nx_nic/lsa_x/stats, where 'x' is NetXen card number.
6.4 LSA StatisticsTable 12 lists the different types of LSA statistics, along with the current support status for each type.
Table 12 LSA Statistics
Name Description Status
TCP StatisticsTcpMaxConn Total number of connections since initial power up. Not supported.
TcpActiveOpens Total number of connections opened via connect. Supported.TcpPassiveOpens Total number of connections opened via accept. Not supported.TcpAttemptFails Number of unsuccessful connection attempts. Supported.TcpEstabResets Resets received by connections in the established
state.Supported.
TcpCurrEstab Number of connections currently in the established state.
Supported.
TcpInSegs Number of received segments. Supported.TcpOutSegs Number of transmitted segments. Supported.
TcpSlowOutSegs Number of transmit segments processed by the slow firmware path.
Supported.
TcpRetransSegs Number of retransmitted Supported.TcpInErrs Number of illegal received segments. Supported.
TcpOutRsts Number of transmitted resets. Supported.TcpOutCollapsed Number of collapsed transmitted segments. A
collapsed segment is generated when smaller segments are combined to form a single large segment
Supported.
IP StatisticsIpInReceives The total number of input datagrams received from
interfaces, including those received in error.Supported.
IpInHdrErrors The number of input datagrams discarded due to errors in their IP headers, including bad checksums, version number mismatch, other format errors, time-to-live exceeded, errors discovered in processing their IP options, etc.
Supported.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 29
6. Linux Socket Accelerator (LSA)
IpInAddrErrors The number of input datagrams discarded because the IP address in their IP header's destination field was not a valid address to be received at this entity. This count. includes invalid addresses (e.g., 0.0.0.0) and addresses of unsupported Classes (e.g., Class E). For entities which are not IP routers and therefore do not forward datagrams, this counter includes datagrams discarded because the destination address was not a local address.
Not supported.
IpInNoRoutes (Not in ip-mib)
Number of input datagrams discarded because there was a problem with either determination of inbound interface or a problem with determination of a route back to the source of the datagram.
Supported.
IpInDiscards The number of input IP datagrams for which no problems were encountered to prevent their continued processing, but which were discarded (e.g., for lack of buffer space). Note that this counter does not include any datagrams discarded while awaiting re-assembly.
Not supported.
IpInDelivers The total number of input datagrams successfully delivered to IP user-protocols (including ICMP).
Supported.
IpOutRequests The total number of IP datagrams which local IP user-protocols (including ICMP) supplied to IP in requests for transmission. Note that this counter does not include any datagrams counted in ipForwDatagrams.
Supported.
IpOutDiscards The number of output IP datagrams for which no problem was encountered to prevent their transmission to their destination, but which were discarded (e.g., for lack of buffer space). Note that this counter would include datagrams counted in ipForwDatagrams if any such packets met this (discretionary) discard criterion.
Supported.
IpOutNoRoutes The number of IP datagrams discarded because no route could be found to transmit them to their destination. Note that this counter includes any packets counted in ipForwDatagrams which meet this ̀ no-route' criterion. Note that this includes any datagrams which a host cannot route because all of its default routers are down.
Supported.
IpReasmTimeout The maximum number of seconds which received fragments are held while they are awaiting reassembly at this entity.
Not Supported.
IpReasmReqds The number of IP fragments received which needed to be reassembled at this entity.
Not Supported.
IpReasmOKs The number of IP datagrams successfully re-assembled.
Not Supported.
Table 12 LSA Statistics (Continued)
Name Description Status
30 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
6. Linux Socket Accelerator (LSA)
6.4.1 Example Read OutThe following is a sample of the TCP and IP statistics displayed when the stats command is executed.
cat /proc/net/nx_nic/lsa_1/stats
IpReasmFails The number of failures detected by the IP re-assembly algorithm (for whatever reason: timed out, errors, etc). Note that this is not necessarily a count of discarded IP fragments since some algorithms (notably the algorithm in RFC 815) can lose track of the number of fragments by combining them as they are received.
Not Supported.
IpFragOKs The number of IP datagrams that have been successfully fragmented at this entity.
Not Supported.
IpFragFails The number of IP datagrams that have been discarded because they needed to be fragmented at this entity but could not be, e.g., because their Don't Fragment flag was set.
Not Supported.
IpFragCreates The number of IP datagram fragments that have been generated as a result of fragmentation at this entity.
Not Supported.
TCP Statistics Port 0 Port 1 Port 2 Port 3
TcpMaxConn 0 0 0 0
TcpActiveOpens 180 0 0 0
TcpPassiveOpens 0 0 0 0
TcpAttemptFails 0 0 0 0
TcpEstabResets 0 0 0 0
TcpCurrEstab 0 0 0 0
TcpInSegs 54593130 0 0 0
TcpOutSegs 173931132 0 0 0
TcpSlowOutSegs 2690 0 0 0
TcpRetransSegs 743 0 0 0
TcpInErrs 0 0 0 0
TcpOutRsts 0 0 0 0
TcpOutCollapsed 0 0 0 0
TcpTimeWaitConns 0 0 0 0
Table 12 LSA Statistics (Continued)
Name Description Status
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 31
6. Linux Socket Accelerator (LSA)
IP Statistic Value
IpInReceives 54593098
IpInHdrErrors 0
IpInAddrErrors 0
IpInNoRoutes 0
IpInDiscards 0
IpInDelivers 54593130
IpOutRequests 173933823
IpOutDiscards 0
IpOutNoRoutes 0
IpReasmTimeout 0
IpReasmReqds 0
IpReasmOKs 0
IpReasmFails 0
IpFragOKs 0
IpFragFails 0
IpFragCreates 0
32 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
7. Teaming Driver
7 Teaming DriverThis section discusses the installation and configuration options for Windows and Linux.
7.1 Windows
7.1.1 InstallationOnce the Windows driver has been installed and the firmware updated, the IM (Intermediate driver) can be installed using the following procedure:
1 Install the Microsoft Visual C++ Distributable package (x86 or x64) -- vcredist_x86.exe or vcredist_x64.exe
The software in included in the NetXen driver release package, or it can be downloaded directly from Microsoft.
Microsoft Visual C++ Redistributable Package. The Microsoft Visual C++ 2005 Redistributable Package (x86) installs run time components of Visual C++ Libraries required to run applications developed with Visual C++ on a computer that does not have Visual C++ 2005 installed. This package installs run time components of C Run Time (CRT), Standard C++, ATL, MFC, OpenMP and MSDIA libraries
2 Select and open "Network Connections" window. Right Click on NetXen NIC Adapter icon to open the NetXen Properties window.
3 In the Local Area Connection Properties window, select Network Load Balancing from the list and click Install, then click OK.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 33
7. Teaming Driver
4 The Select Network Component Type window is displayed.
5 Select Protocol and click the Add button. This causes the Select Network Protocol window to be displayed as shown below.
6 Click on the Have Disk button in the Select Network Protocol window above. This displays the Install From Disk window as shown below.
C:\NetXen\3.4.216\im\x64
34 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
7. Teaming Driver
7 In the Install From Disk window, open and select the NetXen IM subdirectory containing the nxteam driver as shown and click OK. The Select Network Protocol window is again displayed with the information shown below.
8 Select NetXen Network Configuration Utility 7 and click OK. After the driver is installed, Windows should be rebooted to complete the installation of the NetXen Network Configuration Utility.
7.1.2 Configuration
1 The icon for Launch NetXen Network is located on the toolbar icon tray. Double-click this icon to launch the Network Configuration GUI. Alternatively, you can open the NetXen NIC adapter properties, and click the Properties button to launch the NetXen Network Configuration Utility GUI to configure teaming. Either of these procedures causes the Local Area Connection 20 Properties window to be displayed as shown below.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 35
7. Teaming Driver
2 Single click on the NetXen Network Configuration Utility 7 entry, then click Properties to display the NetXen Network Configuration 7 Properties window shown below.
The following window illustrates how to highlight the entries to ‘Team’ the appropriate adapters together.
36 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
7. Teaming Driver
Once the appropriate entries are highlighted, click the Team button above to team the adapters together. An example is shown below.
3 The NetXen Network Configuration Utility 7 Properties window contains all the configuration options to setup the teaming properties for the Intelligent NIC and its ports. Double-click on the NetXen Network Team #1 entry to display the Team Properties window shown below.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 37
7. Teaming Driver
4 In the Team Properties window, click on the Information tab to display general information about the board as shown below.
5 After clicking OK in the Team Properties window above and OK in the NetXen Configuration Utility 7 Properties window, the user must click on the Close button in the Local Area Connections Properties window for the teaming to take effect.
38 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
7. Teaming Driver
Click on Continue Anyway as shown in the window below to permit the completion of the teaming setup.
6 During the teaming setup, the following window is displayed. The time required to configure the adapters varies depending on the system configuration.
7 Once the adapter configuration is complete, the click OK in the following windows.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 39
7. Teaming Driver
7.1.3 Fault Tolerance OptionsThe NetXen adapter provides three fault tolerant options:
Manual: This setting allows the user to select between the Primary NIC and secondary NIC options.
Fail on Fault: Should the primary NIC fail, this option enables the system to automatically switch from the primary NIC to the secondary NIC.
Smart Switch: This option allows the user to designate a certain ‘team’ member as the preferred NIC. This team member is always used as long as it is enabled. However, if the preferred NIC fails, and then some time later is brought back on-line, the system automatically switches back to it since it is the preferred NIC. Smart Switch is the recommended choice for fault-tolerant applications.
Manual ModeManual Mode allows the user to switch between the primary and secondary adapters. Once the teaming control has been installed and configured as described in Sections 5.1.1 and 5.1.2 above, manual mode can be set as described below.
1 In the Team Properties window, click on the Manual button, then click Change Mode and click OK. Note that the Switch Now option is disabled until Manual is selected and OK is clicked.
40 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
7. Teaming Driver
2 After clicking OK in the above window, the following window is displayed. Perform a switch of primary adapter by clicking "Switch Now".
3 Finally, click on OK to complete the manual switch process.
Fail on FaultThe Fail on Fault mode causes the system to automatically switch to the secondary adapter when a failure occurs on the primary adapter.
1 In the Team Properties window, click on the Manual button, then click Change Mode and click OK. This step selects the primary adapter.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 41
7. Teaming Driver
In the Teaming Properties windows, click on Fail on Fault as shown below, then click Change Mode, then click OK.
Smart SwitchThis option allows the user to designate a certain ‘team’ member as the preferred NIC. This team member is always used as long as it is enabled. However, if the preferred NIC fails, and then some time later is brought back on-line, the system automatically switches back to it since it is the preferred NIC. Smart Switch is the recommended choice for fault-tolerant applications.
A Smart Switch from the Current Primary Adapter [13] to the secondary adapter [14] is performed as follows:
1 In the Team Properties window, click on the Manual button, then click Change Mode and click OK. This step selects the primary adapter.
42 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
7. Teaming Driver
2 In the Team Properties window, click on Smart Switch as shown below, then click OK.
3 Click OK on the NetXen Network Configuration Utility 7 Properties
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 43
7. Teaming Driver
4 Click Close on the Local Area Connection Properties.
7.2 Linux Teaming/Bonding ModeThis section describes the Linux teaming/bonding mode. The steps for setting this mode are as follows:
1 cd <release>
2 insmod ./driver/nx_nic.ko
3 insmod bonding mode=1 miimon=100 updelay=50000 primary=eth16
4 ifconfig bond0 25.145.1.199 netmask 255.255.255.0 broadcast 25.145.1.255 up
5 ifenslave bond0 eth15 eth16
To bring down the bond please use: ifconfig bond0 down
Note: These commands and steps have been tested under RHEL5, but for other Linux distributions and versions there may be differences in how bonding is enabled and utilized.
7.2.1 Mode 1active-backup or mode=1
Active-backup policy: Only one slave in the bond is active.
A different slave becomes active if, and only if, the active slave fails.
By setting mode=1, when ifenslave is executed the bonding driver calls into
bond_enslave() and sets the slave link modes for eth15 and
eth16 to BOND_LINK_DOWN.
44 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
7. Teaming Driver
Since the bond mode is 1, ie. BOND_MODE_ACTIVEBACKUP, and primary device is indicated, one slave device is configured as active slave and dmesg will show "bonding: bond0: making interface eth16 the new active one 0 ms earlier." and this will result in bond->current_slave being updated and
MII Status: up will be reported for this link and this is what cat /proc/net/bonding/bond0 prints. For the other port, dmesg will display:
bonding: bond0: link status up for interface eth15, enabling it in 50000 ms.
and 'cat /proc/net/bonding/bond0' will print for eth15, MII Status: down;
After 50 seconds dmesg will print:
bonding: bond0: link status definitely up for interface eth15.
bonding: bond0: making interface eth15 the new active one.
'cat /proc/net/bonding/bond0' will print MII Status: up for eth15.
7.2.2 Mode 0If the mode is set to 0 i.e. not BOND_MODE_ACTIVEBACKUP, (in this case Round Robin)
insmod bonding mode=0 miimon=100 updelay=50000 primary=eth16
then both links delay for 50 secs before their states are set to BOND_LINK_UP state,
blade12$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.0.1 (January 9, 2006)
Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 50000 Down Delay (ms): 0
Slave Interface: eth16
MII Status: down <========= down state
Link Failure Count: 0
Permanent HW addr: 00:0e:1e:00:11:d2
Slave Interface: eth15
MII Status: down <========= down state
Link Failure Count: 0
Permanent HW addr: 00:0e:1e:00:11:d3
after 50 secs:
blade12$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.0.1 (January 9, 2006)
Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 50000 Down Delay (ms): 0
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 45
7. Teaming Driver
Slave Interface: eth16
MII Status: up <=========== up state
Link Failure Count: 0
Permanent HW addr: 00:0e:1e:00:11:d2
Slave Interface: eth15
MII Status: up <=========== up state
Link Failure Count: 0
Permanent HW addr: 00:0e:1e:00:11:d3
And dmesg will print:
bonding: bond0: link status definitely up for interface eth16.
bonding: bond0: link status definitely up for interface eth15.
7.2.3 Bonding Driver OptionsOptions for the bonding driver are supplied as parameters to the bonding module at load time. They may be given as command line arguments to the insmod or modprobe command, but are usually specified in either the /etc/modules.conf or /etc/modprobe.conf configuration file, or in a distro-specific configuration file (some of which are detailed in the next section).
7.2.4 Bonding Driver ParametersThe available bonding driver parameters are listed below. If a parameter is not specified the default value is used. When initially configuring a bond, it is recommended "tail -f /var/log/messages" be run in a separate window to watch for bonding driver error messages.
It is critical that either the miimon or arp_interval and arp_ip_target parameters be specified, otherwise serious network degradation will occur during link failures. Very few devices do not support at least miimon, so there is really no reason not to use it.
Options with textual values will accept either the text name or, for backwards compatibility, the option value. E.g., "mode=802.3ad" and "mode=4" set the same mode.
The parameters are as follows:
arp_interval
Specifies the ARP link monitoring frequency in milliseconds. If ARP monitoring is used in an etherchannel compatible mode (modes 0 and 2), the switch should be configured in a mode that evenly distributes packets across all links.
46 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
7. Teaming Driver
If the switch is configured to distribute the packets in an XOR fashion, all replies from the ARP targets will be received on the same link which could cause the other team members to fail. ARP monitoring should not be used in conjunction with miimon. A value of 0 disables ARP monitoring. The default value is 0.
arp_ip_target
Specifies the IP addresses to use as ARP monitoring peers when arp_interval is > 0. These are the targets of the ARP request sent to determine the health of the link to the targets.
Specify these values in ddd.ddd.ddd.ddd format. Multiple IP addresses must be separated by a comma. At least one IP address must be given for ARP monitoring to function. The maximum number of targets that can be specified is 16. The default value is no IP addresses.
downdelay
Specifies the time, in milliseconds, to wait before disabling a slave after a link failure has been detected. This option is only valid for the miimon link monitor. The downdelay value should be a multiple of the miimon value. If not, it will be rounded down to the nearest multiple. The default value is 0.
lacp_rate
Option specifying the rate in which we'll ask our link partner to transmit LACPDU packets in 802.3ad mode. Possible values are:
- slow or 0: Request partner to transmit LACPDUs every 30 seconds
- fast or 1: Request partner to transmit LACPDUs every 1 second
- The default is slow.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 47
8. NIC VLAN in Windows
8 NIC VLAN in WindowsThis section describes the NIC VLAN on Windows using Miniport Driver (NIC Driver). The IM Driver GUI can set multiple VLAN IDs, but the NIC Driver property sheet can set only one VLAN-ID. The VLAN decision table shown below describes how the packets are handled by the NIC driver.
By default, when the VLAN-ID is set on the NIC driver property, tagging is enabled. Ensure that the switch port connected to the NIC has tagging enabled.
8.1 Optional Configuration on the NIC and Switch SideThe NIC driver property also allows tagging to be disabled when VLAN-ID is
enabled.
There are 2 separate settings: VLANID and TAGGING.
1 Tagging must be enabled for the NIC to tag packets on either 802.1p or 802.1Q (Priority/VLAN) or both. Tagging is always enabled by default on the NIC. If the stack sends out a packet with 802.1p (priority setting), the NIC adds the VLAN tag of 0 along with this priority and send the packets with tagging.
The NIC will also processes packets with or without tagging and passes the information up to the stack with this setting.
2 VLANID is not really needed and is only useful when setting one VLANID on the NIC and tagging it with that VLAN ID. By default, 0 here means NO VLAN.
8.2 NIC VLAN as Send SideTo generate the components of the packet's tag header a miniport driver:
1 Set the tag header's CFI bit to zero
2 Proceed as indicated in the following table, based on the following values:Configured VLAN ID for the NIC.Priority or VLAN ID in the packet's OOB data.
Table 13 Windows VLAN Configuration Summary
VLAN-ID Tagging Packets Comments
Zero Enabled Untagged This is the behavior with default NIC on installation (If you don't configure it). Allows a user to set and configure multiple VLANs from the Windows IM GUI.
Zero Disabled UntaggedNon-Zero Disabled UntaggedNon-Zero Enabled Tagged
48 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
8. NIC VLAN in Windows
VLAN ID in the packet's tag header.
If the priority value that the miniport driver retrieves from the packet is zero, the miniport driver should not mark the packet with a priority value before transmitting the packet over the network. On Ethernet (802.3) networks, an unmarked packet is the same as a packet that is marked with the "best effort" priority value appended to it. Not marking packets ensures backward compatibility with devices on the network that do not support the tag header.
Note: If a send packet's Ethernet header specifies a protocol type (EtherType) of 802.3ad or GARP VLAN Registration Protocol (GVRP), the miniport driver should transmit the packet without modifying it.
The following table indicates the circumstances under which a miniport driver should not send an 802.1Q-marked packet.
Table 14 Windows NIC VLAN Send Side Action Table
Configured NIC VLAN ID
Priority or VLAN ID in the Packet's OOB Data
Packet Tag HeaderVLAN ID Miniport Driver Action
Zero Priority and VLAN ID are zero.
Any value or not present. Send the packet with no changes.
Zero Priority and VLAN ID are nonzero.
Not present Insert tag header into packet. Set VLAN ID in tag header to the value of VLAN ID in OOB data.Send packet.
Nonzero VLAN ID is zero or matches the configured VLAN ID for the NIC.
Not present Insert tag header into packet. Set VLAN ID in tag header to the value of configured VLAN ID for the NIC.Send packet.
Nonzero VLAN ID matches the configured VLAN ID for the NIC.
Zero Set VLAN ID in tag header to the value of configured VLAN ID for the NIC.Send packet.
Nonzero Not present. Matches the configured VLAN ID for the NIC.
Send the packet with no changes.
Table 15 Miniport Driver Packet Send Conditions
Configured NIC VLAN ID
Priority or VLAN ID in Packet's OOB Data
Packet Tag Header VLAN ID Miniport Driver Action
Nonzero VLAN ID does not match configured VLAN ID for NIC.
Any value or not present
Fail the send.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 49
8. NIC VLAN in Windows
8.3 NIC VLAN as Receive SideThe following table summarizes the actions that a miniport driver should perform on an 802.1Q receive packet.
Nonzero Not present Does not match the configured VLAN ID for the NIC.
Fail the send.
Any value Nonzero Nonzero Fail the send because VLAN ID should not be set in both OOB data and ag header.
Table 16 NIC VLAN as Receive Side
Configured NIC VLAN ID Tag Header in Packet Packet Tag Header
VLAN ID Miniport Driver Action
Zero Yes Any value. Remove priority and VLAN ID from the packet. Insert removed priority and VLAN ID into per-packet information.Indicate the packet.
Zero No Not applicable. Set the priority and VLAN ID in per-packet information to zero Indicate the packet.
Nonzero Yes Matches the configured VLAN ID.
Remove priority and VLAN ID From the packet.Insert removed priority and VLAN ID into per-packet Information.Indicate the packet.
Nonzero Yes Does not match the configured VLAN ID.
Do not indicate the packet.
Nonzero No Not applicable. Do not indicate the packet.
Table 15 Miniport Driver Packet Send Conditions
Configured NIC VLAN ID
Priority or VLAN ID in Packet's OOB Data
Packet Tag Header VLAN ID Miniport Driver Action
50 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
9. Troubleshooting Guidelines
9 Troubleshooting Guidelines
9.1 Unable to Boot the Host System with the NIC Card InstalledRemove the NIC and reboot. Does the system boot without the NIC?
If so, power down the system, insert the NIC into a different PCIe slot and boot-up the system.
Is the recommended minimum memory installed on the host system?
Note any error message during the OS boot sequence.
Check for system error messages:
Dbgview.exe (Windows event viewer)
dmesg (Linux)
Use the NetXen diagnostics utility for further testing.
9.2 Unable to ping the NIC Installed on the HostType lspci -v and look for vendor ID ‘4040’.
- Is the NIC visible on the PCI bus?
- Is the NIC is placed in the PCIe x8 slot?
Check if the driver is loaded correctly and the card is initialized
- dmesg (display system messages on Linux)
- Windows Event Viewer tool
Run the NetXen user diagnostics
Check the NIC XFP module and optical cables; For CX4, check the IB cable
Check Network Configuration
- Check for valid IP Address and subnet
- Check for duplicate IP address and subnet numbers
- If connected via gateway, check gateway address and network path
- Check the NIC MAC address values
- Network link status as seen by the host OS
- Check for LED activity (using ping)
- Using network monitoring tools, analyze the Tx and Rx packets for the NIC
- Verify the ARP entry and response
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 51
9. Troubleshooting Guidelines
9.3 Gather System Information and Check for Errors on Linux HostNetXen Product Information
- Card type
- NIC firmware and driver version
Operating System Information
- Windows OS (32 / 64) and Service Pack #
- Linux Distribution and Kernel Release (uname –a)
Host System Information
- CPU, Cores, Chipset – (Run System Information tool)
- BIOS information
- Host memory
- PCIe x8, x4, x1 …
- System setup description
- How many servers, clients etc
- Model of the host system
- Network setup (back to back, router, switch, backplane etc)
Check for error messages from the kernel:
- Type dmesg
- Type cat /var/log/messages
Check routing table, gateway, and IP Address conflicts for all interfaces:
- Type route
Perform a "cat /proc/net/nx_nic/dev0 | grep eth" and "cat /proc/net/nx_nic/dev1 | grep eth" to find the correct "ethx" device name for each of the two ports on the NetXen Mezzanine card.
Check network interface TX and RX counts during pings:
- Type ifconfig <network interface>
Use ethereal, tethereal, or tcpdump utilities to capture packets:
- Analyze the ping packets
9.4 CaveatsJumbo frames of maximum size 8000 are supported.
The current version of the Windows Win2k3 OS does not support MSI in the network drivers (NDIS 5.1). However, hardware support is provided for NDIS versions 5.2 and 6.0.
52 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
9. Troubleshooting Guidelines
9.4.1 MSI Interrupts for LinuxThe Linux OS version supports MSI, and is used by the Intelligent NIC driver on the Linux OS.
On the Linux host, type cat /proc/interrupts to check that PCI-MSI is enabled for the Ethernet interface
Determine if the installed Linux kernel has the MSI configuration option enabled (earlier versions of the kernel may require an MSI patch).
Type make menuconfig from the directory /usr/src/kernels/<kernel version> and verify that MSI configuration option is supported/enabled.
NetXen device ID = 4040
$ lspci -d 4040
9.4.2 PCI-Express SlotThe NetXen reference card can be located on the PCI-Express bus using the following Linux procedures.
Determine Vendor and Device IDTo check if the PCIe slot has an x8 link for optimal PCIe performance, first determine the PCI vendor ID and device ID codes using the following command:
% lspci -n
A message similar to the following should be displayed:
06:00.0 0200: 4040:0001 (rev 24)
The NetXen vendor ID is 0x4040. A list of NetXen device ID’s is shown below.
Determine Interface AttributesTo determine the attributes of the interface, execute the following command:
lspci -n -d 4040:0004 -vvv | grep -i width
A message similar to the one below should be displayed.
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0
Link: Speed 2.5Gb/s, Width x8
Table 17 NetXen Device IDs
Product Device ID Description
NXB-10GXSR 0x0001 10 Gbit Ethernet NIC with XFP optical interface.NXB-10GCX4 0x0002 10 Gbit Ethernet NIC with CX4 copper interface.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 53
9. Troubleshooting Guidelines
Determine Width ParametersCheck if the lspci output "negotiated width" (the second width in the output) reflects the actual "supported width". In an XFP system for example, execute the following command to determine if these two widths are the same:
% lspci -n -d 4040:0004 -xxx
where 4040 indicates the NetXen vendor ID and 0004 indicates an IBM IMezz card device ID. An ID of 0001 indicates a NetXen XFP card.
Examine the following hex output:
From the lspci hex output:
d0: 10 00 01 00 02 80 00 00 40 20 01 00 81 04 00 00 /\
||_________ capability. "81 04" = x8
e0: 01 00 81 00 00 00 00 00 00 00 00 00 00 00 00 00 /\
||_________ negotiated. "81 04" = x8. "41 04" = x4, "11 04" = x1, "01
05" = x16
Note that some PCI-e chip sets with a x16 link may not support a x8 PCI-Express link width. A link is only required to support its maximum width and x1. Check the motherboard specification to determine if its PCI-e x16 slot can autonegotiate to x8.
On some motherboards, the PCI-e slots are all the same physical size required by a x16 slot, but electrically they are different widths. Check the motherboard specification for the supported width of each of the PCI-e slots.
9.5 Additional SupportIf problems are encountered with server applications when either Chimney or RSS are enabled under Microsoft Win2K3 with SP2 (in Offload Mode), please follow the link below to see if Microsoft has addressed the specific issue.
http://support.microsoft.com/kb/936594
...... 81 04 00 00 01 00 81
Supported link width
Negotiated link width
54 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
10. User Diagnostics
10 User DiagnosticsThe user diagnostics can be invoked in Windows and Linux using the following command:
nxudiag.exe -i NX_NIC (Windows)
[version]/bin/nxudiag -i eth[n] (Linux)
NOTE: If the eth[n] number is not known, perform a "cat /proc/net/nx_nic/dev0 | grep eth" and "cat /proc/net/nx_nic/dev1 | grep eth" to find the correct "ethx" device name for each of the two ports on the IBM NetXen Mezzanine card.
10.1 Windows Tests The Windows user diagnostics utility performs the following tests using the nxudiag.exe command.
Table 18 NetXen Windows Diagnostic Utility Commands
Command Description
-i--interface
Specifies the interface type [NX_NIC, NX_NIC1, etc.]
-a--all
Perform all test, regardless of default.
-n--nodefault
No defaults, all tests will be set implicitly.
-R--CRegs
Test all control registers.
-F-Flash
Test the Flash.
-I--IRQS
Test interrupt mechanism.
-L--IntLB
Internal loopback test
-S--LinkST
Link status test.
-I--LED
LED test.
-nR--noCRegs
No control registers test.
-nF--noFLASH
No Flash test.
-nl--noIRQS
No interrupt test.
-nL--noIntLB
No internal loopback test.
-nS--noLinkSt
No Link status test.
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 55
10. User Diagnostics
10.2 Linux Tests The Linux user diagnostics utility performs the following tests using the nxudiag.exe -i eth[n] command.
-nl--noLED
No LED status test.
-E--ExtLB
External full duplex loop-back test.
-h--help
Prints the help message.
Table 19 NetXen Linux Diagnostic Utility Commands
Command Description
-i--interface
Specifies the interface type [eth1, eth2, etc]
-a--all
Perform all test, regardless of default.
-n--nodefault
No defaults, all tests will be set implicitly.
-s--silent
Silent mode. No output to console.
-v--verbose
Verbose mode. Output result to console.
-R--CRegs
Test all control registers.
-F-Flash
Test the Flash.
-I--IRQS
Test interrupt mechanism.
-L--IntLB
Internal loopback test
-H--Hw
Hardware test.
-S--LinkST
Link status test.
-nR--noCRegs
No control registers test.
-nF--noFLASH
No Flash test.
-nl--noIRQS
No interrupt test.
Table 18 NetXen Windows Diagnostic Utility Commands
Command Description
56 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
10. User Diagnostics
-nL--noIntLB
No internal loopback test.
-nH--noHw
No hardware test.
-nS--noLinkSt
No Link status test.
-E--ExtLB
External full duplex loop-back test.
-C--Cmdif
Command interface status test.
-h--help
Prints the help message.
Table 19 NetXen Linux Diagnostic Utility Commands
Command Description
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 57
Appendix A. Tuning Guidelines
Appendix A Tuning Guidelines
A.1 Windows Performance Tools
A.1.1 NTttcp (Windows)There are two switches on ntttcp that boost performance: -a and -rb.
Running ntttcp without an optimal value for the -a switch and without the -rb switch is acceptable for the Gigabit NIC product.
In the 10G NIC product, however, and in particular with fewer threads on Windows, performance is limited by the Receive window for the most part and by the outstanding I/O (-a) value, regardless of hardware limitations.
For receive, an optimal value is 16 for -a and 512000 for -rb (or something similar). With these values, a single thread should perform closer to a multi-thread configuration.
ntttcpr -m 16,0,<rcv ipaddr> -n <# of tcp segs for snd/rcv> -a 16
For transmit, -a should be a value less than 8. The more threads, the lower the number should be.
Note: The send and receive side NTTTCP commands must use the receive side’s IP address.
Steps for NIC Performance Testing using NTTTCP on Windows
1 Install Win2k3 SP2
2 Enable Windows Scaling
Set TCp1323Opts parameter value to 1, if the parameter does not exist, create it:
"regedit" to HKLM\system\currentcontrolset\services\Tcpip\Parameters
3 The drivers property sheet default is 1500. For Jumbo, set the frame size to 8000.
Table 20 ntttcpr Command Options
Command Optionsntttcpr Description
-l Message size in bytes. Default is 64K.-m Number of threads.0 CPU number.IP
AddressReceive IP address.
(Send side also uses receive side's IP Address).-n Number of TCP segments.-a Number of requests in parallel.
(Send and receive side values must match).-rb Receive window size on the receiver.
58 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
Appendix A. Tuning Guidelines
Unidirectional: One Server to One ClientReceiver:
ntttcpr.exe -m 16,1,<Receive Side IPAddr> -a 16 -p 5001 -fr -n 200000
Sender:ntttcps.exe -m 16,1,<Receive Side IPAddr> -a 16 -p 5001 -n 200000
The above commands can be used in both Chimney and non-Chimney modes. With Chimney, enabled performance is higher and CPU utilization is also reduced.
Unidirectional: One Server to One Client — Single Thread, Multiple ProcessorsIf the system has more than one CPU, the load can be distributed across two CPUs and port numbers.
Send Side
ntttcps -m 1,0,192.168.20.2 -n 300000 -a 6 -p 5001ntttcps -m 1,1,192.168.20.2 -n 300000 -a 6 -p 5002
Receive Side
ntttcpr -m 1,1,192.168.20.2 -n 300000 -a 16 -rb 512000 -p 5001ntttcpr -m 1,1,192.168.20.2 -n 300000 -a 16 -rb 512000 -p 5002
A.1.2 nttcps and ntttcpr Command Line Options
C:\NTTTCP_NEW>ntttcps <parameters>orC:\NTTTCP_NEW>ntttcpr <parameters>
Version 2.6ntttcpr.exe: [-l|-n|-p|-a|-x|-rb|-sb|-i|-f|-u|-w|-d|-t|-v|-6|-fr|-mb] -m <mapping> [mapping]
-l <Length of buffer> [default: 64K] -n <Number of buffers> [default: 20K] -p <Port base> [default: 5001] -a [outstanding I/O] [default: 2] -x [PacketArray size] [default: 1] -rb <Receive buffer size> [default: 64K] -sb <Send buffer size> [default: 0] -i Infinite Loop [Only UDP mode] -f <File Name> [default: output.txt] -u UDP send/recv -w WSARecv/WSASend -d Verify Flag -t <Runtime> in seconds -v enable verbose mode -6 enable IPv6 mode
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 59
Appendix A. Tuning Guidelines
-fr Full buffers posted on reads -mb Multiple buffer post mode -m <mapping> [mapping] where a mapping is a session(s),processor,receiver IP set e.g. -m 4,0,1.2.3.4 sets up: 4 sessions on processor 0 to test a network on 1.2.3.4
A.2 Windows Optimization — Tcp1323OptsThis parameter determines whether TCP uses the time-stamping and window scaling features described in RFC 1323, TCP Extensions for High Performance.
Window scaling permits TCP to negotiate a scaling factor for the TCP receive window size, allowing for a very large TCP receive window of up to 1 Gbyte. The TCP receive window is the amount of data the sending host can send at one time on a connection.
To set the window scaling and timestamp parameters, click Start -> Run, then type regedit to start the registry editor. Open the following location:
HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Window ScalingFor more efficient use of high bandwidth networks, a larger TCP window size may be used. The TCP window size field controls the flow of data and is limited to 2 bytes, or a window size of 65,535 bytes.
Since the size field cannot be expanded, a scaling factor is used. TCP window scale is an option used to increase the maximum window size from 65,535 bytes to 1 Gigabyte.
When the value for window size is added to the registry and its size is larger than the default value, Windows attempts to use a scale value that accommodates the new window size.
Data Type Range Recommended Value
REG_DWORD 0 | 1 | 2 | 3 1
Hex Value Meaning
0 Timestamps and window scaling are disabled. 1 Window scaling is enabled. 2 Timestamps are enabled. 3 Timestamps and window scaling are enabled.
60 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
Appendix A. Tuning Guidelines
A.3 Linux Performance ToolsOnce connectivity between the two cards is established, common performance tools can be used for performance measurements.
Note: Various factors including tools, kernel, stacks, and related parameters affect the performance of a system. A discussion on performance criteria is beyond the scope for the getting started section.
Add the following for performance tuning on Linux systems under --
/etc/sysctl.conf
A.3.1 IPERFThe -P option determines the number of threads; default is one.
Server Side:
iperf -s -w256k -l 56k
Client Side:
iperf -c <ipAddr of Server> -w256k -P12 -l 56k
A.4 Linux OptimizationWhile optimization of the Linux operating system is a topic beyond the scope of this guide, below are a few common guidelines.
PCIe x8 provides optimal performance.
MSI interrupts are preferred over legacy interrupts.
Linux Kernel versions 2.6.15 or later provide better performance optimization for the NIC.
Check CPU utilization on the host for CPU-bound bottlenecks.
net.core.rmem_max = 16777216net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216net.ipv4.tcp_wmem = 4096 87380 16777216
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_timestamps = 0net.ipv4.tcp_sack = 1net.ipv4.tcp_rmem = 4096 87380 16777216net.ipv4.tcp_wmem = 4096 87380 16777216net.ipv4.tcp_low_latency = 0net.ipv4.tcp_window_scaling = 1net.ipv4.tcp_moderate_rcvbuf = 1net.ipv4.route.flush = 1
net.core.optmem_max = 524287
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 61
Appendix A. Tuning Guidelines
The tunable performance parameters are system and application specific. Understanding the relationship between the parameter values, target workload, performance applications, OS and system configuration provides a better understanding of performance optimization and bottlenecks.
62 Intelligent NIC Installation and Tuning Guide NetXen, Inc.
Appendix B. Glossary
Appendix B Glossary
Name Description
Bonding Used in a system with multiple ports to implement fault tolerance in Linux.
dbgview.exe Debug viewer for Windows
dmesg Linux system messages
ethereal Packet analyzer
ethtool Linux Ethernet tool for card settings. See man ethtool in Linux
imezz_romimage NetXen firmware image for IBM Mezzanine card.
iperf Linux performance tool
LKM Linux Kernel Module
Load balancing Allows the load to be balanced between 2 or more ports in a system.
LSA Linux Socket Accelerator
NIC Network Interface Card
nxflash NetXen Flash utility
nx_intercept.ko Linux LSA intercept driver
nx_lsa.ko Linux LSA driver
nx_nic.ko Linux NIC driver
nxp2nic.sys Windows NIC driver
nxudiag NetXen user diagnostic utility
Teaming Used in a system with multiple ports to implement fault tolerance in Windows.
xfp_romimage NetXen firmware image for an XFP card
NetXen, Inc. Intelligent NIC Installation and Tuning Guide 63
Appendix B. Glossary
64 Intelligent NIC Installation and Tuning Guide NetXen, Inc.