Server FundamentalsServer Fundamentals

The performance of a server depends The performance of a server depends on the speed of data flow between its on the speed of data flow between its processor and the peripheral device processor and the peripheral device that are connected. These peripheral that are connected. These peripheral device use a set of pathways to device use a set of pathways to communicate with the processor. communicate with the processor. This set of pathways are called as This set of pathways are called as BUS. In server we mainly use PCI bus BUS. In server we mainly use PCI bus technology.technology.

Types of BusesTypes of Buses

System BusSystem Bus Data BusData Bus

System Bus System Bus

Use to connect CPU to its MEMORY.Use to connect CPU to its MEMORY. Also known as Local BusAlso known as Local Bus Faster than other Buses.Faster than other Buses.

Data BusData Bus

Connects the other computer Connects the other computer peripherals with the memory.peripherals with the memory.

PCI is an example of data bus.PCI is an example of data bus.

Usage of PCIUsage of PCI

In the early 1990, INTEL introduce In the early 1990, INTEL introduce PCI with the aim to improve I/O PCI with the aim to improve I/O operation of computers.operation of computers.

PCI is also used with other PCI is also used with other processors, eg. AMDprocessors, eg. AMD

PCI DesignPCI Design

Initially PCI was a 32 bit bus that Initially PCI was a 32 bit bus that operated a speed of 33 MHz operated a speed of 33 MHz (DTR=132Mbps)(DTR=132Mbps)

Now 64 bit PCI with a speed of 33 Now 64 bit PCI with a speed of 33 MHz whose DTR=264 MbpsMHz whose DTR=264 Mbps

64 bit PCI with a speed of 66 MHz, 64 bit PCI with a speed of 66 MHz, with DTR=512 mbpswith DTR=512 mbps

Bus Speed SetupBus Speed Setup

Synchronized--- bus speed is not Synchronized--- bus speed is not depends with the memory bus depends with the memory bus speed.speed. Mainly used in today's bus architectureMainly used in today's bus architecture

Asynchronized--- bus speed depends Asynchronized--- bus speed depends with the memory bus speed. with the memory bus speed. Normally controlled by jumpers or Normally controlled by jumpers or CMOS setup.CMOS setup.

PCI ImplementationPCI Implementation

Data buses connect to system buses Data buses connect to system buses by single chip bridge device. by single chip bridge device.

PCI can be implemented on servers PCI can be implemented on servers through peer PCI bus or hierarchical through peer PCI bus or hierarchical PCI bus technology.PCI bus technology.

Peer PCI BusPeer PCI Bus In Peer PCI, several PCI buses are In Peer PCI, several PCI buses are

connected directly to a host bus. These connected directly to a host bus. These host buses provide connectivity between host buses provide connectivity between server and storage devices.server and storage devices.

This enables the CPU to access each bus This enables the CPU to access each bus directly. directly.

Peer PCI improves the performance of a Peer PCI improves the performance of a server because the processor or the bus server because the processor or the bus bandwidth is evenly shared between the bandwidth is evenly shared between the buses. buses.

Also helps to separate high bandwidth and Also helps to separate high bandwidth and low bandwidth traffic on different PCI low bandwidth traffic on different PCI buses.buses.

Hierarchical PCIHierarchical PCI

When multiple expansion slots and When multiple expansion slots and connectors are needed hierarchical connectors are needed hierarchical PCI can be implemented.PCI can be implemented.

A bridge controller is used to A bridge controller is used to connect independent PCI buses so connect independent PCI buses so that they can communicate with that they can communicate with each other.each other.

Hot plugHot plug

The hot plug facility of PCI enables The hot plug facility of PCI enables to install and remove PCI device to install and remove PCI device without turning off the server.without turning off the server.

Today most servers are equipped Today most servers are equipped with the hot plug facility. with the hot plug facility.

Microsoft Windows Server 2003 Microsoft Windows Server 2003 have native support for hot plug have native support for hot plug operations.operations.

Hot Plug DevicesHot Plug Devices

NIC CardNIC Card SCSI CardSCSI Card RAID CardRAID Card

Hot SwapHot Swap

Cards inserted or remove without Cards inserted or remove without failure when servers are running.failure when servers are running.

To support hot swap PCI hot swap To support hot swap PCI hot swap adapters are required. These device adapters are required. These device provide circuit breaker functions provide circuit breaker functions that protect electric supplies against that protect electric supplies against fault conditions resulting from over fault conditions resulting from over current. A PCI hot swap adapter current. A PCI hot swap adapter provides protection against 12V, 5V, provides protection against 12V, 5V, 3.3V and -12V3.3V and -12V

PCI- Hot Swap LevelsPCI- Hot Swap Levels

Basic hot swapBasic hot swap Full hot swapFull hot swap High availabilityHigh availability

Basic Hot Swap VS Full Basic Hot Swap VS Full Hot SwapHot Swap

Requires human intervention for Requires human intervention for interaction with the host operating interaction with the host operating system when a card is being inserted system when a card is being inserted or removed. or removed.

It automates the procedure of It automates the procedure of replacing the card. replacing the card.

High Availability High Availability

The high availability level focuses on The high availability level focuses on having cards that can be logically having cards that can be logically and electrically added to or removed and electrically added to or removed from a server. from a server.

This ensure hot spare or fail over This ensure hot spare or fail over capability for the server. The high capability for the server. The high availability hot swap level also availability hot swap level also enables an automatic reconfiguration enables an automatic reconfiguration of software and hardware of software and hardware components in a running server. components in a running server.

I2O I2O PCI is also intelligent input/output (I2O) PCI is also intelligent input/output (I2O)

compliant. compliant. I2O is a standard designed by a consortium of I2O is a standard designed by a consortium of

computer companies called as computer companies called as I2O Special I2O Special Interest Group (SIG). This standard simplifies Interest Group (SIG). This standard simplifies and speed up the I/O operations on servers. and speed up the I/O operations on servers.

According to I2O the drivers used for the According to I2O the drivers used for the operating system of a server must be same as operating system of a server must be same as those used for the SCSI and network cards on those used for the SCSI and network cards on the server. the server.

It is recommended to use INTEL 960 chip on the It is recommended to use INTEL 960 chip on the motherboard of a server, to increase the speed of motherboard of a server, to increase the speed of its I/O subsystem.its I/O subsystem.

PCI- Bus MasteringPCI- Bus Mastering

Bus mastering is the capability of devices Bus mastering is the capability of devices other than the system chipset, on the PCI other than the system chipset, on the PCI bus to take control of the bus and perform bus to take control of the bus and perform transfer directly.transfer directly.

Bus mastering allows the transfer of data Bus mastering allows the transfer of data between devices on a PCI bus and the main between devices on a PCI bus and the main system memory even when the other system memory even when the other devices are using CPU. However these devices are using CPU. However these devices should have their own built-in devices should have their own built-in processors to operate independently of the processors to operate independently of the CPU. CPU.

MULTITASTKINGMULTITASTKING

Bus Mastering facilitates multitasking Bus Mastering facilitates multitasking by allowing data transfer between the by allowing data transfer between the devices on a PCI bus and other devices devices on a PCI bus and other devices in the system without involving the in the system without involving the CPU. CPU.

The devices that take control of the bus The devices that take control of the bus keep track of the transactions. There keep track of the transactions. There fore the CPU utilization is low. This fore the CPU utilization is low. This enables the CPU to handle other task in enables the CPU to handle other task in the meantime. the meantime.

Bus Arbitration Bus Arbitration

Bus Mastering is achieved through the bus Bus Mastering is achieved through the bus arbitration process. This process arbitration process. This process determines priority levels for the request to determines priority levels for the request to a PCI bus so that no devices lock each other a PCI bus so that no devices lock each other out.out.

For example, when more than one For example, when more than one processor is connected to the same system processor is connected to the same system bus, there are simultaneous requests for the bus, there are simultaneous requests for the bus. This request are resolved using bus bus. This request are resolved using bus arbitration that allows devices to use the arbitration that allows devices to use the bus based on their priority level.bus based on their priority level.

PCI-ePCI-e

The Peripheral Component The Peripheral Component Interconnect Express architecture is Interconnect Express architecture is a serial interconnect technology a serial interconnect technology designed to meet the bandwidth designed to meet the bandwidth requirement of advanced processors, requirement of advanced processors, memory subsystem and applications. memory subsystem and applications.

It impose some advance features It impose some advance features than a normal PCI.than a normal PCI.

PCI-e PCI-e

PCI-e is a point-to-point serial switch bus PCI-e is a point-to-point serial switch bus technology. It aims at reducing technology. It aims at reducing implementation costs and providing implementation costs and providing higher bandwidth per pin and a scalable higher bandwidth per pin and a scalable performance. performance.

The advantage of PCI-e over PCI is its The advantage of PCI-e over PCI is its point to point bus topology. It enables point to point bus topology. It enables each PCI-e device to communicate the PCI each PCI-e device to communicate the PCI controller directly and autonomously. This controller directly and autonomously. This communication occurs through a PCI link.communication occurs through a PCI link.

PCI-e linkPCI-e link

A PCI-e link comprise of one or more A PCI-e link comprise of one or more lanes. Depending on the number of lanes. Depending on the number of lanes used in a PCI-e link the link is lanes used in a PCI-e link the link is called as x1 link, x2 link and so on. A called as x1 link, x2 link and so on. A link which made up for 4 lanes is link which made up for 4 lanes is known as x4 link.known as x4 link.

PCI-e supports x1,x2,x4,x8,x12,x16 PCI-e supports x1,x2,x4,x8,x12,x16 and x34 lane widths.and x34 lane widths.

PCI-e BandwidthPCI-e Bandwidth

Each PCI-e lane can transfer one Each PCI-e lane can transfer one byte at a time in both the directions byte at a time in both the directions by using full duplex communication. by using full duplex communication. An individual lane can transfer data An individual lane can transfer data at the rate of 2 Gbps in each at the rate of 2 Gbps in each direction simultaneously. direction simultaneously.

The next table shows its detail.The next table shows its detail.

PCI-e bandwidthPCI-e bandwidthPCI-e PCI-e implementatiimplementationon

Encoded data Encoded data transfer ratetransfer rate

Unencoded Unencoded data transfer data transfer raterate

x1x1 5 gbps5 gbps 4 gbps4 gbps

x2x2 10 gbps10 gbps 8 gbps8 gbps

x4x4 20 gbps20 gbps 16 gbps16 gbps

x8x8 40 gbps40 gbps 32 gbps32 gbps

x16x16 80 gbps80 gbps 64 gbps64 gbps

x32x32 160 gbps160 gbps 128 gbps128 gbps

8b/10b Clock-encoding 8b/10b Clock-encoding mechanismmechanism

A PCI-e mechanism comprises dual A PCI-e mechanism comprises dual simplex channels, where each channel simplex channels, where each channel is composed of a transmit pair and a is composed of a transmit pair and a receive pair for concurrent transmission receive pair for concurrent transmission in both directions. Every pair is in both directions. Every pair is composed of dual pairs of signal that composed of dual pairs of signal that are driven by Low Voltage Differential are driven by Low Voltage Differential (LVD). To attain a high speed data (LVD). To attain a high speed data transfer rate, a data clock is built into transfer rate, a data clock is built into each pair by using an 8b/10b data each pair by using an 8b/10b data encoding scheme.encoding scheme.

Point to Point Bus Topology Point to Point Bus Topology of PCI-eof PCI-e

As PCI-e supports the point to point As PCI-e supports the point to point bus topology the total bandwidth of bus topology the total bandwidth of each PCI bus is dedicated to the each PCI bus is dedicated to the device at the end of the link. This in device at the end of the link. This in turn enables multiple PCI-e devices turn enables multiple PCI-e devices to be active at the same time without to be active at the same time without affecting each other.affecting each other.

Advanced Features of Advanced Features of PCI-ePCI-e

Advanced power managementAdvanced power management Support for real-time data trafficSupport for real-time data traffic Hot plug and hot swapHot plug and hot swap Data integrity and error handlingData integrity and error handling

Advanced Power Advanced Power ManagementManagement

PCI-e maintains an active interface PCI-e maintains an active interface constantly to synchronize the constantly to synchronize the transmitter and receiver. The interface transmitter and receiver. The interface is kept active continually transmitting is kept active continually transmitting idle characters when data is not sent. idle characters when data is not sent. As additional power is consumed by As additional power is consumed by the receiver to decode and discard the receiver to decode and discard these idle characters, PCI-e uses two these idle characters, PCI-e uses two low-power link states and the active-low-power link states and the active-state power management (ASPM) state power management (ASPM) protocol.protocol.

Support for Real-Time Data Support for Real-Time Data TrafficTraffic

PCI-e provides native support for PCI-e provides native support for isochronous data transfers and different isochronous data transfers and different Quality of Service (QoS) levels through Quality of Service (QoS) levels through virtual channels. These channels ensure virtual channels. These channels ensure timely and reliable delivery of data packets. timely and reliable delivery of data packets. PCI-e supports numerous isochronous PCI-e supports numerous isochronous virtual channels per lane, which may have virtual channels per lane, which may have varying QoS levels. This feature is varying QoS levels. This feature is developed for applications that require developed for applications that require real-time delivery such as real-time real-time delivery such as real-time audio and video.audio and video.

Hot Plug and Hot SwapHot Plug and Hot Swap

The hot plug and hot swap features The hot plug and hot swap features of PCI-e are extremely beneficial to of PCI-e are extremely beneficial to servers and portable computers. The servers and portable computers. The facility to hot plug I/O devices facility to hot plug I/O devices reduces server downtime when reduces server downtime when peripheral cards need to be installed peripheral cards need to be installed or replaced.or replaced.

Data Integrity and Error Data Integrity and Error HandlingHandling

PCI-e provides support for link-level data PCI-e provides support for link-level data integrity for all kinds of transactions and integrity for all kinds of transactions and data-link packets. As a result, it provides data-link packets. As a result, it provides end-to-end data integrity required by high end-to-end data integrity required by high availability application running on server availability application running on server systems. systems.

Additionally, fault isolation and recovery Additionally, fault isolation and recovery solutions can be enhanced by using the PCI solutions can be enhanced by using the PCI error handling and advanced error error handling and advanced error reporting and handling capabilities of a reporting and handling capabilities of a PCI-e.PCI-e.

PCI-XPCI-X

PCI-X was developed jointly by IBM, PCI-X was developed jointly by IBM, HP and Compaq and was approved HP and Compaq and was approved by PCI Special Interest Group(PCI-by PCI Special Interest Group(PCI-SIG). This group provides technical SIG). This group provides technical support, training and compliance support, training and compliance testing. Now PCI-X has become an testing. Now PCI-X has become an open standard that is used by all open standard that is used by all computer manufactures.computer manufactures.

PCI-X Data SpeedPCI-X Data Speed

The PCI-X design, uses one 64 bit The PCI-X design, uses one 64 bit slot that runs at 133 MHz while the slot that runs at 133 MHz while the other slots run at 66 MHz. This other slots run at 66 MHz. This enables data to be exchanged at the enables data to be exchanged at the speed of 1.6 GBps. speed of 1.6 GBps.

So using PCI-X technology existing So using PCI-X technology existing PCI bus can be increased from 133 PCI bus can be increased from 133 MBps to 1 GBps.MBps to 1 GBps.

PCI-X – backward PCI-X – backward compatibilitycompatibility

PCI-X is backward compatible. It PCI-X is backward compatible. It means, a PCI-X card can be installed means, a PCI-X card can be installed in a standard PCI slot. However, this in a standard PCI slot. However, this reduces the speed of the bus to 33 reduces the speed of the bus to 33 MHzMHz

PCI-X Fault TolerancePCI-X Fault Tolerance

A PCI-X system is more fault tolerant A PCI-X system is more fault tolerant than a Normal PCI system. When a than a Normal PCI system. When a faulty card is installed on a faulty card is installed on a computer, the PCI-X bus can re-computer, the PCI-X bus can re-initialize the faulty card or take it initialize the faulty card or take it offline before the computer fails.offline before the computer fails.

PCI-X power PCI-X power requirementsrequirements

Normal PCI----- 5 VOLTNormal PCI----- 5 VOLT PCI-X ----- 3.3 VOLTPCI-X ----- 3.3 VOLT

Comparison of PCI and PCI-Comparison of PCI and PCI-X SystemsX Systems

Bus Bus WidthWidth

Bus Bus FrequencFrequencyy

PCI/PCI-PCI/PCI-X X bandwidtbandwidthh

No. of No. of slots(PCIslots(PCI))

No. of No. of slots(PCIslots(PCI-X)-X)

32 bit32 bit 66 MHz66 MHz 264 264 MbpsMbps

44 N/AN/A

64 bit64 bit 66 MHz66 MHz 528 528 MBpsMBps

22 44

64 bit64 bit 100 MHz100 MHz 800 800 MBpsMBps

N/AN/A 22

64 bit64 bit 133 MHz133 MHz 1064 1064 MbpsMbps

N/AN/A 11

Adapter TeamingAdapter Teaming

Adapter Teaming is a process of Adapter Teaming is a process of combining two or more physical combining two or more physical adapters into a single logical adapters into a single logical adapter. This enables the network adapter. This enables the network administrator to assign a single IP administrator to assign a single IP address to the logical adapter.address to the logical adapter.

A software driver is usually used to A software driver is usually used to link the physical adapter to the link the physical adapter to the logical adapter.logical adapter.

Adapter Teaming- number Adapter Teaming- number of membersof members

An adapter team can have a minimum An adapter team can have a minimum of two and a maximum of eight of two and a maximum of eight adapters. The main advantage of a adapters. The main advantage of a adapter teaming is that it increases adapter teaming is that it increases server uptime. This is because if a server uptime. This is because if a physical adapter in a team fails, it does physical adapter in a team fails, it does not affect other adapters in the team as not affect other adapters in the team as the IP address is assigned to the the IP address is assigned to the logical adapter and not to a physical logical adapter and not to a physical adapter.adapter.

AFTAFT

Adapter teaming also enables Adapter teaming also enables Adapter Fault Tolerance (AFT) Adapter Fault Tolerance (AFT) against broken or loose or loose against broken or loose or loose cables, hubs or switch ports. cables, hubs or switch ports.

It can also provide fault tolerance to It can also provide fault tolerance to problems caused by a Peripheral problems caused by a Peripheral Component Interconnect (PCI) slot Component Interconnect (PCI) slot malfunction or breakdown in malfunction or breakdown in adapter hardware.adapter hardware.

Adapter Teaming – load Adapter Teaming – load distributiondistribution

Adapter teaming ensures that the Adapter teaming ensures that the failure of an adapter in a team does not failure of an adapter in a team does not affect the load distribution through affect the load distribution through failover capabilities. If any member in failover capabilities. If any member in an adapter team fails because of an adapter team fails because of problems in a network interface card problems in a network interface card (NIC), cable, switch port, or switch, the (NIC), cable, switch port, or switch, the load distribution is evaluated and then load distribution is evaluated and then reassigned to the other members of the reassigned to the other members of the team.team.

Primary AdapterPrimary Adapter

For an effective failover, a primary For an effective failover, a primary adapters must be specified in an adapters must be specified in an adapter team. The primary adapter adapter team. The primary adapter is specified based on the priorities of is specified based on the priorities of the adapters. the adapters.

Usually, the primary adapter is the Usually, the primary adapter is the adapter that has highest prorityadapter that has highest prority

Failover RestorationFailover Restoration

In adapter teaming, only a single active In adapter teaming, only a single active adapter transmits and receives traffic. adapter transmits and receives traffic.

The single adapter is usually the primary The single adapter is usually the primary adapter. However, if the primary adapter adapter. However, if the primary adapter fails, a secondary adapter takes control.fails, a secondary adapter takes control.

After the connection of the primary After the connection of the primary adapter is restored, the control is adapter is restored, the control is automatically passed back to the primary automatically passed back to the primary adapter.adapter.

ALBALB

Adapter teaming also enables Adapter Load Adapter teaming also enables Adapter Load Balancing (ALB). ALB provides fault Balancing (ALB). ALB provides fault tolerance by ensuring load balancing data tolerance by ensuring load balancing data traffic.traffic.

When load balancing takes place, all When load balancing takes place, all adapters in an adapter team share the adapters in an adapter team share the transmission load. transmission load.

To enable load balancing, all the adapters in To enable load balancing, all the adapters in the team should be connected to a switch the team should be connected to a switch and an IP Address must be assigned to the and an IP Address must be assigned to the adapter team.adapter team.

Thank youThank you