10-Gigabit Ethernet Switch Performance Testing · Ethernet (IEEE 802.3u) in 1995, 1-Gigabit Ethernet (802.3z) in 1998 to 10-Gigabit Ethernet (802.3ae) in 2002. The 10-Gigabit Ethernet

Enabling a Converged World™

10-Gigabit Ethernet SwitchPerformance Testing

White Paper

10-Gigabit Ethernet SwitchPerformance Testing

sample test plans included

Copyright © 1998-2004 Ixia. All rights reserved. The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as a commitment by Ixia. Ixia assumes no responsibility or liability for any errors or inaccuracies that may appear in this document. Ixia and the Ixia logo are trademarks of Ixia. All other companies, product names, and logos are trademarks or registered trademarks of their respective holders.

Ixia

26601 W. Agoura Road

Calabasas, CA 91302

Phone: (818) 871-1800

Fax: (818) 871-1805

Email: [email protected]

Internet: www.ixiacom.com

2 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

Contents

Abstract .....................................................................................................................................5Introduction ..............................................................................................................................5

10-Gigabit Ethernet LAN ...................................................................................................610-Gigabit Ethernet Metro ................................................................................................610-Gigabit Ethernet WAN ..................................................................................................6

What Is Inside a Multi-10GE Port Switch? ..............................................................................7Packet buffer .....................................................................................................................7Packet processing function ..............................................................................................7Classification tables ..........................................................................................................7Context memory ................................................................................................................7Traffic management function ...........................................................................................7Switch fabric ......................................................................................................................7

What Happens to a Packet When It Enters a 10GE Port? .....................................................8Storing the packet .............................................................................................................8Processing the packet .......................................................................................................8Metering and statistics recording .................................................................................. 10Packet modification ....................................................................................................... 10Traffic management ....................................................................................................... 10Switch fabric ................................................................................................................... 10

What Are the Challenges in Processing Packets at 10G Rates? ....................................... 11Stress Point 1: Ingress packet buffer ............................................................................ 11Stress Point 2: Packet classification ............................................................................. 11Stress Point 3: Traffic management ............................................................................. 13Stress Point 4: Crossbar switch and backplane interconnect .................................... 14Stress Point 5: Multicast replication and queues ........................................................ 14Stress Point 6: Control plane ......................................................................................... 15

Developing the Right Test Methodology .............................................................................. 16Test methodology example ............................................................................................ 16Module-level testing ....................................................................................................... 16System-level testing ....................................................................................................... 16Test methodologies ........................................................................................................ 16Anticipating the behavior of the 10GE switch .............................................................. 18

Ixia’s Test Solution ............................................................................................................... 20IxExplorer ........................................................................................................................ 20IxScriptMate .................................................................................................................... 20

Appendix: 10GE Testing Examples ...................................................................................... 221. Throughput Testing: RFC 2544 Throughput Test .................................................... 222. Testing for Latency: RFC 2544 Latency Tests ......................................................... 243. Testing how 10GE ports handle prioritized streams based on QoS parameters .. 254. 10GE OSPF Convergence Test .................................................................................. 27

Glossary ................................................................................................................................. 29Acknowledgements ............................................................................................................... 30

10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 3


10-Gigabit Ethernet Switch Performance Testing

Abstract The first generation of 10-Gigabit Ethernet (10GE) switches, introduced in early 2002 after IEEE 802.3ae standardization, proved to be subpar. Some switches barely reached 50 percent of line rate throughput, even for large packet sizes, and exhibited poor latency, limited feature sets, and high per-port costs.

The second generation of 10GE products has arrived, with substantial packet processing capabilities that enable additional services. Key functions in this

technology include performing the necessary packet classification, header modification, policing of flows, and queuing/scheduling — all at wire-speed rates.

This white paper reviews the common building blocks of a multiple line card 10GE switch, identifies various stress points within the switch, and offers various test methodologies to test these stress points.

Introduction Ethernet networking technology is now virtually ubiquitous; it has evolved from 10Base-T (IEEE 802.3) in 1983, Fast Ethernet (IEEE 802.3u) in 1995, 1-Gigabit Ethernet (802.3z) in 1998 to 10-Gigabit Ethernet (802.3ae) in 2002. The 10-Gigabit Ethernet standard provides a significant increase in bandwidth while maintaining compatibility with the installed base of 802.3-standard interfaces, to protect existing investments in Ethernet technology. The IEEE 802.3ae 10-Gigabit Ethernet standard defines both LAN and WAN physical layers, the latter of which is compatible with existing SONET infrastructure.

Ethernet not only dominates the LAN market, but is also taking hold in the Metro Area Network (MAN) market. It has extended into the WAN arena as both its distance and capacity has increased. Ethernet at 1-Gigabit and 10-Gigabit speeds has attracted increasing attention from carriers operating in the MAN and WAN market segments.

After nearly two years of slow growth during 2001 and 2002, worldwide Layer 2 and Layer 3 Ethernet switch revenue began to accelerate in mid-2003, and is expected to grow from $11.8 billion in 2004 to $15 billion in 2006, according to Infonetics Research's quarterly market share and forecast service, L2–L3 Ethernet Switches. Worldwide port shipments are predicted to grow 75 percent — from 161 million in 2003 to 279 million in 2006.

The strongest growth in the Ethernet switch market has come from 10GE chassis ports and revenue, driven by 10GE deployment in service provider metro networks, and by the proliferation of 1GE to desktops/laptops as a standard interface.

10-Gigabit Ethernet has been developed to support a wide range of applications — from the enterprise network, through the edge and metro network, and into the wide area network.


10-Gigabit Ethernet LANThe biggest market for 10-Gigabit Ethernet is likely to be in the corporate backbone. The cost of 1-Gigabit and 10-Gigabit Ethernet has dropped significantly, which will trigger the demand for 10GE services and products.

In the enterprise network, 10-Gigabit Ethernet is used for:

• Switch-to-switch links for very high-speed connections within the same wiring closet or data center or between different buildings.

• Aggregation of multiple-fiber 1000BASE-X or copper 1000BASE-T segments into 10GE uplinks. Nearly all new desktops and laptops are now shipped with 10/100/1000BASE-T ports.

• System Area Networking (SAN) applications providing server interconnects for clusters of servers. 10-Gigabit Ethernet plays a key role in interconnecting the new generation of high-performance servers with multi-gigahertz 2/4/8-way processors required for moving large amount of data between server farms.

10-Gigabit Ethernet MetroAt the Metro Area Network (MAN) level, service providers face tremendous pressure to expand capacity for broadband local access, high-speed WANs, storage, and campus /enterprise grids. Standards-based 10-Gigabit Ethernet enables service providers to extend their 10GE networks into the metro edge using their dark fiber, especially when 10GE is combined with metro area optical fiber networks based on Wave Division Multiplexing (WDM).

10GE MANs can be deployed in either a star or ring topology. Unlike Resilient Packet Ring (RPR) networks, 10GE networks use the standard Ethernet MAC protocol. The latest 10GE metro switches can provide network reliability similar to that of networks based on SONET/SDH rings.

10-Gigabit Ethernet WAN10GE WAN applications are similar to MAN applications, but leverage existing SONET networks rather than new infrastructure. SONET is the dominant transport protocol in the WAN backbone today, and most MAN public services are offered over SONET networks.

The 10-Gigabit Ethernet interface (WAN PHY) is compatible with SONET-based Time Division Multiplexing (TDM) and with the payload rate of OC-192c/SDH VC-4-64c. The 10-Gigabit WAN interface facilitates the transport of native Ethernet packets across the WAN transport network, with no need for protocol conversion. This not only improves the performance of the network, but makes it simpler and less costly to manage.

Ethernet’s flexibility and universal availability will fuel the demand for 10-Gigabit Ethernet deployment in multiple network segments, from corporate networks, to data centers, and throughout service provider networks. This ongoing demand will require system vendors to deliver high-performing 10-Gigabit systems that meet service providers’ stringent requirements — requirements that ensure high availability to corporate networks and performance that meets service level agreements (SLAs).


What Is Inside aMulti-10GE Port

Switch?

Testing the new generation of 10GE switches requires a clear understanding of the various building blocks that make up a switch, and their interaction, to determine the various stress points within a switch and identify the weakest link in the chain.

Figure 1 shows the major components of a 10GE switch, including the line card, the switch fabric card, and the controller card. The following subsections provide an overview of the 10GE switch/router components crucial to performance testing.

Packet bufferThe packet buffer is a temporary repository for arriving packets while they wait to be processed.

Packet processing functionThe packet processor is an optimized ASIC or programmable device (Network Processing Unit, or NPU) for processing and forwarding packets in the data plane or fast path. It performs specific key tasks such as parsing the header, pattern matching or classification, table look-ups, packet modification, and packet forwarding, ideally at wire speed.

Classification tablesThe classification table is a special memory used by the packet processor. It may contain the following databases:

• The routing table determines where to route incoming packets.

• Access Control Lists (ACLs) contain information to grant or deny permission to specific users or groups

• The flow classification table contains information about a particular user or group of users, protocols, and applications. Flow identification information is used to determine QoS

treatment, packet policing, and per-flow queuing and billing.

• The label table contains information about VLANs, stacked VLANs, MPLS labels, etc.

Context memoryContext memory contains instructions about whether to deny or forward packets, where to forward them, all the internal system headers needed to get a packet from an ingress to an egress port, and the external packet header (new MAC, IP, MPLS stacks, etc.). It also holds information relevant to metering the packet and other miscellaneous information about how to process a packet.

Traffic management functionThe traffic management function is used to regulate the flow of traffic. It forwards traffic according to a user-defined set of rules pertaining to priority levels, latency and bandwidth guarantees, and congestion levels. It also provides the buffering required to work with any queuing mechanisms it uses to manage traffic flow across the switch fabric. It may also include the high-speed SERDES (serializer/deserializer) function used to connect to the switch fabric.

Switch fabricThe switch fabric provides data plane interconnection among all line cards in the system. The switch fabric typically employs a crossbar to move packets between its ingress and egress ports. An NxN crossbar switch fabric allows N line cards in a system to be interconnected in a chassis. This allows each slot to simultaneously send and receive traffic over the switch fabric.


Figure 1. 10GE switch building blocks.

What Happens to aPacket When It

Enters a 10GE Port?

This section provides a high-level walkthrough of the “life of a packet” as it goes from a port on an ingress line card through the switch fabric card and exits on an egress line card. Figure 2 shows the logical operation of various line cards in a multi-10GE port switch, and identifies the stress points (numbered red circles).

Storing the packetWhen a packet arrives on a 10GE port, it is stored in ingress buffer memory while waiting to be processed by the packet processor. When the packet processor is ready, the packet header is copied into the packet processor memory for processing.

Processing the packetThe packet processor (ASIC or NPU) examines the packet to determine whether a packet should be filtered or forwarded. It makes this determination by parsing the packet header and then performing packet and flow classification.

The classification process maps information extracted from the packet header to information stored in local tables maintained by the control plane processor. The information in the classification tables typically includes forwarding tables, routing tables, and profiles or rules for a given packet or flow, such as policies for ACLs, Quality of Service (QoS), and Class of Service (CoS).

The classification typically points to many fragments of information about a packet. This information is usually saved in context memory and may contain instructions about whether to deny or forward the packet, where to forward it, all additional header information to get a packet from ingress to egress port, the new header used when it leaves the egress port (new MAC, IP, MPLS stacks, etc.), information relevant to policing the packet (see "Metering and statistics recording" below), and other information about how to process the packet.


Figure 2. Logical operation of line cards in a multi-10GE port switch with stress points.


Metering and statistics recording Packet metering checks the conformance of a flow to a subscribed traffic policy or contract. The metered information is then used to decide whether to discard or forward the packet. Bandwidth allocations may be established in contracts that specify the per-flow, steady state, and short term peak (burst) rates. Typically, a Token Bucket Algorithm is used at the ingress port to check the compliance of each flow to its contract. (A detailed discussion of the Token Bucket Algorithm is outside the scope of this document). The metered packets that violate the assigned contract can be marked for a range of treatments: unconditional drop, drop if no internal resources available, or pass on the packet dropping decision to the downstream device. The marking of packets can also be used by the egress line card to decide whether to delay packets within a packet stream in order to conform to some defined traffic profile, referred to as traffic shaping. Traffic shaping is used to smooth out packet flows and regulate the rate and volume of traffic admitted to the network.

Packet modificationBased on the outcome of the classification process, the next step is to modify the packet header and determine the packet’s egress port. Context memory contains instructions about whether to deny or forward packets, where to forward them, and all the internal system headers needed to get a packet from an ingress to an egress port. On the egress line card, a new external packet header (new MAC, IP, MPLS stacks, etc.) will be inserted by the egress packet processing function.

Traffic management The packet processor typically appends an internal forwarding header to each packet that it forwards to the traffic management device. This header contains information that tells the ingress/egress traffic

managers where to forward the packet (which port and line card) and the characteristics of the flow. The traffic manager then passes through or discards flagged packets, depending on its policies for the ingress direction.

Depending on the architecture, forwarded packets are organized into FIFO (First-In, First-Out) queues, input queues, or Virtual Output Queues (VOQs). Queued traffic is then scheduled for transmission to the switch fabric based on a scheduling algorithm — for example, Weighted Round Robin (WRR), Weighted Fair Queuing (WFQ), etc.

Depending on whether or not the switch fabric supports fixed or variable length packets, packet segmentation into fixed cells may occur before packets are transmitted onto the backplane. The ingress traffic manager typically interacts with the switch fabric and/or egress traffic manager to meet switch fabric scheduling requirements and avoid congestion.

On the egress side, the traffic manager first performs a reassembly operation if segmentation was performed on the ingress side. Received packets are then placed into an output queue based on information from the header that was appended at ingress. If per-flow queuing is supported, the egress traffic manager maintains an additional output queue for each flow. Each output queue provides traffic shaping to maintain conformance to QoS policy.

Switch fabricThe switch fabric functions as a high-performance data plane interconnect between its ingress and egress ports. Most new crossbar switch architectures have embedded high speed SERDES (serializer/deserializer) interconnecting line cards through centralized switch devices. A packet entering a switch is transferred to the switching engine. The switching engine


then examines the packet header and decides where to forward the packet.

When multiple ingress ports contend for an egress port, congestion becomes the primary issue at the egress switch port and, depending on the architecture, may cause Head-of-Line blocking (HOL). HOL blocking occurs with input queuing, when a cell (a segmented packet to be transmitted across the backplane) has to be held up, thereby blocking the progress of any cells

behind it, even if they could otherwise be switched.

Some switch architectures also provide multicast packet replication. To get around packet replication issues, most modern switch fabrics now allow for an incoming packet to be broadcast to any number of egress ports. Typically, on the ingress and egress line cards, separate queues and scheduling disciplines are maintained for unicast and multicast traffic.

What Are theChallenges in

Processing Packetsat 10G Rates?

This section examines the most common stress points within a switch, and discusses conditions that might push them toward a strained state. Refer to Figure 2.

Stress Point 1: Ingress packet bufferThe ingress packet buffer is a temporary repository for arriving packets waiting to be processed by the packet processor. Depending on the architecture and efficiency of the packet processor, data in the packet buffer could build up, resulting in intermittent (poor) latency, jitter, packet loss, or even service outage.

In most architectures, when packet buffers begin to fill beyond a preset threshold, the packet processor initiates flow control to the upstream MAC device, requesting it to stop passing packets. The MAC device then transmits a special packet requesting remote ports to delay sending packets for a period of time. This special packet is called a pause frame. This helps prevent buffer overflow, but it does not solve the packet loss problem completely. If the packet flow continues and the flow control signal is not removed by the packet processor before the MAC device’s buffer fills up, the MAC device will start dropping packets.

Generally, the buffer in any part of the system can build up for two reasons:

• Local or downstream devices are exceeding their allocated processing budget; or

• There is contention for resources — for instance, multiple ingress ports on a switch/router contending for an egress port.

Buffer buildup can create a chain reaction, leading to unpredictable behavior in a switch or router.

Another challenge in packet buffering for 10GE switches is dealing with back-to-back small packets. For example, at 10 Gbps speeds, arriving 64-byte packets must be deposited into buffer memory every 67 nanoseconds (ns), and departing packets must be retrieved from buffer memory every 67ns. Thus, to process a stream of back-to-back 64-byte packets, the packet buffer memory subsystem must support a write and a read every 67ns.

Stress Point 2: Packet classificationPacket classification is one of the most susceptible stress points. Classification maps information from the packet header to information stored in local tables maintained by a control plane processor (see "Stress Point 6: Control plane"). The packet processor parses various fields in the packet header to construct search keys. These keys are then used to address various tables. In most high-end architectures, Ternary Content Addressable Memory (TCAM) devices or equivalent technologies are used to map these keys to addresses, and are capable of holding millions of entries and


performing key searches in a matter of few internal clock cycles. However, complex applications or routing protocols may require multiple key searches to drive a look-up result. Furthermore, separate classification sequences may be required to determine what to do with a packet. For example, the packet processor may perform an ACL look-up first to decide whether to forward or deny a packet, and then do a route look-up to decide where to forward it. It might also perform a flow control look-up to provide enhanced services.

The required degree of packet parsing and processing is one of the main criteria that identifies the class of a switch/router. A simple Layer 2-3 switch only inspects the L2 header (i.e., MAC header, VLAN), and the L3 header (i.e., IPv4, IPv6), and, in some cases, performs limited flow classification.

Complex packets might require the classification of multiple L2-L3 headers for a given packet. Packets of a given protocol may be encapsulated within one or more tunnels of varying protocols, as shown in Figure 3. For example, a system supporting IPv6 over GRE requires two Layer 3 headers (IPv4 and IPv6) in addition to the Layer 2 MAC addresses. A more complex example is a Layer 2 VPN Martini Draft

packet with frames arriving over Ethernet in a wide range of dispositions, including IPv4 routing, MPLS switching or Ethernet bridging.

Flow classification. In addition to complex packet classification, flow classification might be required to provide enhanced services and policies. Flow classification provides a level of granularity that allows policies to be established based on the applications. Any number of combinations of Layer 3 and Layer 4 information could be employed to define the QoS or security policies that are then enforced.

A flow is a collection of one or more packet streams. In some classes of switches/routers, in addition to packet classification, flow classifiers perform stateful analysis of packets within the packet streams. Flow classifiers track the protocol state of each flow as the connection develops. This makes it possible to track control connections on well-known ports that spawn data connections on ephemeral ports. This is important, since many protocols establish connections and negotiate services on well-known Transmission Control Protocol (TCP) ports and then establish another ephemeral port to transfer the data for the network session.

Figure 3. Packets may be encapsulated in tunnels of varying protocols.

DestMAC

SrcMAC

TypeMPLSlabel

EXP S TTLDestMAC

SrcMAC

TypeIPv4

headerTCP/UDPdatagram

Ethernet over MPLS packet

DestMAC

SrcMAC

TypeIPv4

headerGRE

IPv6header

TCP/UDPdatagram

IPv6 over GRE packet


Look-ups and performance. Classification table information is typically stored in a look-up table, usually held in a large TCAM or equivalent technology. When processing encapsulated packets or packets with multiple L2/L3 headers (i.e., IPv6 over IPv4, or MPLS stacks with Ethernet header, IP header, and TCP) the classification process might require multiple accesses to the look-up table for each packet.

Table 1 shows the worst-case performance packet arrival rate for small packets. Depending on the packet arrival rate and number of required look-ups per packet, the packet processor or the classification device could become a resource bottleneck. For example, a back-to-back Ethernet packet with an IPv6 datagram requiring ACL and flow look-up might require 7–8 look-ups per packet. With 12 million packets arriving per second, this will require handling ~96 million look-ups/sec.

Stress Point 3: Traffic managementThe traffic management function provides advanced queuing, congestion management, and hierarchical scheduling of network traffic for large numbers of flows. It forwards traffic according to a user-defined set of rules pertaining to priority levels, latency and bandwidth guarantees, and varying congestion levels. It also provides the packet buffering required to work with any queuing mechanisms used to manage traffic flow

across the switch fabric. It may also include the SERDES function.

Communication from line card to switch fabric requires additional flow control information, creating overhead and increased bandwidth. This additional bandwidth is called speedup. See "Speed-up." on page 14.

The architecture of a switch/router can affect how the network behaves in times of heavy demand. An important concern is how packets are queued as they enter the switching fabric, that is, how traffic prioritization is handled and how different traffic flows are merged through the fabric.

Some products forward all high-priority packets before any lower-priority packets; this is called strict priority queuing. Other products use mechanisms such as Weighted Fair Queuing (WFQ) to statistically multiplex packets into the fabric. On the ingress line card, WFQ allows packets from lower priority queues to be interleaved with higher priority traffic into the switch fabric. This prevents the higher priority traffic from completely blocking the lower priority traffic, since the queues are guaranteed access to the switch fabric for a predefined proportion of the time.

Packet discarding is also part of traffic management. During times of congestion, the traffic manager may need to make discard decisions based on the availability of queue space, priority, or destination port, using a packet discard algorithm like

Table 1. Maximum packet arrival rates over 10-Gigabit Ethernet.

Wire rate LAN throughput for minimum-size packet

Ethernet IPv4 14.88 million packets/sec

Ethernet IPv6 12.02 million packets/sec

Ethernet Over MPLS IPv4 12.25 million packets/sec

Ethernet Over MPLS IPv6 10.25 million packets/sec


Random Early Detection (RED) or Weighted RED (WRED) for IP traffic.

Some routers and switches are fundamentally limited by the small number of queues they have for QoS. Small numbers of queues are common for class-based queuing, which limits prioritization and fairness. Class-based queuing typically prioritizes traffic based on the Layer 2 header (virtual LAN and source or destination MAC address, for example), rather than on higher level information such as application or protocol type.

Systems with large numbers of queues facilitate more granular prioritization and greater fairness. Queues established on a per-flow basis provide the possibility for each user session to get its own queue. However, a large number of CoS and QoS policies requires large amounts of memory and hierarchical schedulers capable of handling hundreds of thousands of flows.

Stress Point 4: Crossbar switch and backplane interconnectAlthough most switch architectures for modern systems are non-blocking, three types of blocking can limit performance when multiple ingress ports are contending for an egress port: Head-of-Line (HOL) blocking, input blocking, and output blocking. HOL blocking can waste nearly half a crossbar switch's bandwidth if the cells waiting at each input are stored in a single First-In, First-Out (FIFO) queue. Modern switch architectures employ Virtual Output Queuing (VOQ). VOQ, in conjunction with a scheduling algorithm, eliminates most blocking issues. These scheduling algorithms require the traffic manager device and the switch fabric to exchange information, including requests for permission to transmit, granting of permissions to transmit, and other information.

Speed-up. The additional bandwidth required to support VOQ and related scheduling algorithms is called speed-up.

A 10GE line card that supports 15 Gbps to the switch fabric offers 50 percent speedup. Speed-up is a common way to reduce input and output blocking by running the crossbar switch faster than the external line rate. For example, if the crossbar switch runs twice as fast as the external line, the traffic manager can transfer two cells from each input port, and two cells to each output port during each cell time.

The advantage of speed-up is obvious — it offers more predictable delay and jitter across the switch ports by delivering more cells per cell time, and thus reducing the delay of each cell through the switch. In fact, sufficient speed-up can guarantee that every cell is immediately transferred to the output port, where its departure time can be precisely scheduled.

Stress Point 5: Multicast replication and queuesThe biggest challenge in handling multicast packets is packet replication. Packet replication is generally accomplished in two stages. The first stage handles the branch replications from one ingress line card to multiple egress line cards. The second stage handles the leaf replications, and is typically accomplished on the egress line card. Depending on the switch architecture, the packet replication function could cause resource starvation in memory and CPU processing, as well as contention with unicast packets, as it accesses the data plane to forward the replicated packets.

New generation switches take advantage of the natural multicast properties of a crossbar switch and perform cell replication within the fabric by closing multiple cross points simultaneously. This method relieves the ingress line card from performing packet replication. However, the second-stage replication on the egress line card can still cause resource starvation and congestion.


Stress Point 6: Control planeThe control plane processor handles the routing and switching control plane, as well as many system management operations, such as user configuration, background diagnostics, statistics and alarm collection and reporting, network management, etc. This document focuses only on how the control and data plane interact for the purpose of packet processing.

The control plane processor runs the switch/router’s operating system and is responsible for the operation of network routing protocols (OSPF, BGP, IS-IS, IGRP), network management (SNMP), console port, diagnostics, etc. In a distributed architecture, where each line card has its own control plane processor, a master control plane processor typically generates, synchronizes, and distributes routing tables and other information among line cards for local forwarding decisions.

The control plane path interconnects the management processor(s) with various data plane blocks, to initialize, configure, perform diagnostics, and most importantly, to set up or update routing tables, Layer 2 tables, policies, and QoS/CoS tables.

The control processor can read/write to any location in the forwarding table, context memory, and other memories to support route removal and addition, table flushing in route flaps, and policy for a given flow.

The look-up and table management operation is asynchronous. The route table may be updated by the control plane processor while the packet processor is performing a look-up.

During ordinary system operation and moderately stressed conditions, the control plane would not be called upon to modify more than a few thousand routes per second in response to a routing protocol update, while the system is at the same time forwarding data plane packets. However, during error conditions, or a topology alteration known as route flapping or route convergence, hundreds of thousands of routes may be modified each second while packets are still arriving on each interface at up to line rate.

Therefore, determining how fast a switch/router can update its routing table requires test beds that can create/modify hundreds of thousands of route updates per second while performing normal data plane operation, that is, forwarding packets at line rate.


Developing theRight Test

Methodology

Understanding the stress points in a switch allows the development of a test methodology that can focus on testing these areas. The test methodology should address multiple layers, and could include measuring for:

• Wire speed unicast data throughput and latency for Layer 2/3 traffic.

• The ability to filter packets at wire-speed based on MAC addresses, IP addresses, TCP or UDP ports, or a combination of these (N-tuple).

• The ability to perform prioritization based on QoS marking.

• The ability to police traffic based on user-defined rate limits.

• The ability to handle Head-of-Line blocking (HOL).

• Wire speed multicast performance.

Test methodology exampleFollowing is an example of a test methodology for a multi-10GE port switch that supports the following features:

1. Wire-rate Layer 2/3 (for IPv4) switching with a minimum packet size of 64 bytes.

2. Switching capacity per slot that supports total port capacity on line card.

3. Filtering based on ACLs.

4. QoS handling based on 802.1p or IP Type of Service (TOS) bits.

5. Priority scheduling based on weighted round robin (WRR).

6. Traffic shaping.

7. Routing protocols, including multicast.

The test methodology is broken out into module- and system-level testing. In the real world, local switching on the module occurs; this is the best-case scenario for switch performance, because there is no contention for the fabric. The worst-case scenario is when all traffic entering the switch must traverse the fabric,

contending for backplane capacity and causing over-subscription.

Module-level testingIn this scenario, traffic will stay local to the line card. This means that switching will occur locally between ports on the same line card, and minimal traffic will traverse the backplane.

System-level testingIn this scenario, all of the ingress traffic is switched to ports on different line cards. This means that traffic will be contending for backplane capacity and will determine how the system handles oversubscription. It will be set up with partially meshed traffic patterns, with unique ingress ports mapped to unique egress ports, and multiple ingress ports mapped to a single egress port.

Test methodologies

1. Layer 2 bidirectional throughput and latency test. This test determines the Device Under Test’s (DUT’s) maximum Layer 2 forwarding rate without traffic loss as well as average latency for different packet sizes. This test is performed full duplex with traffic transmitting in both directions. The DUT must perform packet parsing and Layer 2 address look-ups on the ingress port and then modify the header before forwarding the packet on the egress port.

2. Layer 2 throughput, QoS, and latency test.

This test determines the DUT’s maximum Layer 2 forwarding rate with packet loss and latency for different packet sizes. The DUT must perform a Layer 2 address look-up, check the 802.1p priority bit value on the ingress port, send it to the designated queue, and then modify the header before forwarding the packet on the egress port.

3. Layer 3 (IPv4) performance test with ACL and latency. This test determines the DUT’s maximum IPv4 Layer 3 forwarding rate with packet loss and latency for different packet sizes. The DUT must perform


packet parsing and route look-ups for both Layer 2 and Layer 3 packets on the ingress port and then modify the header before forwarding the packet on the egress port. The ACL test involves blocking or allowing traffic through, based on user-defined classifiers such as IP addresses or Layer 4 port numbers. Ixia routing emulation software is used to populate the DUT’s routing table,. For example, OSPF emulation is used to generate OSPF LSAs to construct topological databases.

4. Layer 3 (IPv4) performance test with ACL, QoS, and latency. In addition to test 3 above, QoS values in each header will force the classification of the traffic based on IP Type of Service (TOS) field settings. On the ingress side, this QoS policy could also be used for assigning a packet to a specific queue, packet metering, and policing; on the egress side, it could be used for packet shaping.

5. Layer 3 (IPv6) performance test with ACL and latency. This test methodology is the same as the previous Layer 3 IPv4 with ACL performance test, except that it runs IPv6 traffic with a minimum-size packet of 84

bytes instead of 64 bytes. Due to the larger IPv6 header, the classification and table look-up functions will require more bandwidth and processing.

6. Layer 3 (IPv6) performance test with ACL, QoS, and latency. In addition to test 5 above, QoS values in each header force the classification of the traffic, based on the TOS field setting. On the ingress side, this QoS policy could also be used for assigning a packet to a specific queue, packet metering, and policing; on the egress side, it could be used for packet shaping.

7. Multicast test. This test uses a multicast protocol such as IGMP (IPv4) or MLD (IPv6) to set up multicast sessions between a multicast transmitter and groups of receivers. A multicast protocol emulation can be used to simulate one or more hosts while the DUTs function as IGMP/MLD routers. The simulation calls for groups of simulated hosts to respond to IGMP/MLD router-generated queries and to generate reports automatically at regular intervals. A number of IGMP groups are randomly shared across a group of hosts.


Anticipating the behavior of the 10GE switchEach row in the following table represents a test methodology and its expected impact on the different stress points mentioned earlier in this document, based on the switch’s design criteria.

As an example, in Table 2, row 2 (module-level, Full duplex Layer 2 performance and latency, with prioritization): all stress points show low or no stress, except for Stress Point 3, which points to the line

card function that services the different priority queues.

However, in row 6 (Layer 3 IPv6 performance, ACL and QoS), Stress Points 1 and 2 show that a high level of strain is to be expected. This is because we know that the switch is designed to handle wire-speed Layer 3 IPv4 packets, but not IPv6. This may mean that packet classification for IPv6 addresses may take more clock cycles, which may require more buffering on the ingress.

Table 2. Module-level test methodologies and stress points.

Stress Points

Test MethodologyIngress

Packet BufferPacket

ClassificationTraffic

Management

X-Bar Switch& BackplaneInterconnect

MulticastReplication& Queues

ControlPlane

1 2 3 4 5 6

1. Full duplex Layer 2performance & latency test

2. Layer 2 QoSthroughput & latency test

3. Layer 3 (IPv4) with ACLperformance & latency test

4. Layer 3 (IPv4)with QoS & ACLperformance & latency test

5. Layer 3 (IPv6) with ACLperformance & latency test

6. Layer 3 (IPv6)with QoS & ACLperformance & latency test

7. Full duplex IGMPmulticast test h i g h s t r e s s

h i g h s t r e s s

l o w / n o s t r e s s



m o d e r a t e s t r e s s

m o d e r a t e

s t r e s s



With system-level testing, additional strain will occur with the traffic management and switching functions (Stress Points 3 and 4) because of a high level of traffic contending for backplane switching capacity. For example, row 4 (Mesh Layer 3 IPv6, with route flapping), shows additional

strain not only in Stress Points 3 and 4, but also in Stress Point 6 (the control processor). Because of route flapping, it needs to modify the routing table as packets continue to arrive on each interface.

Table 3. System-level test methodologies and stress points.

Stress Points

Test MethodologyIngress

Packet Buffer

Mesh L3 IPv4 withACL performance & latency

Mesh L3 IPv4 withQoS and ACLperformance & latency

Mesh L3 IPv6 withACL performance & latency

Mesh L3 IPv6 withQoS & ACL performance& route flapping

Mesh IGMP multicast

PacketClassification

TrafficManagement

X-Bar Switch& BackplaneInterconnect

MulticastReplication& Queues

ControlPlane

1 2 3 4 5 6



h i g h s t r e s s

h i g h s t r e s s

l o w / n o

s t r e s s

l o w / n os t r e s s

l o w / n os t r e s s




Ixia’s Test Solution Ixia provides testing solutions that measure the performance and conformance of complex IP networks, devices, and applications. It was the first to offer support for a comprehensive 10GE wire rate test solution — one that can generate, capture, characterize, and emulate network and application traffic, establishing definitive performance and conformance metrics for a 10GE network device or system under test. It is also the first to offer support for XAUI, XENPAK, XFP, XPAK, and X2 interfaces in the test industry. For more information on Ixia’s 10GE products, refer to: http://www.ixiacom.com/products/interfaces/index.php

Ixia has developed two primary applications for Ethernet performance testing, IxExplorer and IxScriptMate, each with a distinct testing focus.

IxExplorer IxExplorer provides a high level of flexibility and functionality in protocol emulation, traffic generation, and analysis. IxExplorer is the primary controlling application for Ixia’s purpose-built hardware test platform, allowing detailed configuration of protocols and analysis of test results.

Within IxExplorer, a comprehensive set of IP signaling and routing protocols is

supported, including OSPF, IS-IS, RIP, BGP, LDP, and RSVP-TE, as well as multicast protocols. IxExplorer controls the protocol’s operation on Ixia’s test hardware architecture, which supports a CPU running Linux on each test port. This dedicated emulation environment allows hundreds of 10GE routers to be emulated on each network interface. Up to hundreds of thousands of LSPs can be signaled from each interface. Line rate traffic can be generated over the established connections.

IxScriptMate IxScriptMate provides a framework for running automated test scenarios. Numerous test suites have been developed within the IxScriptMate environment for testing the session and circuit scalability, traffic throughput performance, latency, and failover recovery capabilities of switch/routers. IxScriptMate simplifies the configuration process by defining a configuration for the test and displaying the relevant parameters for user input. Tests then run automatically, and the results are presented to the user. This is especially useful where the test configuration can involve multiple protocols and complex traffic generation requirements.


Table 4 maps the specific Ixia solution available to test each of the seven test methodologies previously outlined for 10GE switches.

Table 4. Test methodologies and Ixia test solutions.

Test Methodology Ixia's Solution

1. Full duplex Layer 2 performance & Latency

IxExplorer,IxScriptMate using RFC 2544 and RFC 2889 tests

2. Layer 2 QoS throughput and latency test

IxExplorer,IxScriptMate QoS Test

3. Layer 3 (IPv4) with ACL performance and latency test

IxExplorer,IxScriptMate RFC 2544, RFC 2889, ATSS tests

4. Layer 3 (IPv4) with QoS and ACL performance and latency test

IxExplorer, IxScriptMate QoS Test, BGP, OSPF tests

5. Layer 3 (IPv6) with ACL performance and latency test


6. Layer 3 (IPv6) with QoS and ACL performance and latency test


7. Full duplex IGMP multicast IxExplorer,IxScriptMate IP Multicast test


Appendix: 10GETesting Examples

The test plans presented in this section include several examples demonstrating how Ixia's solution addresses the challenges of 10GE switch testing.

1. Throughput Testing: RFC 2544 Throughput Test

Objective. To characterize the 10GE switch data plane performance in forwarding traffic. IETF RFC 2544 defines a test methodology for this characterization, divided into the following three tests:

• The back-to-back test determines how the DUT responds to different quantities of frames with the minimum

gap allowed by the protocol specification.

• The frame loss test determines how the DUT responds to streams with different loading.

• The throughput test finds the highest rate at which the DUT can forward frames.

Setup. A minimum of two 10GE test ports is required for this test. Ixia's IxScriptmate application can be used to run RFC 2544 tests.

Parameters. Select the test port pairs, frame size mode and frame sizes, and number of iterations.

Figure 4. RFC 2544: Throughput Test — setup.


Methodology. This test should involve several test ports to match the DUT's port density. Ideally, the test should flood traffic to every input port of the DUT. A number of Ixia load modules will be connected to the DUT. Ixia's IxScriptMate is used to perform the RFC 2544 benchmark test.

Results. The results of the test show the throughput rates, in frames per second, obtained for each frame size. The results also show the average throughput rates for all the trials.

Figure 5. RFC 2544: Throughput Test — results.


2. Testing for Latency: RFC 2544 Latency Tests

Objective. To determine the latency of the DUT, and how much it varies with different frame sizes.

In this test, frames are transmitted for a fixed duration. Once per second, the test tags a frame and transmits it halfway through the duration time. The test compares the tagged frame's timestamp when it was transmitted with the timestamp when it was received. The difference between the two timestamps is the latency.

Setup. A minimum of two 10GE test ports will be used for this test, in conjunction with the IxScriptmate RFC2544 test. Latency will be measured in both directions.

Parameters. Select the test port pairs, frame size mode and frame sizes, and number of iterations.

Methodology

1. Packets will be sent to the DUT from each 10GE load module starting with the selected minimum frame size for an interval that lasts for selected test duration.

2. The average latency is then measured.

3. These two steps will continuously repeat, using selected frame sizes every interval until it reaches the maximum frame size selected.

Results. The results of the test show the latency in nanoseconds (ns) obtained for each frame size and test port pair. The results also show the average latencies across the pairs.

Figure 6. RFC 2544: Latency Test — results.


3. Testing how 10GE ports handle prioritized streams based on QoS parameters

Objective. To determine the maximum rate at which the DUT can forward frames correctly according to their priority settings.

The test sets up a configuration in which many ports, each with a different priority, sends traffic to one single port. The receive port may be overloaded in order to test the receipt of the highest priority frames.

This test supports both MAC and IP layer frames; priority bits may be specified either in the IP precedence bits or in the

802.1p header. Latency may optionally be calculated per priority level. Test results are: the transmit and receive rates per priority, percent loss, and optionally, latency, for each priority per port.

Setup. A minimum of two test ports will be used for this test, in conjunction with the IxScriptmate QoS Many-to-One test. In this example, one Ixia 10GE test port and two Ixia 1 GE test ports connected to the DUT – all traffic directed to a single 10GE port on the DUT.

Parameters. Select the three test port pairs, frame sizes, QoS parameters for each port, and number of iterations.

Figure 7. Testing how 10GE ports handle prioritized streams based on QoS parameters — setup.

10 GE

1 GE

1 GE

10 GE


Methodology

1. Packets will be sent from all ports. Multiple streams with varying QoS parameters will be sent to the one egress port.

2. Total frames received, latency, and frame forwarding rate are measured per priority.

3. Steps 1 and 2 continuously repeat, using selected frame sizes every interval until the test reaches the maximum frame size selected.

Results. The results of the test show, on a per priority stream, the total frames received, the receive rate, and percentage loss for each frame size.

Figure 8. Testing how 10GE ports handle prioritized streams based on QoS parameters — results.


4. 10GE OSPF Convergence Test

Objective. Verify the ability of a router to switch between preferred and less-preferred routes on its 10GE ports when the preferred routes are withdrawn and re-advertised. This will test the control plane function on the DUT’s 10GE line card.

In this test, two 10GE ports simulate OSPF routers, Router 1 and Router 2. Both routers advertise the same route prefixes to the simulated network. However, the routes advertised by Router 1 will have smaller metrics (lower costs), which should cause the DUT to forward traffic over them instead of Router 2’s routes. After advertising the routes, the transmit port begins transmitting a stream of packets to an address in each of the advertised route prefixes. The DUT should forward all the packets over the route with the lower metric (Router 1). After a time interval has

elapsed, the receive port simulating Router 1 withdraws its routes. The DUT should detect that the preferred route has gone down and switch traffic to Router 2’s routes. Router 1 then re-advertises its routes. The DUT should again detect a change re-route traffic to the receive port simulating Router 1.

Setup. This test uses three test ports — one to transmit and two to receive (see Figure 9 below). One Ixia test port (acting as the transmit port) and two Ixia 10GE (test ports 1 and 2 acting as receive ports) connect to the DUT. Receive ports will emulate OSPF routers. The traffic is unidirectional. The DUT must have three ports utilized with two enabled for OSPF. All three ports should be configured for IP and have unique subnets in which to communicate with the tester ports.

Figure 9. 10GE OSPF Convergence Test — setup.


Results. The test results provide an average convergence time for all routes. Figure 10 below displays example results for the automated OSPF convergence test in

IxScriptMate. In addition to convergence time, this test also indicates the amount of lost packets caused by the convergence.

Figure 10. 10GE OSPF Convergence Test — results.


GlossaryAccess Control List (ACL) A list of the services available on a server, each with a

list of the hosts permitted to use the service.

Context Memory Context memory contains instructions about whether to deny or forward a packet, where to forward and all internal system headers needed to get a packet from an ingress to egress port and the external packet header (new MAC, IP, MPLS stacks, etc.).

Head-of-Line (HOL) Blocking HOL blocking occurs when the packet at the head of a queue cannot be transmitted to an output due to a contending packet from another input. At the same time, a packet further back in the queue is blocked although its destination port is free to receive the packet.

Packet Buffer A temporary repository for arriving packets while they wait to be processed.

Packet Processor Optimized application-specific integrated circuit (ASIC) or programmable device (NPU) for processing and forwarding packets in the data plane or fast path. It performs specific key tasks such as parsing the header, pattern matching or classification, table look-ups, packet modification, and packet forwarding, ideally at wire speed.

Random Early Detection (RED) Random Early Detection (RED) is a mechanism for avoiding congestion in packet switched networks. RED controls the average queue size by indicating to the end hosts when they should temporarily stop sending packets.

SERDES Serializer/Deserializer.

Switch Fabric Provides data plane interconnection between each line card in the system. The switch fabric typically employs a crossbar to move packets between its ingress and egress ports.

Synchronized Digital Hierarchy(SDH)

An international digital telecommunications network hierarchy that standardizes transmission around the bit rate of 51.84 megabits per second, which is also called STS-1. The SDH specifies how payload data is framed and transported synchronously across optical fiber transmission links, without requiring all the links and nodes to have the same synchronized clock for data transmission and recovery.

Synchronized Optical Network(SONET)

SONET is a standard for optical transport. It allows different types of formats to be transmitted on one line.


Acknowledgements Authors: Ted Fornoles, Alireza Safari

Editor, Illustrator: Elliott Stewart

Ternary Content AddressableMemory (TCAM)

Content addressable memory refers to a kind of storage device that includes comparison logic with each bit of storage.

Time Division Multiplexing (TDM) A method of putting multiple data streams in a single signal by separating the signal into many segments, each having a very short duration. Each individual data stream is reassembled at the receiving end.

Traffic Management Function The traffic management function is used to regulate the flow of traffic. It will forward traffic according to a user-defined set of rules pertaining to priority levels, latency and bandwidth guarantees, and varying congestion levels. It also provides the necessary buffering required to work in conjunction with any queuing mechanisms it uses to manage traffic flow across the switch fabric.

Wave Division Multiplexing(WDM)

A type of multiplexing developed for use on optical fiber. WDM modulates each of several data streams onto a different part of the light spectrum.

Weighted Fair Queuing (WFQ) A scheduling algorithm that identifies conversations (in the form of traffic streams), separates packets that belong to each conversation, and ensures that capacity is shared fairly between these individual conversations. WFQ is an automatic way of stabilizing network behavior during congestion and results in increased performance and reduced retransmission.

Weighted Random EarlyDetection (WRED)

Cisco’s implementation of RED, called Weighted Random Early Detection (WRED), combines the capabilities of the RED algorithm with IP Precedence to provide preferential traffic handling for higher priority packets.

Weighted Round Robin (WRR) A scheduling algorithm that assigns a weight to each client (it can be a labeled stream or a port within a switch/hub, or a group of streams belonging to the same service level); the assigned weight corresponds to the pre-assigned bandwidth allocation for the client.


Documents

10-Gigabit Ethernet Switch Performance Testing · Ethernet (IEEE 802.3u) in 1995, 1-Gigabit Ethernet (802.3z) in 1998 to 10-Gigabit Ethernet (802.3ae) in 2002. The 10-Gigabit Ethernet