WI-FI PERFORMANCE BENCHMARK TESTING: Aruba Networks … · Wi-Fi Performance Benchmark Testing: Aruba Networks AP225 and Cisco Aironet 3702i Aruba Networks, Inc. 3 Executive Summary

WI-FI PERFORMANCE BENCHMARK TESTING: Aruba Networks AP-225 and Cisco Aironet 3702i Conducted at the Aruba Proof-of-Concept Lab January 24, 2014

Wi-Fi Performance Benchmark Testing: Aruba Networks AP-225 and Cisco Aironet 3702i

Aruba Networks, Inc. 2

Statement of Test Result Confidence • Aruba makes every attempt to optimize all vendors for performance and follow best

practices for configuration as published by the vendor.

• Aruba ensures that tests are performed in a fair manner including AP mounting position, client location, transmit power, channel and the latest firmware available from each vendors’ support website at the time of testing.

• Aruba ensures the test bed environment to be free of any interference sources. Also, the 802.11 Wi-Fi channels configured are ensured to be consistent when testing 2.4 GHz and 5 GHz bands for all vendors.

• Aruba believes the test results are both repeatable and reproducible in similar testing environments.

• Configurations used in the testing for both vendors are posted here: www.arubanetworks.com/pdf/technology/TR_Aruba_WiFi_Performance_APPENDIX.pdf

Table of Contents Executive Summary ................................................................................................................. 3 Test Environment ..................................................................................................................... 4 Test Scenarios .......................................................................................................................... 6 What Do The Tests Reveal? .................................................................................................. 14 About Aruba Networks, Inc. .................................................................................................. 15



Executive Summary The key enhancement that 802.11ac delivers in comparison to the 802.11n standard is the support of higher data rates. This is achieved mainly by doubling the maximum channel width to 80MHz as well as the addition of a more efficient modulation technique, 256-QAM, which is 33% more efficient. With a 3 spatial stream 802.11ac device it is now possible to achieve wireless data rates up to 1.3 Gbps on the 5 GHz radio. However, the data rates are only part of the picture. Although it is possible to achieve these maximum data rates with many of the 802.11ac access points currently in the market this does not mean they provide equal throughput. For the end user the high data rate has very little value if the actual throughput seen is still low. Real world traffic is composed of a mix of packets of various sizes so we have performed tests of several packet sizes here. Small packets are very demanding of the AP and require a powerful CPU in order to deliver the maximum performance. The purpose built Aruba AP-225 that was tested has been designed to provide not only high data rates but also high throughput with real world traffic with a mix of small, medium and large size packets. As part of the benchmarking process, the tests were run with both an Aruba Networks AP-225 and a Cisco Aironet 3702i access point. The key highlights from the testing are as follows. Table 1: Test results summary

Test Scenario Aruba AP-225

Cisco 3702i

Aruba Advantage

Single Client downstream TCP throughput at 10ft 882 574 54%

Single Client upstream TCP throughput at 30ft 534 376 42%

Multi-client bidirectional TCP throughput with 30 clients using large packets (1500B) 412 288 43%

Multi-client downstream UDP throughput with 30 clients using small packets (256B) 157 43 3.6x

Multi-client downstream TCP throughput with 30 clients when co-channel interference is present 378 187 2x

These test results indicate a significant variation in performance between Aruba and Cisco products. Customers are advised to exercise their own judgment before making a vendor decision for their Wi-Fi network. The rest of the document provides comprehensive details of the test cases, test bed setup, observations and results collected.



Test Environment Table 2: Devices under test

Vendor Device Quantity Firmware version

Aruba AP-225 1 6.3.1.2 build 41701

7210 Mobility Controller 1 6.3.1.2 build 41701

Cisco Aironet 3702i 1 7.6.100.0

5508 WLAN Controller 1 7.6.100.0

The latest firmware versions available through the support sites of each vendor were used. The following table shows the detailed information on various network components that were part of the infrastructure used for performance tests. Table 3: Test equipment

Item Component Specifications Details

1

Device OS/Type Make and Model Quantity OS Version

Laptops MacBook Pro (3SS 802.11ac) 12 10.9.1

MacBook Air (2SS 802.11ac) 20 10.9.1

2 Performance evaluation tools

IxChariot, one wired endpoint, client on each device 7.10 SP5

3 Switch Cisco Catalyst 3750G 12.2 (50) SE5

4 Power Injector PowerDsine 9001G PoE+ injector N/A

5 Co-Channel Interference source

Aruba AP-225, 7210 Mobility controller 6.3.1.2 build 41701



Test Bed Setup Figure 1 below shows the logical network topology of the test bed. The wireless controllers, IxChariot wired endpoint and access points were all connected at Layer 2 using a Cisco Catalyst 3750G switch. A PoE+ (802.3at compliant) power injector was used to power the access points. The access points were mounted on the ceiling and installed in the same location for the duration of that vendor test. Static IP addresses were used on the clients.

Figure 1: Test bed setup

Legend



Test Scenarios The following scenarios were benchmarked in the proof of concept lab.

1. Single Client TCP throughput at 10ft and 30ft 2. Multi-client TCP throughput with 30 clients using large packets 3. Multi-client UDP throughput with 30 clients using small packets 4. Multi-client (30) TCP throughput when co-channel interference is present

During all tests, controllers were configured to use an 80MHz wide channel, and 802.11ac beamforming was enabled. Channel 100E was used in this test. The 2.4 GHz radio was also enabled but no clients were associated with the AP on 2.4 GHz band during this test.



Scenario 1: Single Client TCP throughput at 10ft and 30ft This test measures the maximum TCP throughput for a single client at two client locations; 10ft and 30ft away from the AP located in a typical office cube with line of sight (LOS). The client used in this test was a MacBook Pro supporting 802.11ac and three spatial streams capable of data rates up to 1300Mbps. TCP traffic was generated between the wireless and wired endpoint using Ixia IxChariot, High Throughput TCP script, 1MB file size, 20 pairs per client and run time of one minute. The test was repeated 3 times and the average of these three runs was recorded. The test results are shown in Table 4 below. Table 4: Single Client TCP Throughput at 10ft and 30ft

Test Case Aruba AP-225

Cisco 3702i

Aruba Advantage

Downstream TCP Throughput (Mbps) @ 10ft LOS 882 574 54%

Upstream TCP Throughput (Mbps) @ 10ft LOS 669 428 57%

Bidirectional TCP Throughput (Mbps) @ 10ft LOS 625 532 17%

Downstream TCP Throughput (Mbps) @ 30ft LOS 690 537 28%

Upstream TCP Throughput (Mbps) @ 30ft LOS 534 376 42%

Bidirectional TCP Throughput (Mbps) @ 30ft LOS 567 477 19%

882

669 625

690

534 567 574

428

532 537

376

477

0

100

200

300

400

500

600

700

800

900

1000

Down @ 10' LOS

Up @ 10' LOS

Bidirection @ 10' LOS

Down @ 30' LOS

Up @ 30' LOS

Bidirection @ 30' LOS

Sin

gle

Clie

nt T

hrou

ghpu

t (M

bps)

Arub

a AP

-225

Cis

co 3

702i



As can be seen from the results above, the Aruba AP-225 performance is superior to the Cisco 3702i in all test cases. Both APs achieved the maximum 1300Mbps data rates but the Aruba AP-225 provided up to 57% higher throughput compared to Cisco 3702i. The maximum TCP downstream throughput with the Aruba AP-225 was 882Mbps, which is approaching the limits of wired Gigabit Ethernet. Maximum downstream TCP throughput of the Cisco 3702i was a modest 574Mbps, 308Mbps lower than throughput with Aruba AP-225. While the Aruba AP-225 showed up to 42% higher throughput at 30ft compared to the Cisco 3702i in real world deployments this difference is most likely bigger. Aruba ClientMatch™ technology helps to keep the clients connected to the most optimum AP. Because of this in real world deployments you seldom see clients connecting to APs that are further than 30ft away from the AP. Cisco does not currently have ClientMatch equivalent functionality and “sticky clients” can sometimes be associated with AP quite far from the client degrading the performance seen by these clients and the whole network.



Scenario 2: Multi-client TCP throughput with 30 clients using large packets The second scenario evaluates the performance with multiple clients. A total of 30 clients were associated with the AP, up to 15 ft from the AP and had a clear line of sight to the AP. These 30 clients consisted of a mix of 10 MacBook Pro laptops (802.11ac, 3 spatial streams) and 20 MacBook Air laptops (802.11ac, 2 spatial streams). TCP traffic was generated between the wireless and wired endpoints using Ixia IxChariot, High Throughput TCP script, 1MB file size, 4 pairs per client and run time of one minute. The test was repeated 3 times and average of these three runs was recorded. The test results are shown in Table 5 below. Table 5: Multi-client TCP throughput with 30 clients using large packets

Test Case Aruba AP-225 Cisco AP3702i Aruba Advantage

Downstream TCP Throughput (Mbps) 409 321 27%

Upstream TCP Throughput (Mbps) 434 322 35%

Bidirectional TCP Throughput (Mbps) 412 288 43%

409

321

434

322

412

288

0

50

100

150

200

250

300

350

400

450

500

Aruba AP-225 Cisco 3702i

Mul

ti-c

lient

TC

P T

hrou

ghpu

t (M

bps)

Dow

nstre

am

Ups

tream

Bidi

rect

iona

l

Dow

nstre

am

Ups

tream

Bidi

rect

iona

l



Scenario 3: Multi-client UDP throughput with 30 clients using small packets This scenario evaluates the AP’s ability to handle small packets. A total of 30 clients were associated with the AP all within 15 ft of the AP and with clear line of sight. The 30 clients consisted of a mix of 10 MacBook Pro laptops (802.11ac, 3 spatial streams) and 20 MacBook Air laptops (802.11ac, 2 spatial streams). UDP traffic was generated between the wireless and wired endpoints using Ixia IxChariot, modified UDP throughput script with either 256B packets (4 pairs per client, 2Mbps per pair) or 512B packets (4 pairs per client, 4Mbps per pair) and run time of one minute. The test was repeated 3 times and average of these three runs was recorded. The test results are shown in Table 6 below. Table 6: Multi-client UDP throughput with 30 clients using small packets

Test Case Aruba AP-225

Cisco AP3702i

Aruba Advantage

Downstream UDP Throughput (Mbps) with 256B packets 157 43 3.6x

Downstream UDP Throughput (Mbps) with 512B packets 317 117 2.7x

As can be seen from the results, the Aruba AP-225 performance with small packets is significantly superior to the Cisco 3702i. What is especially notable there is the throughput with 512B packets. In today’s enterprise networks the average packet size of typical traffic is closer to 512B than 1500B. While the Aruba AP-225 UDP throughput with 512B packets was not far

317

157

117

43 0

50

100

150

200

250

300

350

512B Packets 256B Packets

Mul

ti-c

lient

UD

P T

hrou

ghpu

t (M

bps)

Arub

a AP

-225

Cis

co

3702

i

Arub

a AP

-225



from the TCP throughput with large (~1500B) packets, the throughput of the Cisco 3702i totally collapses. As these are test cases that cause quite a high load on the CPU we wanted to also see the CPU utilization of the APs during these tests. These are shown in the figures below.

Figure 2: CPU utilization of Cisco 3702i during UDP throughput test with 256B packets

Figure 3: CPU utilization of Aruba AP-225 during UDP throughput test with 256B packets

As can be seen from the figures above the CPU utilization of the Cisco 3702i is really high (>90%) and approaching 100% in both test cases. The high CPU utilization clearly shows that the CPU just cannot handle the load and that is leading to a significant performance degradation for the Cisco 3702i. The Aruba AP-225 can deliver high throughput in both test cases and the CPU peaks at a maximum utilization of 75% (>25% idle). This enabled the AP-225 to deliver high performance even with small packets while also leaving room for adding new features in the future without sacrificing performance.

Cisco CPU 90%+ utilization

Aruba CPU 75% utilization



Scenario 4: Multi-client (30) TCP throughput when co-channel interference is present This scenario focuses on testing the AP performance when co-channel interference is present. In real world environment this can happen e.g. when people bring in their own “mobile hotspots” like the MiFi, or enable personal hotspot feature on their smartphones. As in scenario 2, a total of 30 clients were associated with the AP all within 15ft from the AP and with clear line of sight. The 30 clients consisted of a mix of 10 MacBook Pro laptops (802.11ac, 3 spatial streams) and 20 MacBook Air laptops (802.11ac, 2 spatial streams). To generate co-channel interference (CCI) a separate Aruba AP-225 was used connected to an Aruba 7210 Mobility controller other than the device under test. The AP used as a CCI source was located ~15ft away from the AP under test, configured to use the same channel as the AP under test (100E). The signal strength observed at the location of the AP under test was around -60dbm. Two clients (802.11ac capable MacBook Pros) were then associated with the AP used as a CCI source and traffic was generated using IxChariot to cause ~20% channel utilization. TCP traffic was generated between the wireless and wired endpoints using Ixia IxChariot, High Throughput TCP Script, 1MB file size, 4 pairs per client and run time of one minute. The test was repeated 3 times and average of these three runs was recorded. The test results are shown in Table 7 below. Table 7: Multi-client TCP throughput with 30 clients when co-channel interference is

present

Test Case Aruba AP-225 Cisco AP3702i Aruba Advantage

Downstream TCP Throughput (Mbps) 378 187 102%

Upstream TCP Throughput (Mbps) 417 239 75%

Bidirectional TCP Throughput (Mbps) 374 226 65%

The impact of the co-channel interference on the throughput is shown in the Figure 4 below when comparing these results with the test run when interference was not present.



Aruba AP-225 Cisco 3702i Without Interference 409 321

With Interference 378 187

Figure 4: Impact of co-channel interference on throughput

Once again the Aruba AP-225 clearly outperforms the Cisco 3702i. The Cisco 3702i throughput drops 41.7% in the presence of co-channel interference whereas the Aruba AP-225 throughput dropped only 7.6%. Even in the presence of co-channel interference the Aruba AP-225 provides higher throughput than the Cisco 3702i in clean spectrum.

409

321

378

187

0

50

100

150

200

250

300

350

400

450

Aruba AP-225 Cisco 3702i

Dow

nstr

eam

Thr

ough

put

(Mbp

s)

41.7%

7.6%



What Do The Tests Reveal? The tests conclusively show that the Aruba AP-225 is superior in performance to the Cisco 3702i in all test scenarios. Many of these test cases cause considerable load on the CPU of the access point. This is also one of the areas were we see biggest differences in design of these two access points. Whereas the Aruba AP-225 has been designed from ground up to meet the demands of 802.11ac networks using the latest processor technology, Cisco has opted to use the same old CPU on the 3702i that they used two years ago on the 3602i. As a result the Aruba AP-225 has double the CPU capacity of the Cisco 3702i. This higher CPU capacity is required on 802.11ac access points to deliver what the 802.11ac standard promises. This is evident in the test results. The maximum TCP downstream throughput of Cisco 3702i was capped at modest 574Mbps. However, the Aruba AP-225 was able to deliver an impressive 882Mbps of TCP throughput, which is something one expects with physical layer data rate of 1300Mbps. The throughput test with small packets really highlights the differences in performance of these access points. With 256B packets, the Aruba AP-225 delivered 3.6 times the throughput of the Cisco 3702i. With 512B packets, the Aruba AP-225 throughput was 2.7 times the throughput of the Cisco 3702i. The average packet size in real world enterprise networks is still much closer to 512B than it is to 1500B, so this performance difference with 512B packets can be considered to reflect the performance difference of these two access points in real world deployments. Co-channel interference is quite often unavoidable, and commonly seen in the enterprise networks. The test results show that the Aruba AP-225 maintains excellent performance even in the presence of co-channel interference, and delivered throughput that exceeded the throughput that the Cisco 3702i had in clean spectrum environment. In the presence of co-channel interference the Cisco 3702i throughput dropped by 41.7% to only 187Mbps with 30 clients. This level of performance is achievable with older 3-stream 802.11n access points in a clean spectrum environment, which is not necessarily what you want to see after investing in new 802.11ac access points.



About Aruba Networks, Inc. Aruba Networks is a leading provider of next-generation network access solutions for the mobile enterprise. The company’s Mobile Virtual Enterprise (MOVE) architecture unifies wired and wireless network infrastructures into one seamless access solution for corporate headquarters, mobile business professionals, remote workers and guests. This unified approach to access networks enables IT organizations and users to securely address the Bring Your Own Device (BYOD) phenomenon, dramatically improving productivity and lowering capital and operational costs. Listed on the NASDAQ and Russell 2000® Index, Aruba is based in Sunnyvale, California, and has operations throughout the Americas, Europe, Middle East, Africa and Asia Pacific regions. To learn more, visit Aruba at http://www.arubanetworks.com. For real-time news updates follow Aruba on Twitter and Facebook.

© 2014 Aruba Networks, Inc. Aruba Networks’ trademarks include AirWave®, Aruba Networks®, Aruba Wireless Networks®, the registered Aruba the Mobile Edge Company logo, Aruba Mobility Management System®, Mobile Edge Architecture®, People Move. Networks Must Follow®, RFProtect®, and Green Island®. All rights reserved. All other trademarks are the property of their respective owners.

Documents

WI-FI PERFORMANCE BENCHMARK TESTING: Aruba Networks … · Wi-Fi Performance Benchmark Testing: Aruba Networks AP225 and Cisco Aironet 3702i Aruba Networks, Inc. 3 Executive Summary