2015 Miercom Unified Threat Management …The Spirent Avalanche tool provides multi-10 Gbps capacity, security, and performance testing for network infrastructures. WatchGuard Firebox

2015 Miercom

Unified Threat Management

Throughput Performance

DR150514P

June 2015

Miercom

www.miercom.com

http://www.miercom.com/

WatchGuard Firebox M300 UTM 2 DR150514P

Copyright © 2015 Miercom 9 June 2015

Contents

1.0 Executive Summary ............................................................................................................................... 3

2.0 How We Did It ......................................................................................................................................... 4

3.0 Throughput Test Results ................................................................................................................... 10

4.0 Fair Test Notification ........................................................................................................................... 18

5.0 About Miercom ..................................................................................................................................... 18

6.0 Use of This Report ............................................................................................................................... 18



1.0 Executive Summary

Miercom was engaged by WatchGuard to validate the performance of its Firebox M300

Unified Threat Management (UTM) appliance, as well as the comparative performance

of Vendor A’s products, the Fortinet FortiGate 100D, Vendor B’s product, and the

Sophos SG 210.

A series of six tests were applied to verify throughput performance of the firewall

operating alone and with other policies enabled.

This report summarizes the throughput of the appliances. Each appliance was

configured with vendor defaults enabled to ensure a baseline configuration. Features

were then enabled incrementally and the resulting throughput rates were recorded.

Key Findings:

The Firebox M300 had excellent throughput performance for any enabled

feature, whether it was tested alone or in combination

WatchGuard’s provides firewall and antivirus protection over encrypted traffic

while outperforming the competitive average throughput performance by 385%

With full UTM enabled, WatchGuard had over a 45% lead in throughput than its

best competitor, against HTTP traffic and performed 218% better than the

industry average

The WatchGuard Firebox M300 UTM appliance out-classed the competitors in almost

every test. The M300 is one of the best mid-sized and distributed enterprise class UTM

appliances on the market, given competing products in the same price range

and scale.

Based on the impressive results of the testing, we proudly

award the Miercom Performance Verified Certification to

WatchGuard, having delivered outstanding performance

by the WatchGuard Firebox M300 UTM.

Robert Smithers

CEO

Miercom



2.0 How We Did It

About the Test

Unified Threat Management (UTM) appliances are intended to operate as a combination

of firewall, intrusion prevention, and antivirus functionalities. Testing evaluated the

throughput performance of various threat management functionalities, in isolation and

in combination, for each vendor.

These functionalities protect a network by evaluating traffic through abstract layers of

the Open Systems Interconnection (OSI) model. In order to simulate a real-world

environment, tests were performed across multiple layers to portray how real traffic

would travel and be handled.

Two layers of the standardized OSI model were of focus for this product evaluation: the

transport layer and the application layer. A majority of the pertinent network traffic

processed by the appliances under test operates on these layers, making them

significant layers to verify throughput performance.

The following six tests were used to validate performance:

Test Function Parameter

1. Firewall

2. Firewall (FW)

These tests establish line speed on each appliance and

ensure that results are within a reasonable range of

those claimed by vendors.

UDP, UDP IMIX, HTTP,

HTTPS

3. FW+ IPS

4. FW + AV

5. FW+ IPS + AV

6. Full UTM: FW +

IPS + AV + AppCtrl

These tests establish proper functioning and expected

traffic analysis of each appliance.

These tests are the primary focus since they describe the

throughput of appliances with real-world deployment.

HTTP, HTTPS w/ deep

packet inspection

The first two tests were performed for two purposes: to ensure appliances were properly

configured and processing traffic, and to compare appliances based on throughput

speed with only firewall enabled.

Four different types of traffic were used in the throughput tests: user datagram protocol

(UDP), UDP IMIX, hypertext transfer protocol (HTTP), and HTTP Secure (HTTPS). UDP

operates on the transport layer and is intended for fast data transmission. It simply

includes the source and destination ports, the length (bytes), and the “checksum”

parameter to ensure data is intact when it does arrive. UDP Internet Mix, or IMIX, is used

to resemble real-world internet traffic conditions. Standardized IMIX profiles are based

on statistical sampling of traffic passed through routers and is used for testing to certify



comparable results for each vendor. HTTP operates on the application layer and serves

as request-response communication protocol between clients and servers. HTTPS is

when the data being communicated is encrypted and travels on the secure socket layer

(SSL). HTTPS implies a more complex process where certificates must be authenticated,

and session keys conveyed to continue encrypted transmission. Throughput is expected

to drop considerably during SSL traffic, for firewall and subsequent tests.

Both user datagram and hypertext transfer protocols allow the comparison of respective

throughput speeds of each UTM appliance while under various real-world network

traffic scenarios.

The second and following throughput tests were conducted using both HTTP and HTTPS

traffic. HTTP is unencrypted hypertext transfer protocol traffic. HTTPS traffic is encrypted

with secure socket layer (SSL) enabled and provides more realistic network conditions to

test deep packet inspection (DPI) functionality of each device, which is implemented

when IPS and antivirus functionality is enabled.

The third test, FW + IPS, validated HTTP and HTTPS throughput speeds for each

appliance with both the firewall and the intrusion prevention system (IPS) functions

enabled. IPS functionality can be deployed on both HTTP traffic and encrypted HTTPS

traffic, so the test was run for each traffic type. IPS tends to be rule or signature-based,

and monitors network traffic and/or system activity to actively detect, prevent and block

intrusions.

The fourth test, FW + AV, compared throughput speeds for each appliance with firewall

and antivirus functionality enabled. This allows an accurate record of throughput speeds

while the UTM appliance is performing virus scanning activities on HTTP and HTTPS

traffic.

Since Unified Threat Management tends to involve at a bare minimum: FW + IPS + AV

functionality, the fifth test compares throughput speed in a real-life UTM deployment

scenario. This test generated results as if each UTM appliance was acting as a next-

generation firewall with IPS and AV systems both inspecting and acting upon

unencrypted (HTTP) and encrypted (HTTPS) traffic.

The sixth and final test, full UTM: FW + IPS + AV + AppCtrl, compared throughput rates

for each appliance with all functionality enabled. It measured throughput in a

deployment scenario where the device under test is operating as a next-generation

firewall with IPS, AV, and application control functionality enabled. This would describe a

situation involved signature-based monitoring of network connections, DPI functionality

for HTTPS (SSL) decryption and analysis, as well as a fine-tuned application layer security

control scheme.



Products Tested

WatchGuard Firebox M300

This appliance aims to provide security for mid-sized enterprises using its unified

approach which includes the following features: firewall, application control, antivirus,

and IPS. These features are also offered to operate with HTTPS inspection enabled. In

addition to these core features, it is also capable of additional subscriptions which

enable advanced persistent threat blocker, data loss prevention, reputation enabled

defense, spam blocker, and URL filtering.

Vendor A Product One

This appliance intends to deliver a wide scope of security for medium enterprises,

without compromising its performance. This product is claimed to have industry leading

SSL-decryption rates to block malware, an authentication server to enforce policies for

application control, and its NSA Series integration of firewall and intrusion prevention,

all manageable by a single console.

Vendor A Product Two

This appliance supports security for distributed enterprises and remote offices, all

managed by a central office. Its deep packet inspection examines traffic across all ports,

with integrated intrusion prevention engine, anti-virus, anti-spyware, and application

controls, and web filtering over both unsecure and encrypted SSL connections.

Fortinet FortiGate 100D

This appliance is used to protect medium enterprises or remote branches with one

management console, capable of multiple technologies for detecting network attacks.

Features include: firewall, IPsec and SSL-VPN, application control, intrusion prevention,

anti-malware, anti-spam, P2P security, and web filtering. These features can be enabled

alone or combined for a full unified threat management system.

Vendor B

This appliance offers security to mid-sized enterprises by providing application visibility,

antivirus, IPS, data filtering, modern malware protection, URL filtering, and mobile

security. It is capable of policy-based decryption and inspection to ensure that SSL and

SSH encrypted traffic is pertinent to business purposes only.

Sophos SG 210

This appliance offers a consolidated security solution which protects a medium sized

business or organization for both standard and secured internet traffic. This unified

threat management consists of firewall, intrusion prevention, and antivirus with high

rates of throughput claimed for each.



Test Bed Setup

Figure 1: Test Setup Diagram

All hardware appliances were tested in an identical environment.

Bidirectional test traffic from each load generator was routed to each appliance using

three port pairs and back to the generator for Layer 4 testing. Packet size consisted of

1518 bytes.

Unidirectional test traffic from each load generator was routed to each appliance using

three port pairs for Layer 7 testing. Protocols used were port 80 HTTP and port

443 HTTPS.

Source: Miercom May 2015




Sophos SG 210

Vendor B

Management

Workstation




List of Equipment

Name Function Version

WatchGuard Firebox M300 UTM Appliance V5.518 Build

11.9.6.B475461

Vendor A Product One Next Generation Firewall XXX

Vendor A Product Two Next Generation Firewall XXX

Fortinet FortiGate 100D UTM Appliance V5.2.3 Build 670

Sophos SG 210 UTM Appliance 9.310-11

Vendor B UTM Appliance XXX

Spirent TestCenter Traffic Generator 4.43

Spirent Avalanche Traffic Generator 4.43

The Spirent TestCenter tool delivers comparative analysis of devices or services with

deterministic traffic during product development cycles or vendor comparisons.

The Spirent Avalanche tool provides multi-10 Gbps capacity, security, and performance

testing for network infrastructures.



Configurations

All appliances were capable of firewall, antivirus, application control and IPS. They were

tested in-line with default features and up-to-date firmware. All of the devices were

tested with three port pairs enabled.


This product was configured for this testing primarily as a high-capacity, next generation

firewall.


This product was a Next Generation Firewall appliance.


This product was a Next Generation Firewall appliance.


This product was configured as a NGFW appliance with default features enabled.

However, it did not support TLC v1.2, so RC4-MD5 and RC4-SHA were used instead.

Vendor B

This product was configured with default features enabled. However, it had a packet

fragmentation issue that required a reduction in packet size from the standard 1518

bytes to 1500 bytes.

Sophos SG 210

This product was configured as a NGFW appliance with default features enabled.



3.0 Throughput Test Results

Summary of Results

Throughput results by vendor and traffic:

Throughput (Mbps)




Fortinet FortiGate

100D Vendor B

Sophos SG 210

UDP Firewall

UDP 1518 byte 4000 1398 1170

1440 414 6000

UDP IMIX 2160 370 660 532 414 2212

HTTP/S Firewall (FW)

HTTP 2663 940 1000 1147 975 3000

HTTPS 1500 982 1000 1309 975 2200

FW + IPS

HTTP 2513 903 1000 1130 217 233

HTTPS 1231 795 1000 1216 975 2246

FW + AV

HTTP 1149 152 742 538 976 675

HTTPS w/ DPI 987 103 144 256 152 361

FW + IPS + AV

HTTP 847 178 343 438 215 185

HTTPS w/ DPI 417 103 139 218 142 443

Full UTM: FW + IPS + AV + AppCtrl

HTTP 778 150 252 423 215 183

HTTPS w/ DPI 393 103 123 207 142 374



WatchGuard shows a consistent decrease in throughput as more featured are applied, in

comparison to other vendors which change much more drastically even upon the first

additional feature enabled.

WatchGuard had a much more steady decrease as more features were enabled,

maintaining its baseline throughput better than any vendor for full UTM enabled.

0 1000 2000 3000 4000

SophosSG 210

Vendor B

FortinetFortiGate 100D

Vendor A ProductTwo

Vendor A ProductOne

WatchGuardFirebox M300 UTM

Throughput Performance over HTTP Traffic

Baseline Firewall

FW + IPS

FW + AV

FW + IPS + AV

FW + IPS + AV + AppCtrl


0 500 1000 1500 2000 2500

SophosSG 210

Vendor B

FortinetFortiGate 100D

Vendor A ProductTwo

Vendor A ProductOne

WatchGuardFirebox M300 UTM

Throughput Performance over HTTPS Traffic

Baseline Firewall

FW + IPS

FW + AV

FW + IPS + AV

FW + IPS + AV + AppCtrl




The following charts graphically represent the comparative throughput performance for

each appliance, by traffic type and features enabled.

Firewall Performance on Transport Layer

Firewall throughput with bidirectional UDP and UDP IMIX test traffic.

The purpose of this test was to ensure each appliance was configured correctly and

processing traffic before applying the baseline and subsequent tests.

Figure 2: Firewall Performance with UDP traffic

WatchGuard outpaced all other vendors for both types of UDP traffic except for Sophos.

Vendor B’s device was only configurable to handle UDP 1500 byte traffic due to high

packet loss, implying a possible issue regarding fragmentation.

0

1000

2000

3000

4000

5000

6000

7000

WatchGuardFireboxM300

Vendor AProduct

One

Vendor AProduct

Two

FortinetFortiGuard

100D

Vendor B SophosSG 210

Thro

ugh

pu

t (M

bp

s)

Firewall with UDP traffic

UDP 1518 byte

UDP iMix




Firewall Performance on Application Layer

Firewall throughput with unidirectional HTTP and HTTPS traffic generated by the

Spirent TestCenter and Spirent Avalanche test system. HTTP and HTTPS traffic

provided a more realistic test environment for the UTM appliances.

This test served as a baseline test. Subsequent tests increase levels of protection and

are expected to have a decreasing performance in throughput.

Figure 3: Firewall Performance with Encrypted and Unencrypted HTTP traffic

Sophos displayed the highest baseline throughput for both unencrypted and encrypted

traffic. WatchGuard displayed the second highest throughput for both. These throughput

results will be the values for comparison against those for other tests where features are

enabled to determine how much supplementing features deter performance.

0

500

1000

1500

2000

2500

3000

3500


Vendor AProduct

One

Vendor Aproduct

Two

FortinetFortiGuard

100D

Vendor B Sophos SG210

Thro

ugh

pu

t (M

bp

s)

Firewall with HTTP and HTTPS traffic

HTTP

HTTPS




Intrusion Prevention System Performance on Application Layer

Intrusion Prevention System (IPS) throughput with unidirectional HTTP and HTTPS

traffic generated by the Spirent TestCenter and Spirent Avalanche test system.

The purpose of this test was to verify any degradation to throughput when an

additional feature was enabled.

Figure 4: Firewall and IPS Performance with Encrypted and Unencrypted

HTTP Traffic

WatchGuard predictably had lower throughput for both secure and unsecured HTTP traffic

when its IPS feature was additionally enabled. However, it maintained the highest

throughput of all vendors for unsecured traffic. Vendor A Product One and Fortinet showed

between 1-19% decrease in performance for either HTTP or HTTPS. Vendor A Product Two

showed no change in throughput from its baseline. Vendor B’s throughput dropped by 78%

for HTTP but not at all for HTTPS. Sophos saw a huge drop of 92% for HTTP but actually

increased throughput by 2% for secured traffic.

0

500

1000

1500

2000

2500

3000


Vendor AProduct

One

Vendor AProduct

Two

FortinetFortiGuard

100D


Thro

ugh

pu

t (M

bp

s)

Firewall and IPS with HTTP and HTTPS Traffic

HTTP

HTTPS




Antivirus Performance on Application Layer

Antivirus (AV) throughput with unidirectional HTTP and HTTPS traffic generated by

the Spirent TestCenter and Spirent Avalanche test system.

Deep packet inspection (DPI) filters packets during data transactions. DPI examines

the packet’s raw data and possibly header as it passes through the appliance’s

inspection point. It is here that it searches for viruses, spam, and other predefined

criteria before it is passed on through the network, or rerouted to another

destination. Performance of the antivirus feature with HTTPS traffic under DPI, rather

than with solely encrypted traffic, yields a more realistic throughput evaluation.

Figure 5: Firewall and Antivirus Performance with Encrypted and Unencrypted

HTTP Traffic

WatchGuard had highest throughput over HTTP and HTTPS than any other vendor. As

expected, WatchGuard’s throughput decreased with antivirus enabled, however it only

decreased slightly for both types of traffic. Vendor A Product One had significant

degradation from its baseline, showing 84% and 90% decrease for HTTP and HTTPS traffic,

respectively. Vendor A Product Two degraded slightly over HTTP and by 86% over secured

traffic. Fortinet had a major shift from its baseline, and even the previous test, having a

degradation of 53% and 80% for HTTP and HTTPS, respectively. Vendor B displayed no

change for HTTP, but dropped significantly for HTTPS by 84% from its baseline. Sophos

experienced a drop from its baseline, but an increase from the previous test for IPS enabled

over HTTP. However, it fell by 84% from its baseline over HTTPS.

0

200

400

600

800

1000

1200

1400


Vendor AProduct

One

Vendor AProduct

Two

FortinetFortiGuard

100D


Thro

ugh

pu

t (M

bp

s)

Firewall and Antivirus with HTTP and HTTPS Traffic

HTTP

HTTPS w/DPI




UTM: Firewall, IPS, and Antivirus Combined Performance on Application Layer

Throughput of AV and IPS features of each device with unidirectional HTTP and

HTTPS traffic generated by the Spirent TestCenter and Spirent Avalanche test system.

Figure 6: Firewall, IPS, and Antivirus Performance with Encrypted and

Unencrypted HTTP Traffic

WatchGuard had the highest throughput of all vendors for AV and IPS features additionally

enabled over HTTP, and trailed only slightly behind Sophos in throughput over HTTPS by

26 Mbps. Its throughput decreased from its baseline by 76% for HTTP and 72% for HTTPS,

its biggest drop yet. Vendor A Product One maintained approximately the same

throughput as the previous test with only firewall and antivirus enabled. Vendor A Product

Two, Fortinet, Vendor B and Sophos degraded in performance by at least 62% for HTTP

and 80% for HTTPS. Sophos displayed the largest drop in HTTP throughput of 94% from its

baseline.

0

100

200

300

400

500

600

700

800

900


Vendor AProduct

One

Vendor AProduct

Two

FortinetFortiGuard

100D


Thro

ugh

pu

t (M

bp

s)

Firewall, IPS, and Antivirus with HTTP and HTTPS traffic

HTTP

HTTPSw/ DPI




UTM: Application Control, Antivirus, and IPS Combined Performance on Application Layer

Throughput of full unified threat management: application control (AppCtrl), AV, and

IPS features of each device on the application with unidirectional HTTP and HTTPS

traffic generated by the Spirent TestCenter and Spirent Avalanche test system.

Application control relates to the data transactions pertaining to specific, individual

applications and is important for protecting endpoint users. DPI is relevant because

the packets of these data transactions need to be carefully analyzed for adept

detection and protection.

Figure 7: Firewall, IPS, Antivirus, and Application Control Performance with

Encrypted and Unencrypted HTTP Traffic

WatchGuard had the highest throughput of all vendors for UTM throughput over both

HTTP and HTTPS, and maintained approximately the same throughput as the previous test

where application control was not yet enabled. Every other vendor had degradation in

throughput by at least 63% for HTTP traffic and 83% for HTTPS. Sophos showed the most

significant drop, by 94% over HTTP as it did when application control was not enabled.

0

100

200

300

400

500

600

700

800

900


Vendor AProduct

One

Vendor AProduct

Two

FortinetFortiGuard

100D


Thro

ugh

pu

t (M

bp

s)

Firewall, IPS, AV, and AppCtrl with HTTP and HTTPS traffic

HTTP

HTTPSw/ DPI




4.0 Fair Test Notification

All vendors with products featured in this report were afforded the opportunity before,

during, and after testing was complete to comment on the results and demonstrate the

performance of their product(s). Any vendor with a product tested by Miercom in one of

our published studies that disagrees with our findings is extended an opportunity for a

retest and to demonstrate the performance of the product(s) at no charge to the

vendor. All vendors are welcome to demonstrate their performance on their own to

Miercom. Miercom will update these results if new data presents itself.

5.0 About Miercom

Miercom has published hundreds of network product analyses in leading trade

periodicals and other publications. Miercom’s reputation as the leading, independent

product test center is undisputed. Private test services available from Miercom include

competitive product analyses, as well as individual product evaluations. Miercom

features comprehensive certification and test programs including: Certified

Interoperable, Certified Reliable, Certified Secure and Certified Green. Products may also

be evaluated under the Performance Verified program, the industry’s most thorough

and trusted assessment for product usability and performance.

6.0 Use of This Report

Every effort was made to ensure the accuracy of the data contained in this report but

errors and/or oversights can occur. The information documented in this report may also

rely on various test tools, the accuracy of which is beyond our control. Furthermore, the

document relies on certain representations by the vendors that were reasonably verified

by Miercom but beyond our control to verify to 100 percent certainty.

This document is provided “as is,” by Miercom and gives no warranty, representation or

undertaking, whether express or implied, and accepts no legal responsibility, whether

direct or indirect, for the accuracy, completeness, usefulness or suitability of any

information contained in this report.

No part of any document may be reproduced, in whole or in part, without the specific

written permission of Miercom or Websense All trademarks used in the document are

owned by their respective owners. You agree not to use any trademark in or as the

whole or part of your own trademarks in connection with any activities, products or

services which are not ours, or in a manner which may be confusing, misleading or

deceptive or in a manner that disparages us or our information, projects

or developments.

Documents

2015 Miercom Unified Threat Management …The Spirent Avalanche tool provides multi-10 Gbps capacity, security, and performance testing for network infrastructures. WatchGuard Firebox