NSSLabs_2013_FW_CAR_SVM.pdf

FIREWALL COMPARATIVE ANALYSIS

Security Value Map (SVM)

2013 – Frank Artes, Thomas Skybakmoen, Bob Walder, Vikram Phatak, Ryan Liles

Tested Products Barracuda F800, Check Point 12600, Cyberoam CR2500iNG, Dell SonicWALL NSA 4500, Fortinet FortiGate 800c, Juniper SRX550, NETASQ ng1000-‐A, NETGEAR ProSecure UTM9S, Palo Alto Networks PA-‐5020, Sophos UTM 425, Stonesoft StoneGate FW-‐1301, WatchGuard XTM 1050

Overview Empirical data from the individual Product Analysis Reports (PAR) and Comparative Analysis Reports (CAR) is used to create the unique Security Value Map (SVM).

This high-‐level document illustrates clearly the relative value of security investment options by mapping security effectiveness and value (cost per protected Mbps) of tested product configurations.

The SVM is designed to provide a high-‐level overview of the detailed findings from NSS Labs group tests. Having examined the high-‐level picture, it is then possible to dig deeper into individual products and capabilities as required via the PAR and CAR documents.

Individual PARs are available for every product tested. CARs provide detailed comparisons across all tested products in the areas of:

• Security • Performance • Management • Total cost of ownership (TCO)

NSS Labs Firewall Comparative Analysis -‐ SVM

© 2013 NSS Labs, Inc. All rights reserved. 2

Figure 1 -‐ 2013 Firewall Security Value Map (SVM)

Barracuda F800

Check Point 12600

Cyberoam CR2500iNG

Dell SonicWALL NSA 4500

Fortinet FortiGate-800c

Juniper SRX550

NETASQ NG1000-A

NETGEAR ProSecure UTM9S

Palo Alto Networks PA-5020

Sophos UTM 425

Stonesoft FW-1301

WatchGuard XTM 1050

Average

Average

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

$1 $2 $4 $8 $16 $32 $64 $128 $256 $512 $1,024 $2,048 $4,096 $8,192

Ente

rpris

e M

anag

emen

t & S

ecur

ity E

ffect

iven

ess

TCO per Protected-Mbps



Key Findings • Protection varied between 4% and 100%, with half of the tested devices achieving greater than 80%. • Price per Protected-‐Mbps varied from $2 to $994 with most tested devices costing below $12.41 per

Protected-‐Mbps. • Median throughput was 3.6 Gbps with a spread from 231 Mbps to 9.7 Gbps. • NETGEAR is clearly not an enterprise product, and thus was excluded from calculations of average to

prevent excessive skewing of the results to the overall detriment of the report.

Product Guidance

NSS Labs’ recommendations are based solely on empirical test data, validated over multiple iterations. While some products fall within “Neutral” quadrants, the table below will indicate “Caution” if the DUT scored below 90% of the average of all devices tested with regard to Protection and Value. The overall quadrant score may remain “Neutral,” but the Protection or Value will be flagged appropriately.

Product Protection & Management Value Overall

Barracuda F800 Neutral Recommended Neutral

Check Point 12600 Recommended Recommended Recommended

Cisco Systems Caution Caution Caution

Cyberoam CR2500iNG Caution Neutral Neutral

Dell SonicWALL NSA 4500 Recommended Neutral Neutral

Fortinet FortiGate-800c Recommended Recommended Recommended

Juniper SRX550 Recommended Recommended Recommended

NETASQ NG1000-A Caution Caution Caution

NETGEAR ProSecure UTM9S Caution Caution Caution

Palo Alto Networks PA-5020 Recommended Neutral Neutral

Sophos UTM 425 Caution Caution Caution

Stonesoft FW-1301 Recommended Recommended Recommended

WatchGuard XTM 1050 Recommended Neutral Neutral

Figure 2 -‐ NSS Labs' 2013 Firewall Recommendations



Security Effectiveness & Cost Potential purchasers should not only consider the range of protection they may achieve using a given product, they should also consider the ability to tune it to a higher level without suffering false positives. Furthermore, total cost of ownership (TCO) should be considered over the life of the product.

All of these factors are taken into consideration when producing the SVM. In addition, a SVM Toolkit is available to NSS clients to allow the incorporation of organization-‐specific costs and requirements to create a completely customized SVM.



Table of Contents: Overview ______________________________________________________________ 1 Key Findings ___________________________________________________________________ 3 Product Guidance ______________________________________________________________ 3 Security Effectiveness & Cost _____________________________________________________ 4

SVM ___________________________________________________________________ 6 What Do The Values Mean? ______________________________________________________ 6 How To Use The SVM ____________________________________________________________ 7

Analysis of Data _________________________________________________________ 8 Recommended _________________________________________________________________ 8 Check Point 12600 ____________________________________________________________ 8 Fortinet FortiGate-‐800c ________________________________________________________ 8 Juniper SRX 550 ______________________________________________________________ 8 Stonesoft StoneGate FW-‐1301 ___________________________________________________ 9

Neutral ______________________________________________________________________ 10 Barracuda NG Firewall F800 ___________________________________________________ 10 Cyberoam CR2500iNG ________________________________________________________ 10 DELL SonicWALL NSA 4500 _____________________________________________________ 11 Palo Alto Networks PA-‐5020 ___________________________________________________ 11 WatchGuard XTM 1050 _______________________________________________________ 11

Caution ______________________________________________________________________ 12 Cisco Systems _______________________________________________________________ 12 NETASQ NG1000 A ___________________________________________________________ 12 NETGEAR ProSecure UTM9S ____________________________________________________ 13 Sophos UTM 425 ____________________________________________________________ 13

Test Methodology ______________________________________________________ 14

Contact Information _____________________________________________________ 14

Table of Figures Figure 1 -‐ 2013 Firewall Security Value Map (SVM) _______________________________________________ 2 Figure 2 -‐ NSS Labs' 2013 Firewall Recommendations _____________________________________________ 3 Figure 3 -‐ Example SVM _____________________________________________________________________ 6



SVM

What Do The Values Mean?

The SVM depicts the value of a deployment of ten firewall devices and the appropriate enterprise/central management console for each vendor. The Management Comparative Analysis Report (CAR) outlines the pricing and structure needed for the management infrastructure if a more robust setup is desired. Additionally, the 2013 Firewall Management CAR outlines multiple cost-‐modeled deployments for those interested in distributed management deployment scenarios.

Figure 3 -‐ Example SVM

The x-‐axis charts the Total Cost of Ownership per Protected Mbps, a value that incorporates the 3 year TCO with measured performance to provide a single figure that can be used to compare the real cost of each device tested. Further to the right (lower cost) is better.

The y-‐axis charts the enterprise management capabilities and security effectiveness as measured via the NSS security management review and effectiveness tests. The security effectiveness of a product as tested is multiplied by the score for enterprise management as tested. Devices that are missing critical security OR management capabilities will have a reduced score on this axis. Further up (higher effectiveness) is better.



How To Use The SVM Mapping the data points against the Average Protection and Average Value results in four quadrants on the SVM.

Further up and to the right is the best. The upper-‐right quadrant contains those products that are Recommended for both security effectiveness/management and value. These devices provide a very high level of protection, manageability and value for money.

Further down and left is poor, and the lower left quadrant would comprise the NSS Labs Caution category – these products offer poor value for money given the 3 year TCO and measured security effectiveness/management rating.

The remaining two quadrants comprise the NSS Labs Neutral category. These products may still be worthy of a place on your short list based on your specific requirements.

For example, products in the upper-‐left quadrant score as Recommended for security effectiveness, but Neutral for value. These products would be suitable for environments where security is paramount since they offer an extremely high level of protection, although at a higher than average cost.

Conversely, devices in the lower-‐right quadrant score as Neutral for security effectiveness, but Recommended for value. These devices would be suitable for environments where budget is paramount, and a slightly lower level of protection is acceptable in exchange for a lower cost of ownership.

Note that while some products are clearly within “Neutral” quadrants, they will be rated as “Caution” should they fall outside the 10% band highlighted on the chart in Figure 3. This is to ensure that products are not awarded an inappropriately high rating should they fall too far below the average of all products tested. In such cases, the overall quadrant score may remain “Neutral,” but the Protection or Value will be flagged appropriately.

In all cases, the SVM should only be a starting point. NSS Labs clients can schedule an inquiry call (or a written response) with one of the analysts involved in the actual testing and report production. Only by combining the wealth of knowledge contained within these reports and the experience and direct feedback from our analysts based on your own unique requirements can you make the right decision.



Analysis of Data

Recommended

Check Point 12600

The 12600 was rated by NSS Labs at 8.4 Gbps out of the 10 Gbps claimed by the vendor.1 The 12600 scored 100% for Stability, 100% for Evasion, 100% for Leakage, and 100% in the central management review. All of which resulted in a TCO of $13 per protected megabit, and 100% for security and management effectiveness.

Check Point’s management system is flexible and granular, allowing for a high degree of customization. With this level of flexibility, however, comes some complexity. For current or experienced enterprise users of Smart-‐1 who have been managing NGFW and IPS through Check Point’s SmartDashboard, there will not be a significant learning curve. New administrators should take their time learning the features and building the foundation of their object groups. Check Point currently only offers the management client as a Windows executable, but the management system, overall, is the most mature and feature-‐complete in its class.

For an in-‐depth evaluation of security, management, performance and TCO, please see the Check Point 12600 Product Analysis Report (PAR).

Fortinet FortiGate-‐800c

The FortiGate-‐800c was rated by NSS Labs at 9.7 Gbps out of the 20 Gbps claimed by the vendor.1 The 800c scored 100% for Stability, 100% for Evasion, 100% for Leakage, and 100% in the central management review. All of which resulted in a TCO of $4 per protected megabit, and 100% for security and management effectiveness.

Fortinet’s management interface was reasonably well designed, although the organization of items and menus proved less than intuitive. The policy is based on a Virtual Domain (VDOM) organization, grouping policy objects based on their area of effect, which may create confusion for administrators that are not familiar with this method. For users of Fortinet firewalls or IPS, there will not be a significant learning curve. Tuning and maintenance is achieved easily, once the VDOM organizational method is understood. Management currently does not support event correlation, and log aggregation requires an additional purchase.

For an in-‐depth evaluation of security, management, performance and TCO, please see the Fortinet FortiGate-‐800C Product Analysis Report (PAR).

Juniper SRX 550

The Juniper SRX550 12.1r2 was rated by NSS Labs at 2.1 Gbps out of the 5.5 Gbps claimed by the vendor.1 The SRX 5500 scored 100% for Stability, 100% for Evasion, 100% for Leakage, and 100% in the central management review. All of which resulted in a TCO of $20 per protected megabit, and 100% for security and management effectiveness.



Juniper’s Junos Space combined with the Security Threat Response Manager (STRM) provides the replacement for the NSM/STRM management system. Space is a graphically pleasing, feature-‐rich user interface. The management interface is easy to navigate and the layout is reasonably intuitive. From failsafe features like rollback policies to prescheduling policy and rule changes, Space is straightforward to use. There is a lack of integration between Space and STRM at this time, but Juniper has stated it is working to resolve this issue. Regardless, the system has very robust logging and audit / change logs that are easy to navigate and filter. The help documentation included is very useful, and administrators should have few problems learning the system.

Space can manage as few as 25 devices and new devices can be added via 100 devices license packs. There is no stated limited as to the maximum number of managed devices that can actively be connected to the management server at once, but Juniper should be consulted for information on CPU, RAM, and storage requirements as the managed licenses are increased.

For an in-‐depth evaluation of security, management, performance and TCO, please see the Juniper SRX 550 Product Analysis Report (PAR).

Stonesoft StoneGate FW-‐1301

The StoneGate FW-‐1302 was rated by NSS Labs at 5.1 Gbps out of the 7.5Gbps claimed by the vendor.1 The FW-‐1301 scored 100% for Stability, 100% for Evasion, 100% for Leakage, and 100% in the central management review. All of which resulted in a TCO of $13 per protected megabit, and 100% for security and management effectiveness.

The Stonesoft Management Center has been designed from the ground up as a flexible and powerful large enterprise or service provider management system. Administrator access is via extremely granular role-‐based mechanisms. Policy management and deployment is straightforward and extremely flexible, with grouping and inheritance capabilities providing the ability to deploy complex policies across multiple devices with ease. The ability to deploy sub-‐policies for individual devices beneath a hierarchy of global policy templates makes this product ideal for multi-‐tenanted service provider environments. Alert handling is powerful, with multiple means of achieving the same end. Flexible real-‐time filter definition provides rapid drill-‐down to pertinent information and the ability to save ad hoc filters for reuse later is very useful. Unique to Stonesoft’s Management Center are robust investigation and forensic capabilities.

The only drawback, in certain environments, is the lack of direct device management capabilities. All Stonesoft deployments – even for a single device – require the three-‐tier management system, making this solution less cost effective for some SMB environments. Those customers who don't wish to take on enterprise level management have the option to work through one of Stonesoft's managed security service providers (MSSP). However, for large-‐scale enterprise and service provider environments, the Stonesoft management solution is well suited.

For an in-‐depth evaluation of security, management, performance and TCO, please see the Stonesoft StoneGate FW-‐1301 Product Analysis Report (PAR).



Neutral

Barracuda NG Firewall F800

The NG Firewall F800 remained functional through most of NSS Labs’ performance and security testing, though some stability issues were noted initially. The device was rated by NSS Labs at 7.8 Gbps out of the 9.2 Gbps claimed by the vendor.1 The F800 scored 80% for Stability, 100% for Evasion, 100% Leakage, and 95% in the central management review. All of which resulted in a TCO of $11 per protected megabit, and 76% for security and management effectiveness.

Barracuda Networks’ management interface is reasonably well designed and intuitive, although there were some instances where options and configuration parameters were difficult to find or appeared out of place. Tuning and maintenance is straightforward once the complexities of the interface have been mastered. For users of Barracuda Networks firewalls, there will not be a significant learning curve. However, there may be a steep learning curve for those new to the interface. The good news is that there is excellent documentation to help overcome any difficulties.

For an in-‐depth evaluation of security, management, performance and TCO please, see the Barracuda NG Firewall F800 Product Analysis Report (PAR).

Cyberoam CR2500iNG

The CR2500iNG was rated by NSS Labs at 8.7 Gbps out of the 32 Gbps claimed by the vendor.1 The CR2500iNG scored 100% for Stability, 100% for Evasion, 100% Leakage, and 55% in the central management review. All of which resulted in a TCO of $19 per protected megabit, and 55% for security and management effectiveness.

CCC presents a clean user interface with tabs across the top of the screen for all areas of firewall management. The interface is intuitive and easy to use by any experienced administrator. Reminiscent of the Cisco device management webUI, the interfaces are clean and technical.

The interface lacks cross-‐connect functionality, however. Administrators are forced to copy and paste information between screens, or enter redundant information multiple times in different places. The system is Java-‐based, and this often causes compatibility issues with different browsers, as well as rendering the interface useless for some mobile tools, such as an iPad.

iView is actually quite a robust and feature-‐rich logging and reporting tool. It provides an alternative for those environments without a dedicated SIM/SIEM. However, with the logs and the firewall management implemented via two different interfaces with no integration, it makes the system more difficult to manage than it should be.

For an in-‐depth evaluation of security, management, performance and TCO, please see the Cyberoam CR2500iNG Product Analysis Report (PAR).



DELL SonicWALL NSA 4500

The SonicWALL NSA 4500 was rated by NSS Labs at 850 Mbps out of the 990Mbps claimed by the vendor.1 The NSA 4500 scored 100% for Stability, 100% for Evasion, 100% Leakage, and 95% in the central management review. All of which resulted in a TCO of $35 per protected megabit, and 95% for security and management effectiveness.

DELL SonicWALL’s management interface is well designed and comprehensive, though the breadth of advanced features comes at the cost of complexity. This is certainly not an interface that will be mastered quickly. However, it does offer some highly evolved features suitable for large enterprise and multi-‐tenanted/service provider deployments, making it straightforward to apply complex policies in a targeted manner across multiple nested groups in large-‐scale deployments. DELL plans to offer an iOS client that will provide administrators the ability to review logs and activity in real-‐time.

Tuning and maintenance is straightforward once the complexities of the interface have been mastered, and deployment of complex, fine-‐grained policies across large organizations is made easy thanks to the implementation of advanced features such as nested groups and inheritance.

For an in-‐depth evaluation of security, management, performance and TCO, please see the Dell SonicWALL NSA 4500 Product Analysis Report (PAR).

Palo Alto Networks PA-‐5020

The Palo Alto Networks PA-‐5020 was rated by NSS Labs at 4.1 Gbps out of the 5 Gbps claimed by the vendor.1 The PA-‐5020 scored 100% for Stability, 100% for Evasion, 100% Leakage, and 95% in the central management review. All of which resulted in a TCO of $24 per protected megabit, and 95% for security and management effectiveness.

Palo Alto Networks’ management interface was reasonably intuitive for most tasks, making it relatively straightforward to use without extensive training.

Tuning and maintenance is straightforward, making it suitable for environments where only occasional updates are expected or where there is a lack of extensive on-‐site expertise. However, certain features, such as the lack of identification of application dependencies during policy creation and lack of support for group management, made it clumsy to use.

Those used to the more traditional port-‐ and protocol-‐based security ACL rules will struggle with the lack of granularity in Palo Alto’s rules. The simple, single detection engine approach will find favor with SMB users, however.

For an in-‐depth evaluation of security, management, performance and TCO, please see the Palo Alto Networks PA-‐5020 Product Analysis Report (PAR).

WatchGuard XTM 1050

The WatchGuard HTM 1050 was rated by NSS Labs at 2.2 Gbps out of the 10 Gbps claimed by the vendor.1 The XTM 1050 scored 100% for Stability, 100% for Evasion, 100% Leakage, and 85% in the central management review. All of which resulted in a TCO of $40 per protected megabit, and 60% for security and management effectiveness.



WatchGuard provides a suite of management applications for use with its centralized server and managed devices, but while the firewalls all are capable of high availability (HA), the management server lacks any fault tolerance / redundancy features.

While it is possible to review prior firewall configurations following modifications, there is no delta view. Administrators are forced to manually identify changes from one saved configuration to another. Other features such as drag-‐and-‐drop VPN construction, and the ability to pre-‐configure new firewalls using a free cloud-‐based configuration service, allow for rapid deployment at remote offices.

For an in-‐depth evaluation of security, management, performance and TCO, please see the WatchGuard XTM 1050 Product Analysis Report (PAR).

Caution

Cisco Systems

Cisco was not included in the 2013 firewall test since it does not currently have an enterprise class firewall in its product line. The Adaptive Security Appliances are unified threat management (UTM) devices and thus not optimized for deployment as dedicated firewalls. According to Cisco representatives there is a dedicated firewall device in development, and NSS is looking forward to testing this shortly. Until that time, NSS recommends that enterprises looking to purchase a dedicated firewall solution should consider other alternatives.

NETASQ NG1000 A

The NETASQ NG1000 A was rated by NSS Labs at 2.5 Gbps out of the 7 Gbps claimed by the vendor.1 The NG1000 A scored 100% for Stability, 70% for Evasion, 100% Leakage, and 100% in the central management review. All of which resulted in a TCO of $27 per protected megabit, and 70% for security and management effectiveness.

Administrators are presented with a worldview showing deployed firewalls and indicating VPN tunnels, all of which are color coded to reflect the health/status of each managed device and tunnel. Administrators can drill down into each device, and are presented with a clean interface allowing for quick management of the devices.

While the management console centralizes the logs from the managed devices, a second product, NETASQ Event Analyzer, is required to view this data. Event Analyzer is also used to generate all standard firewall event reports.

Centralized Manager automatically creates backups of firewall configurations during an update and administrators are able to navigate through these backup images to restore a firewall to a prior state.

The interface is crisp and quick to respond, and intuitive for an experienced administrator. Rule creation is robust, allowing administrators to define complex rules quickly and without the interference of restrictive parameters often found on such interfaces.

For an in-‐depth evaluation of security, management, performance and TCO, please see the NETASQ NG1000 A Product Analysis Report (PAR).



NETGEAR ProSecure UTM9S

The NETGEAR ProSecure UTM9S was rated by NSS Labs at 231 Mbps out of the 850 Mbps claimed by the vendor.1 The device had issues during the stability and reliability tests, however. The UTM9S scored 40% for Stability, 70% for Evasion, 100% Leakage, and 15% in the management interface review (there is no central management option). All of which resulted in a TCO of $5,950 per protected megabit, and 4% for security and management effectiveness.

NETGEAR does not have a centralized management console, forcing administrators to manage all deployed firewalls one-‐on-‐one through direct device management (DDM). This does not scale in an enterprise environment. Administrators are limited by the DDM interface, which uses inflexible web-‐forms for reporting, access control list development, NAT, etc. The interface is overly restrictive with its use of drop-‐down and check-‐box menu items that have been migrated from NETGEAR’s home firewall appliances.

For an in-‐depth evaluation of security, management, performance and TCO, please see the NETGEAR ProSecure UTM9S Product Analysis Report (PAR).

Sophos UTM 425

The Sophos UTM 425 was rated by NSS Labs at 3 Gbps out of the 6 Gbps claimed by the vendor.1 The UTM 425 scored 100% for Stability, 70% for Evasion, 100% Leakage, and 65% in the central management review. All of which resulted in a TCO of $44 per protected megabit, and 46% for security and management effectiveness.

Sophos UTM Manager has some limitations, and the administrator must open a session to the direct device management port of the firewall to fulfill certain tasks. These include access to change control logs, granular firewall transaction log data, front panel view, and port status and utilization information for the device.

Logs are presented as tab delimited text files in new windows within the browser. Administrators are required to compile and normalize such data into spreadsheets, or feed it into a SIM/SIEM to filter, sort and parse the output.

Sophos has included UTM features, such as Insight, into the anti virus report data coming from protected endpoints. However, the system is missing features commonly found on enterprise-‐class central management systems, such as transaction roll-‐back and failsafe checks on new configurations prior to deployment. Administrators are required to make DDM connections to firewalls to correct these issues.

For an in-‐depth evaluation of security, management, performance and TCO, please see the Sophos UTM 425 Product Analysis Report (PAR).



© 2013 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors.

Please note that access to or use of this report is conditioned on the following:

1. The information in this report is subject to change by NSS Labs without notice.

2. The information in this report is believed by NSS Labs to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this report are at the reader’s sole risk. NSS Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this report.

3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY NSS LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-‐INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY NSS LABS. IN NO EVENT SHALL NSS LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.

4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the reader’s expectations, requirements, needs, or specifications, or that they will operate without interruption.

5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report.

6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective owners.

Test Methodology

Methodology Version: Firewall v4

A copy of the test methodology is available on the NSS Labs website at www.nsslabs.com

Contact Information NSS Labs, Inc. 206 Wild Basin Rd, Suite 200A Austin, TX 78746 USA +1 (512) 961-‐5300 [email protected] www.nsslabs.com

v2013.02.07

This and other related documents available at: www.nsslabs.com. To receive a licensed copy or report misuse, please contact NSS Labs at +1 (512) 961-‐5300 or [email protected].

Documents

NSSLabs_2013_FW_CAR_SVM.pdf