Tolly report: Trend Micro Deep Security 7.5 vs. McAfee and Symantec

  • Published on
    27-Nov-2014

  • View
    3.871

  • Download
    4

Embed Size (px)

DESCRIPTION

Tolly Report: Trend Micro Deep Security 7.5 vs. McAfee and SymantecVoor meer informatie:itsolutions@unit4.com

Transcript

  • 1. #211101February 2011Commissioned by Trend Micro, Inc. Trend Micro Deep Security 7.5 vs. McAfee and SymantecAnti-virus Performance in VMware ESX Virtual EnvironmentsExecutive Summary TEST HIGHLIGHTSServer and desktop virtualization are essential elements of any ITThe Trend Micro Deep Security Virtual Appliance:strategy that seeks to decrease capital and operationalexpenditures . In the rush to implement virtualization technologies, 1Demonstrated consistently lower demand forsystem CPU, memory and disk I/O over traditionalmany organizations simply deploy the same anti-virus solution thatis in use on their physical server and desktop systems. Because these agent-based solutions even during periods whentraditional anti-virus solutions are not designed specifically forthe workload was designed not to stress AVvirtual environments, they can create significant operational issuessuch as anti-virus (AV) storms, resource wastage and administrativeoverhead, and hamper the organizations objective of maximizing2Successfully avoided AV storm issues withscheduled scans and pattern updates thatVM densities.prevented other solutions from testing beyondTrend Micro, Inc. commissioned Tolly to benchmark the 25 VMsperformance within virtual environments of the Trend Micro DeepSecurity solution vs. McAfee Total Protection for Endpoint and 3Symantec Endpoint Protection 11.0. Specifically, this testingDemonstrated density improvements of 29% toevaluated the impact each solution had on host system (physical275% over McAfee and Symantec running testserver) resources especially as guest machine density increased to workloadsup to 100 virtual machines simultaneously running in a VMware ESX4.1 environment.Tests showed that Trend Micro Deep Security, which provides an agentless virtual appliance-based approach to anti-virus protection optimizedfor virtualization, consistently consumed less CPU, RAM and disk I/O resources than the non VM-aware implementations where anti-virusagents and processing resided in each and every Windows 7 virtual machine.In addition to consuming 1.7 to 8.5 times the resource overhead of the Trend Micro solution in the general workload test, the traditional AVsolutions were seen to face AV storm challenges when tested at peak activities (i.e., running on-demand scans and signature updates) whenoperations on 25 VMs were triggered simultaneously. Specifically, when Tolly engineers attempted to remediate the competing systemsimmediately and, because the traditional solutions were not VM-aware, management station requests for, say, 25 virtual machines to run on-demand scans or update signature files triggered all of the virtual machines to begin execution of the function simultaneously resulting in asurge in demand on host resources such as CPU and memory.Ultimately the savings in resource consumption afforded by Trend Micro Deep Security allows organizations to increase virtual machinedensities, i.e. the number of VMs that can be run per host, enabling capital expenditure (CAPEX) and operational expenditure (OPEX) savingsfor the organization. The VM density improvement made possible because of Trend Micros lower resource consumption and AV stormavoidance in the proprietary workload tests ranged from a minimum of 29% (when running a workload that did not stress AV) to a maximumof 275% (during AV storm periods) over McAfee and Symantec 2011 Tolly Enterprises, LLCTolly.comPage 1 of 9

2. Trend Micro Deep Security 7.5 #211101Anti-virus VMware ESX 4.1 Host Resource Consumption vs. BaselineTolly engineers benchmarked security Up to 100 Virtual Machines Running Proprietary Workload under Microsoftsystem resource utilization by runningWindows 7various workloads on up to 100 virtual As reported by vCenter (Lower numbers are better)machines simultaneously. A baseline wasestablished by running a workload CPU% vs. Baselinesimulating various end-user functions onsystems that had no endpoint securitysolution installed and measuring resource40consumption. 20Testing included benchmarking resource0consumption when running specific anti-25 50 75 100virus tasks (on-demand scan and signature # of Virtual Machinesupdates) as well as a more general userworkload with anti-virus protection present RAMon each virtual machine. 70Anti-virus Resource Utilization with 60% vs. BaselineSimulated Workload (25 to 10050Virtual Machines)40 30Figure 1 illustrates average utilization levelsof key system resources at the VMware ESX 20server level when running the primary test 10workload with up to 100 simultaneous0virtual machines. (See Table 4 for individual25 50 75 100data points.)# of Virtual MachinesThese figures include the resources used bythe virtual machines as well as, for Trend DISKMicro, the resources used by the Deep % vs. BaselineSecurity virtual appliance. See the Test 400Methodology and Testbed Setup section fordetails on the workloads and environment.Both McAfee and Symantec solutions -15required that a separate instance of the AV 25 5075 100agent run in each virtual machine. TrendMicro Deep Security required one instance# of Virtual Machinesof its virtual appliance per host. The figure Trend MicroMcAfeeSymantecillustrates how, at all VM density levels, and Note: All systems running proprietary workload in addition to scan. Baseline is proprietaryacross all three resources - CPU, Memory workload running with no endpoint security solution installed. See report body for baseline valuesand Disk Usage. Symantec and McAfeeand detailed results. Utilization over baseline is calculated by subtracting baseline from result,consumed 1.7 to 8.5 times the amount ofdividing by baseline and multiplying by 100. As McAfee was unable to complete the 100 VM test,resource overhead required by Trend Micro.1results for 100 were extrapolated from the 25, 50 and 75 VM tests. Average of 30 minute run. Disk usage results vary up to 30% and are include for reference purposes only. Source: Tolly, October 2010Figure 11 The McAfee solution was unable to complete the 100 VM test despite multiple attempts and re-runs.. Tolly engineers extrapolated the McAfee 100 VM results from the McAfee 25, 50 and 75VM test results. 2011 Tolly Enterprises, LLCTolly.comPage 2 of 9 3. Trend Micro Deep Security 7.5 #211101Anti-virus VMware ESX 4.1 Host Resource Consumption Overhead vs. Baseline Request On-Demand Scan of 25 Virtual Machines Running Microsoft Windows 7As reported by vCenter (Lower numbers are better)394 350 308 400 2500 300274 2,053 2,143 320 2000% vs. Baseline% vs. Baseline% vs. Baseline 250 200 240 1500183 150 160 100081 693 1008032 50050 0 0 0CPURAMDiskTrend MicroMcAfee Symantec Note: All systems running proprietary workload in addition to scan. Baseline is proprietary workload running with no endpoint security solution installed. Baseline values: Average CPU = 4,109.76 MHz, Average RAM = 7,893.28 MB, Average Disk = 1,741.23 KBps. Trend automatically runs only a single scan at one time. Other vendors triggered 25 simultaneous scans. Each vendor recommends various methods such as randomization for load- leveling on-demand scans. See report body for details. Utilization over baseline is calculated by subtracting baseline from result, dividing by baseline and multiplying by 100. Average of 30 minute run. Source: Tolly, October 2010 Figure 2Anti-virus On-Demand Scans By default, the other solutions (that are during the test and disk latency (not(25 VMs) Testunaware of the shared, virtual environment) illustrated in the figures) was noted to attempted to initiate simultaneous scans of average 31 ms. With the McAfee on-Engineers evaluated how each solutionall 25 machines. Figure 2 provides thedemand scan scenario, disk latency wasresponded to a security management average resource results for those testsnoted to average 80 ms. During the test, 14system request to conduct a full scan on 25where McAfee resource consumption out of 25 users were not able to access theirvirtual machines. Being resource intensive inoverhead was 2.8 times more than Trenddesktops. See Table 2 for additionalnature, simultaneous scans can degrade Micro for CPU and 11 times for RAM. commentary.overall user experience. Symantec resource consumption overhead was 2.4 times more than Trend Micro for Traditional solutions generally recommendTrend Micro Deep Security was aware that ittwo approaches to avoid vir tual CPU and 4.7 times for RAM.was running in an environment whereenvironment resource contention -resources were shared across all VMs and In addition, the 25 VM data set for Symantecrandomization and grouping. Neither ofautomatically scheduled scans to run seriallyand McAfee does not provide the completet h e s e a p p ro a c h e s p ro v i d e s a ny- a maximum of 1 machine running at apicture with respect to reliability and uservirtualization awareness and, thus, weretime. As a result, Deep Security was able to experience. The surge in resource demandoutside the scope of this test.successfully test at 25 and 50 VMs. Based on from the McAfee and Symantec solutionsthe resource utilization observed in these often degraded the user systems. In With randomization, an administrator cantests, Tolly projects that the Trend Micro particular, neither Symantec nor McAfee set up the randomization period to letsolution could support a scenario of moresolutions were able to be tested beyond 25endpoints run tasks with random startthan 100 VMs.VMs. In the Symantec test, 2 agents losttimes. For time consuming tasks like full connectivity with the management server scan, this time period needs to be very long 2011 Tolly Enterprises, LLCTolly.comPage 3 of 9 4. Trend Micro Deep Security 7.5 #211101(more than a day or a week depending other. As a result, when facing a critical immediately. Also, the random tasks mayupon the hosts VM density) to increase thesecurity threat, enterprise administrators degrade user experience if they run whenchances that client tasks wont overlap each may not be able to remediate their systems system usage is already high.Anti-virus Solution Scalability Under VMware ESX 4.1 On-Demand Scan Scenarios of Virtual Machines Running Microsoft Windows 7 Vendor ProductNumber of Virtual Machines Targeted for On-Demand Scan e a 25 50 75100Trend Micro, Deep Security Yes, completelyYes, completely stableYes (projected, notYes (projected, not tested)Inc. 7.5 stable tested)McAfee Total Yes, but withBecause of instability problems w 25 simultaneous scans, To engineers did notwith olly Protection forstability eattempt greater numbers. McAfee oers a randomization optio in its client task thaton Endpointproblemsorcould provide load distribution fo such both scheduled and m manually triggered tasks.Symantec Endpoint Trend Yes, but with MicroMcAfeeSymantecBecause of instability problems w 25 simultaneous scans, To engineers did notwith olly Protection stability attempt greater numbers. Symantec recommends conguring scheduled tasks for g 11.0 problemsrandomization. This would spread the on-demand scan reques for 100 virtual machinesd ststo approximately 160 hours by deefault. Manually triggered tasks cannot have randomizedstart times. Note: Trend Micro is the only virtualization-aware solution tested and automatically staggers on-demand scans so that scans are performed serially. Source: Tolly, October 2010 Table 1Anti-virus Solution VMware ESX 4.1 Host Resource Consumption vs. BaselineRequest Signature Update of 50 Virtual Machines Running Microsoft Windows 7 As reported by vCenter (Lower numbers are better)197 1202002000104 1001,558 1601600% vs. Baseline% vs. Baseline% vs. Baseline80 12012006075806680040 29 26 2933772040400 0 00CPURAM DiskTrend Micro McAfeeSymantec Note: All systems running proprietary workload in addition to test task. Baseline is proprietary workload running with no endpoint security solution installed. Baseline values: Average CPU = 8,434.91 MHz, Average RAM = 14,119.62 MB, Average Disk = 2,341.41 KBps. Trend only needs to download the signature le to its single virtual security appliance. Other vendors triggered 25 simultaneous updates. Each vendor recommends various methods for load-leveling updates. See report body for details. Utilization over baseline is calculated by subtracting baseline from result, dividing by baseline and multiplying by 100. Average of 15 minute run. Source: Tolly, October 2010 Figure 3 2011 Tolly Enterprises, LLC Tolly.com Page 4 of 9 5. Trend Micro Deep Security 7.5 #211101With grouping, an administrator can assign enterprise IT management more balancing or other reasons, administratorsVMs to different groups and schedule clientcomplicated. New VMs need to be allocated have to update the group assignmentstasks by group. This approach requires manually to groups and, if VMs get migrated accordingly.administrative work and makes thefrom one host to the other for loadVirtualized Anti-virus Test Environment Source: Tolly, October 2010 Figure 4 Systems Under TestVendorProduct Components VirtualImplementation MachineAwareTrend Deep Security Trend Micro Deep Security Manager version 7.5.1378; Trend Micro Deep Yes Automatic, single virtualMicro, Inc. 7.5 Security Virtual Appliance 7.5.0.1600; Filter Driver 7.0.0.894; Defaultappliance. Agentlessconguration. Assigned the pre-congured Windows Anti-Malwareclient communicates viaProtection security prole.VMware vShield APIMcAfeeTotal McAfee ePolicy Orchestrator 4.5; McAfee Agent for Windows 4.5.0 MinorNoTraditional endpointProtection forVersion 1270; McAfee VirusScan(R) Enterprise 8.7.0 Minor version 570 withclientEndpointHot Fix 2; McAfee AntiSpyware Enterprise 8.7 Minor version 129; McAfeeHost Intrusion Prevention 7.0.0 minor Version 1070; McAfee SiteAdvisor(R)Enterprise Plus 3.0.0 Minor version 476 All with default policies. Cancelledpre-congured Full Scan and Update client tasks.SymantecEndpointVersion 11.0.6100.645NoTraditional endpointProtection 11.0client Source: Tolly, October 2010Table 2 2011 Tolly Enterprises, LLC Tolly.com Page 5 of 9 6. Trend Micro Deep Security 7.5 #211101Anti-virus Signature (Pattern)Although not used for this test, engineersUpdate (50 VMs) Testnoted that the McAfee solution included aTrend Micro, Inc.task for idle VMs to update their signatureEngineers evaluated how each solution files once each day. While engineers Deep Securityresponded to a systemwide anti-viruscancelled this task, it was noted that the taskpattern update request. Pattern updates,would still initiate automatically. 7.5while less resource-intensive than full scans,are still known to create performance As with Symantec, the resources consumed VMwaredegradation and raise operational when 50 VMs are being updatedAnti-viruschallenges especially if they are run duringsimultaneously can be significant and Tested Performanceregular business hours. engineers noted that VMware ESX system OctoberCPU usage remained at 100% for more than 2010Engineers ran the signature update scenario 10 minutes in some test runs and that...