© 2017 SPLUNK INC.© 2017 SPLUNK INC.

Hunting Botnets:Suricata & Splunk Advanced Security Analytics

Anthony Tellez, CISSP, CEH, CNDALead Consultant, Emerging Technologies | SplunkNovember 2017

© 2017 SPLUNK INC.

During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC.

The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.

Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved.

Forward-Looking Statements

THIS SLIDE IS REQUIRED FOR ALL 3 PARTY PRESENTATIONS.

© 2017 SPLUNK INC.

Biography Who Am I?

• Time In Service @ Splunk > 3 Years• Previous: U.S. Gov Contractor, Geospatial Analyst• Specializations

• Cryptography• Information Security – Red Team

• Data Scientist• Security Analytics• Data Visualization

• Responsible for the relationship between emerging technologies and field organization• Acquisitions• Incubation• Product Development• https://github.com/anthonygtellez/

© 2017 SPLUNK INC.

What is Splunk?A T-shirt company that also sells software.

© 2017 SPLUNK INC.

Turning Machine Data Into Business Value

5

Index Untapped Data: Any Source, Type, Volume Ask Any Question

Application Delivery

Security, Compliance, and Fraud

IT Operations

Business Analytics

Industrial Data andIoT

Online Service

sWeb

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Application

s

CustomApplication

sMessaging

TelecomsOnline

Shopping Cart

Web Clickstrea

ms

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

© 2017 SPLUNK INC.

Deviation from past behaviorDeviation from peers

(aka Multivariate AD or Cohesive AD)Unusual change in features

ITSI MAD Anomaly Detection

Predict Service Health ScorePredicting ChurnPredicting EventsTrend Forecasting

Detecting influencing entitiesEarly warning of failure – predictive

maintenance

Identify peer groupsEvent CorrelationReduce alert noiseITSI Event Analytics

Anomaly detection Predictive Analytics Clustering

Splunk Customers Have ML Problems

© 2017 SPLUNK INC.

Copyright©2016SplunkInc.

Historical Data Real-time Data Statistical Models

DB, Hadoop/etc, Splunk

T – a few days T + a few days

Why is this so challenging using traditional methods?

• DATA IS STILL IN MOTION, still in a BUSINESS PROCESS. • Enrich real-time MACHINE DATA with structured HISTORICAL DATA• Make decisions IN REAL TIME using ALL THE DATA• Combine LEADING and LAGGING INDICATORS (KPIs)

Splunk

SecurityOperationsCenter

NetworkOperationsCenter

BusinessOperationsCenter

Machine Learning

© 2017 SPLUNK INC.

Overview of ML at Splunk

CORE PLATFORM SEARCH

PACKAGED PREMIUM SOLUTIONS

MACHINE LEARNING TOOLKIT

Platform for Operational Intelligence

© 2017 SPLUNK INC.

Machine Learning in Splunk ITSI

Adaptive Thresholding:● Learn baselines & dynamic thresholds● Alert & act on deviations● Manage for 1000s of KPIs & entities● Stdev/Avg, Quartile/Median, Range

Anomaly Detection:● Find “hiccups” in expected patterns● Catches deviations beyond thresholds● Advanced proprietary algorithms

© 2017 SPLUNK INC.

Machine Learning in Splunk ITSI (cont.)

Event Analytics:● Prioritize event insights with

service context, logs & metrics● Group related events to highlight

the most meaningful ones● Reduce noise and alert on

root causes of issues

● Use ML algorithms to group similar events (Smart Mode)

© 2017 SPLUNK INC.

Splunk User Behavior Analytics (UBA)

● Understand normal & anomalous behaviors for ALL users

● UBA detects Advanced Cyberattacks and Malicious Insider Threats

● Lots of ML under the hood:– Behavior Baselining & Modeling– Anomaly Detection (30+ models)– Advanced Threat Detection

● E.g., Data Exfil Threat:– “Saw this strange login & data transfer

for user mpittman at 3am in China…”– Surface threat to SOC Analysts

© 2017 SPLUNK INC.

▶ Assistants: Guide model building, testing, & deployment for common tasks

▶ Showcase: 25+ interactive examples from IT, security, business, and IoT

▶ Algorithms: 25+ standard algorithms plus an extensibility API

▶ SPL ML Commands: New commands to fit, test, and operationalize models

▶ Python for Scientific Computing Library: 300+ open-source algorithms

Splunk Machine Learning Toolkitextends Splunk with new tools and guided modeling

© 2017 SPLUNK INC.

▶Data Science Methodology• What is the process a data scientist follows?

▶Quantitative v. Qualitative Analysis• Avoiding alert fatigue w/ Descriptive Statistics

▶Exploratory Data Analysis (EDA)• Visualization & Searching

▶Machine Learning & Security Analytics• The new “hotness”

Data Science & SuricataQuick review

Identity and Access

Internal Network Security

Endpoints

WAF & AppSecurity

Threat Intelligence

Network

Web ProxyFirewall

© 2017 SPLUNK INC.

• Problem ML is trying to solve:• Information Overload

• IDS alerts, Virus Scans, tools.• ML is not a replacement for expert

analysts, or engineers.• Multidisciplinary approach is needed for

next gen problems, fusion of 3 approaches is required to frame and understand the problem before applying analytics.

Security Analytics & Machine Learning

14

© 2017 SPLUNK INC.

Data Science Process using SplunkUsing a process helps define use case requirements:

Step 1 Scope relevant machine data to onboard. 🤓

Step 2 Collect requirements and validate relevant machine data. 🙃

Step 3 Exploratory Data Analysis. (Searching & Visualizing!) 🤔

Step 4 Formulate hypothesis working with Domain Experts. 😏

Step 5 Test and repeat steps as needed until hypothesis is answered. 😩

© 2017 SPLUNK INC.

Security Patterns in IT DataUtilizing the Analytic Capabilities of Splunk!What To Look For Data Source

Abnormally high number of file transfers to USB or CD/DVD Operating system

Abnormally high number of files or records downloaded from an internal file store or database containing confidential information File server / Database

Abnormally large amount of data emailed to personal webmail accounts or uploaded to external file hosting site

Email server /web proxy

Unusual physical access attempts (after hours, accessing unauthorized area, etc.) Physical badge records / Authentication

Excessive printer activity and employee is on an internal watch list as result of demotion / poor review / impending layoff

Printer logs / HR systems

User name of terminated employee accessing internal system Authentication / HR systems

IT Administrator performing an excessive amount of file deletions on critical servers or password resets on critical applications (rogue IT administrator) Operating system /Authentication / Asset DB

Employee not taking any vacation time or logging into critical systems while on vacation (concealing fraud) HR systems / Authentications

Long running sessions, bandwidth imbalance between client & server, Bad SSL Configurations IPS / IDS / Stream

Known cloud or malware domains, bad SSL Configurations Threat Intelligence, Custom Lookups

High Entropy Subdomains Web proxy, DNS, Wiredata

© 2017 SPLUNK INC.

Descriptive Statistics

© 2017 SPLUNK INC.

▶ In high school statistics you learned about mean, mode, median, min, max, & frequency aka “Descriptive Statistics”.

▶ Data Scientists make use of these terms to describe the data, explore potential relationships within the data, ask questionsof data and test various hypothesis.

▶ This iterative process is called “Exploratory Data Analysis” it is critical toMachine Learning and Security Analytics.

Descriptive StatisticsHow do you measure qualitative concepts?

18

© 2017 SPLUNK INC.

▶ Compare different duration times of data set for a specific time period.▶ index=suricata event_type=flow

| stats count as number_events, min(duration) as min_duration, max(duration) as max_duration, avg(duration) as avg_duration, median(duration) as median_duration, perc95(duration) asperc95_duration, stdev(duration) as stdev_duration

▶ Are there any long running sessions in the last 60 minutes?

▶New ways of describing the data:• Skew – Left, Right, Normal• Range – Lowest & Highest Values• Mode – Unimodal, Bimodal

Descriptive StatisticsA practical example, looking at flow duration times

19

© 2017 SPLUNK INC.

Describing network flows with Producer Consumer Ratio (PCR)1. Create a ratio of bytes_in to bytes_out2. Apply case logic to determine inbound or outbound imbalance between client & server▶ index=suricata event_type=flow

| eval bytes_total=bytes_in+bytes_out| eval bytes_ratio= ((bytes_out-bytes_in)/bytes_total) | eval bytes_pcr_range = case(bytes_ratio > 0.4, "Pure Push", bytes_ratio > 0, "70:30 Export", bytes_ratio == 0, "Balanced Exchange", bytes_ratio >= -0.5, "3:1 Import", bytes_ratio > -1, "Pure Pull") | stats sparkline(count) AS activity by src_ip src_port dest_ip dest_port bytes_in bytes_out bytes_pcr_range

Applying Descriptive Statistics - PCRUsing math to engineer features and describe entity behavior

20

PCR Ranges:1.0 – Pure Push - FTP upload, multicast, beaconing0.4 – 70:30 export - Sending Email0.0 – Balanced Exchange -NTP, ARP probe-0.5 – 3:1 import - HTTP Browsing-1.0 – pure pull - HTTP Download

© 2017 SPLUNK INC.

▶ Predict the next 3 days of traffic flow based on 5 weeks data with a 90% confidence interval, normalized around the average PCR ratio:

▶ index=suricata event_type=flow | `pcr(bytes_in,bytes_out)`| timechart span=10m avg(pcr_ratio) as avg_pcr_ratio | `forecast5w(avg_pcr_ratio,90.0,+1d,3)`

Forecasting PCR Crystal Ball forecasting for entity analysis

© 2017 SPLUNK INC.

▶ Store prediction into a summary index▶ Store actual metric in to the same summary index or compare in real time▶ Compare observation to prediction

• Alert when observation is outside the upper or lower bounds of the prediction

Forecasting PCR Compare prediction to observation for anomaly detection

22

© 2017 SPLUNK INC.

Exploratory Data Analysis

© 2017 SPLUNK INC.

▶ Visualization is a powerful EDA tool• Not everything can be described as bits, bytes, plaintext or pie charts.

▶ Correlation to add context to your data during the EDA process or test hypothesis.

Visualization & Creating Context (EDA)

24

© 2017 SPLUNK INC.

Geographical EDA - Visualization

25

• Visualization useful for exploring multi-dimensional relationships.• Tells a story about the data you can’t describe in text or tables.• “Where are connections ‘originating’, and how often am I seeing this activity?”

© 2017 SPLUNK INC.

▶DNS Water Torture• Botnet sends queries with 16 letters randomly prepended to the victim’s domain.

• xyuiasdfcosic.www.halpme.com• alkdfejenjasd.www.halpme.com

▶C&C Beaconing Activity (Dynamic DNS)• Advanced malware uses a Domain Generation Algorithm (Random Subdomain)

• d0290d00xasdf.no-ip[.]org▶Data Exfiltration

• DNS Tunneling (Query)• dnscat.912701a98e9bde415c4ad70007beaf54d2• dnscat.925401a98ebe0cf540b20d001a4b5e726494b001bb4c192bb68fe73c000bf7c1c0e

▶ Two Techniques to detect this activity in Suricata:• Shannon Entropy of DNS Query, HTTP destination• Character Length of DNS Query, HTTP destination

Practical Applications to DGA type activities

26

© 2017 SPLUNK INC.

Challenges to detect DGAs:▶ Static matching runs against

potentially infinite blacklist entries O(∞)

▶ Regex can narrow down this list, but still hard to compute and find rules (and define exceptions for rules)

▶ Unknown unknowns?▶ Want to get fuzzy?▶ Good use case for ML!

▶ Example of DGAs:

Example IoCs for Wannacry(https://cert.europa.eu/static/SecurityAdvisories/2017/CERT-EU-SA2017-012.pdf)

Domain Generating Algorithms (DGA)What’s the problem?

© 2017 SPLUNK INC.

▶What is Shannon Entropy?• “… a measure of uncertainty in a random variable”

▶How does it help us find malware and anomalousactivity?• The more random a string is, the higher its calculation of randomness.

• aaaaa.com (Score 1.8)• Google.com (Score 2.6)• Ic49f66b73141b5c1.com (Score 4.1)

• Domains and subdomains with high entropy are good indicators of malicious behavior.• We can filter to domains or subdomains with a score above 3 or 4.

▶ Cons:• False positives

• CDNs like Amazon, Akamai, and others use pseudorandom generated subdomains• Requires to you to keep a blacklist or whitelist of domains to reduce noise when hunting (but,

relatively easy to do in Splunk)• Malware evolves

• Locky & others using shorter subdomains or domains to reduce randomness, reducing entropy score

Shannon Entropy for DGA Hunting

28

© 2017 SPLUNK INC.

▶ Python Lookups - Entropy Analysis of DNS / HTTP ▶ # Full Query for Suricata HTTP▶ index=suricata host=suricata event_type=http

| lookup ut_parse_extended_lookup url AS dest| lookup ut_shannon_lookup word AS ut_subdomain OUTPUT ut_shannon AS ut_shannon_subdomain | lookup ut_shannon_lookup word AS dest OUTPUT ut_shannon AS ut_shannon_dest | search ut_shannon_dest > 4OR ut_shannon_subdomain > 4| table ut_subdomain ut_shannon_subdomain dest ut_shannon_dest | dedup dest ut_subdomain

Shannon Entropy for DGA Hunting# Results of Suricata HTTP Entropy Scoring

29

© 2017 SPLUNK INC.

Machine Learning & Security Analytics

DGA Domains & SSH Client activity

© 2017 SPLUNK INC.

Supervised • Classification (Nearest Neighbors, Support Vector Machines, Naïve Bayes, Decision

Tree)− Group “like” things together based on selected features.

• Regression (Linear & Logistic)− Infer a relationship between two variables (x) & result (y).

Unsupervised• Clustering (K Means)

− Partition events with multiple numeric fields into clusters.

• Decomposition(PCA, SVD)− Dimension Reduction, explains the maximum variance of the higher dimension

Machine Learning

31

© 2017 SPLUNK INC.Algorithms supported (v2.0, .conf2016)

© 2017 SPLUNK INC.

● A toolset for asking research questions which we want to operationalize.

● Problem: DGA domains are computer generated pseudo-random character strings, blacklisting an infinite number of domains is not feasible.

● Hypothesis: “Are there patterns in domain generation algos that can be leveraged to identify these as threats and predict new domains?”

Machine Learning

33

© 2017 SPLUNK INC.

© 2017 SPLUNK INC.

▶ More features can significantly improve your machine learning models• Be wary of overfitting!

▶ Extensible feature engineering ideas• (e.g. subdomains, age of domain registration, rating/scoring from threatlists for known

malicious domains etc.)

© 2017 SPLUNK INC.

© 2017 SPLUNK INC.

© 2017 SPLUNK INC.

© 2017 SPLUNK INC.

▶ Check how our trained model performs against WannaCry C&C domains that the model has NOT been trained on.

▶ Model predictions can be made actionable immediately with Splunk Alerts or turn into notable event frameworks like in ES

Reality check: Detect Unknown Unknowns?Example WannaCry

© 2017 SPLUNK INC.

▶ Operational:• Predicting certain types of SSH clients based on network features• Identifying Red Team activity • Botnet populations

▶ Overview• Label Suricata flow data with SSH event_type data• EDA to validate sessions are correctly assigned• Feature Engineering & Selection• Create model• Operationalize model

Machine Learning - Predicting SSH Client Can we predict the SSH Client based on the packet ratio?

© 2017 SPLUNK INC.

▶ | transaction flow_id

▶ Validate flow_id time span between first & last event

Labeling Suricata flow data & EDACorrelation of flowid

© 2017 SPLUNK INC.

▶ Producer Consumer Ratio• (bytes_out-bytes_in)/bytes_total• pcr_total_bytes

• Total number of bytes transmitted for each flow_id

• pcr_range• Categorical range the traffic falls into

(download, v. upload) for each flow_id• pcr_ratio

• Numerical value that describes the ratio between download & upload activity for flow_id

Feature Engineering & SelectionPCR

© 2017 SPLUNK INC.

▶ RandomForestClassifier is used to create the model

Model Selection & ValidationRandomForestClassifier & RandomizedLogisticRegression

© 2017 SPLUNK INC.

▶ Randomized Logistic Regression to validate feature selection

▶ | summary rand_log_regression_ssh_30d

▶ http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RandomizedLogisticRegression.html

Model Selection & ValidationRandomForestClassifier & RandomizedLogisticRegression

© 2017 SPLUNK INC.

OperationalizeApply the model to unlabeled flow data

© 2017 SPLUNK INC.

▶ Compare flow log + model which creates the field "predicted(ssh_client_software)” to the ssh event_type to grade model for accuracy

▶ Group & Cluster different ssh clients, develop whitelist/blacklist, integrate with threat intel to further label & feature engineer

Expanded Analysis & OperationalizationCompare predicted to observed ssh client

© 2017 SPLUNK INC.

Correlation – Finding Mirai

47

● Mirai Scanner.C module

• Technique• Default credentials hard-coded in the

Scanner.C module give us a behavioral signature to look for.

• Telnet/SSH attempts using invalid users (tech, mother, ubnt, 666666, 888888) are unique to Mirai, & other botnets (post source code leak).

• Correlate list of IPs with Suricata to find other activity from these IoT nodes attempting to breach my network.

© 2017 SPLUNK INC.

▶ Random Forest▶ Results

• High precision at predicting 0

• Small false positive (8/25,000)

• 10.6% False Negatives

• 89.4% Correct at getting Mirai Traffic correct

▶ Summary• IPS sensor allowed all of these connections (Not Blocked), while we missed 10.6%

• We now have a model which we can further refine to identify malicious SSH traffic to investigate.

• Adds a new layer to our security stack

Machine Learning- Predict Mirai

© 2017 SPLUNK INC.

▶ Create a new model using Random forest on labeled Mirai traffic • | inputlookup labeled_ssh_flows_suricata_30day_miraiV.csv

| head 50000| `pcr(bytes_in,bytes_out)`| fit RandomForestClassifier "isMiraiVariant" from "pcr_ratio" "pcr_total" "pcr_range" into "kmeans_rnd_mirai”

Labeled Mirai TrafficCreate a model using labeled Mirai traffic

© 2017 SPLUNK INC.

▶ Apply the model to our ssh data• | apply "pcr_rnd_mirai”• Use PCA & Kmeans to reduce features & cluster

▶ Proactively alert on new botnets derived from Mirai source code scanner module

Labeled Mirai TrafficCreate a model using labeled Mirai traffic

© 2017 SPLUNK INC.© 2017 SPLUNK INC.

Thank You

© 2017 SPLUNK INC.

Install ML Toolkit

Step 1: install ML Toolkit apphttps://splunkbase.splunk.com/app/2890

Step 2: install Python for Scientific Computing add-onhttps://splunkbase.splunk.com/app/2890/#/details

Step 3: restart Splunk

That’s it! Explore the ML Toolkit:Prediction, Outlier Detection, Forecasting, ClusteringShowcase examples for IT/Security/Business/IoT use casesAssistants: use your own data, build models, view in search

© 2017 SPLUNK INC.

▶ Philipp Drieger: DGA App & Content https://splunkbase.splunk.com/app/3559/• Conf17 Presentation Recording – http://conf.splunk.com/files/2017/recordings/automating-threat-hunting-

with-machine-learning.mp4▶ Mike Fisher: Building a crystal ball

• Conf16 Presentation – https://conf.splunk.com/files/2016/slides/building-a-crystal-ball-forecasting-future-values-for-multi-cyclic-time-series-metrics-in-splunk.pdf

▶ Macy Cronkrite: Anomaly Hunting with Splunk• Conf16 Presentation – https://conf.splunk.com/files/2016/slides/anomaly-hunting-with-splunk-software.pdf

▶ Xander Johnson & Zidong Yang: ML API• Conf17 Presentation – http://conf.splunk.com/files/2017/slides/advanced-machine-learning-using-the-

extensible-ml-api.pdf▶ Andrew Stein

• General ML advice & mentoring

ResourcesAKA who I owe credit to:

© 2017 SPLUNK INC.

▶ Spurious Correlations http://www.tylervigen.com/spurious-correlations▶ PCR – A New Flow Metric

http://qosient.com/argus/presentations/Argus.FloCon.2014.PCR.Presentation.pdf▶ Data Driven Security http://datadrivensecurity.info/▶ Splunk Syntax Highlighting http://blog.metasyn.pw/splunk-syntax-highlighting/▶ Doing Data Science http://shop.oreilly.com/product/0636920028529.do▶ Hunting the Known Unknowns (with DNS) https://conf.splunk.com/speakers/2015.html#search=Kovar&▶ Lookups, and other goodies https://github.com/anthonygtellez/conf2016_extras▶ IDS Evasion w TTL - http://insecure.org/stf/secnet_ids/secnet_ids.html▶ Applying Machine Learning to Network Security Monitoring http://www.mlsecproject.org/#conference-

presentations▶ Scikit-Learn http://scikit-learn.org/▶ Machine Learning Toolkit https://splunkbase.splunk.com/app/2890/▶ URL Toolbox https://splunkbase.splunk.com/app/2734/

References & Resources

54

© 2017 SPLUNK INC.

ML Talks @ .conf2017 https://conf.splunk.com/sessions/2017-sessions.html#search=Machine%20Learning

TELUS: Numeric Outlier Detection

AETNA: Be a Rock Star! Real-World Use Cases From Aetna with MLTK

BMW: The Next Level of Quality Assurance at BMW With the MLTK

ESE GmbH: Splunk ML Capabilities and Condition-Based Maintenance: Train Doors on

the German Public Rail Transport System

What's new in ML across products

Using the Splunk Machine Learning Toolkit to Create Your Own Custom Models

Building ML Solutions using MLTK

Advanced Machine Learning in SPL with the Machine Learning Toolkit

© 2017 SPLUNK INC.

Customer/Partner ML Talks @ .conf2016https://conf.splunk.com/sessions/2016-sessions.html#search=Machine%20Learning

OTTO & LC Systems: Anomaly Detection on Business Items with Machine Learning

Algorithms

DOCOMO: Predicting Incidents with Supervised and Unsupervised Machine Learning on

Splunk

TELUS: Building a Smarter Strategy for Alarms with Machine Learning!

Cisco: Infrastructure Analytics: Driving Outcomes Through Practical Use Cases & Applied Data

Science

Emerson: Demystifying Machine Learning and Anomaly Detection: Practical Applications in

Splunk for Insider Threat Detection and Security Analytics

Creative Artists Agency: The Practical Benefits of a Behavioral Solution for Enterprise

© 2017 SPLUNK INC.

Splunk ML Talks @ .conf2016 https://conf.splunk.com/sessions/2016-sessions.html#search=Machine%20Learning

Using the Splunk Machine Learning Toolkit to Create Your Own Custom Models

Splunk at a Telco: Assessing Outages and Improving Customer Experience with Machine

Learning

Tracking Trading (FIX) Environments with Splunk

Solve Big Problems with Machine Learning

Anomaly Hunting with Splunk Software

Splunk UBA – A Data Scientist in a Box

Advanced Machine Learning in SPL with the Machine Learning Toolkit

Machine Learning and Anomaly Detection in Splunk IT Service Intelligence