© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Practical examples of Big Data, security analytics and visualization Jeff McGee, Data Scientist Josh Stevens, Enterprise Security Architect
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 3
Objective
• Identify problems in Security that could be solved with better analytics
• Discuss recent efforts on Big Data and Visualization
• Share examples of how HP’s Cyber Defense Center has leveraged these capabilities
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Big Data and the data overload
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 5
Good guys are making things less predictable
Challenge: There is more noise
• Mobile • Bring your own device • Virtual machines and “the Cloud” • SAAS • New sources of logs • HIPPA, SOX, PCI…
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 6
And bad guys know how to stay inside the bell curve. Challenge: There is less signal
Unknown: Harder to detect
• New behavior • Goes to an approved place • Works encrypted • Authorized Use • Inside of baseline • Outside monitored infrastructure
• Matches a signature • Goes to a bad place • Works in the clear • Unauthorized Use • Outside of baseline • Within monitored infrastructure
Known: Easier to detect
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 7
Solutions to big data problems
Let’s take techniques originally built for other domains and apply them to security: • Map-reduce • Columnar Data Stores • Machine Learning • Visualization Tools
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 8
Tools and technologies
• Hadoop – Framework for distributed computing • Vertica – Columnar database • Tableau – Visualization software • Numpy/Scikit-learn – Machine learning tools
ArcSight
Vertica analytic platform
Hunt teams
Security intelligence
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 9
Overview: The Vertica analytic platform
Rapid iterative conversations with your data
Real time Analytics
Purpose built for Big Data from the first line of code
Store & Analyze PBs Ingest 30 TB/hour
Proven Scalability Works with Hadoop, R Ecosystem of Visualization Tools, SDKs and Community
Open & Extensible Efficient compressed storage Scale-out architecture Easy to setup & manage
Low TCO
Private Cloud Public Cloud Appliance Software Only
Flexible to deploy
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 10
Security visualization Practical examples
Router
Operating System/Informational
Network-basedIDS/IPS
/Suspicious
Network-basedIDS/IPS/Recon
Network-based IDS/IPS/Informational
Network-basedIDS/IPS
/Compromise
Network Monitoring/Informational
Network Monitoring
Firewall
Firewall/Normal
Firewall
Firewall
Content Security
Applications/InformationalApplications
bust6Category Device Type
ApplicationsContent Security
Firewall
Host-based IDS/IPS
Mainframe
Network Monitoring
Network-based IDS/IPS
Operating SystemRouter
VPN
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Security management
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 12
Category Significance
Informational/Error /Hostile /Compromise /Suspicious /Normal /Informational/Warning /Recon /Informational0M
10M
20M
30M
40M
50M
60M
70M
80M
90M
100M
Co
Network-based IDS/IPS
Network-based IDS/IPS Network-based IDS/IPS
Network-based IDS/IPS
Network-based IDS/IPS
Host-based IDS/IPS
Network Monitoring
Operating System
Applications
Applications
Applications
Firewall
Firewall
Firewall
VPN
fromAfarSourcePtCount of Destination Port
0
20,000,000
40,000,000
66,854,010
0 66,854,010
Count of Destination Port
View at a glance
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 13
Router
Operating System/Informational
Network-basedIDS/IPS
/Suspicious
Network-basedIDS/IPS/Recon
Network-based IDS/IPS/Informational
Network-basedIDS/IPS
/Compromise
Network Monitoring/Informational
Network Monitoring
Firewall
Firewall/Normal
Firewall
Firewall
Content Security
Applications/InformationalApplications
bust6Category Device Type
ApplicationsContent Security
Firewall
Host-based IDS/IPS
Mainframe
Network Monitoring
Network-based IDS/IPS
Operating SystemRouter
VPN
Proportional relationships
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Security analysts
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 15
Category Device Type
Applications ContentSecurity
Database Firewall Host-basedIDS/IPS
Mainframe NetworkMonitoring
Network-based ID..
OperatingSystem
Policy Management
Router SecurityMangement
VPN
0M
100M
200M
300M
400M
500M
600M
700M
Cou
DeviceSeveritybyDevice
Count of Device Severity for each Category Device Type. The view is filtered on Category Device Type, which keeps 13 of 20 members.
Starting points
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 16
Category Outcome / Category Significance/Attempt /Failure /Success
/Comp /Informational
/Informational/Error
/Normal /Recon
/Suspicious
Null
/Compromise /Informational
/Informational/Alert /Informational/Error/Informational/Warning
Null
/Compromise /Informational
/Informational/Alert /Informational/Error/Informational/Warning
/Normal /Recon
/Suspicious
0M
50M
100M
150M
200M
250M
300M
350M
Cou
destHostnameAttemptFailSuccess
Count of Destination Host Name for each Category Significance broken down by Category Outcome. The data is filtered on Destination Host Name, which excludes Null. The view is filtered on Exclusions (Category Outcome,Category Significance) and Category Outcome. The Exclusions (Category Outcome,Category Significance) filter keeps 35 members. TheCategory Outcome filter excludes Failure.
Carving in
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 17
Trending attempts
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 18
Category Outcome / Category Significance/Attempt /Failure /Success
/Comp /Informational
/Informational/Error
/Normal /Recon
/Suspicious
Null
/Compromise /Informational
/Informational/Alert /Informational/Error/Informational/Warning
Null
/Compromise /Informational
/Informational/Alert /Informational/Error/Informational/Warning
/Normal /Recon
/Suspicious
0M
50M
100M
150M
200M
250M
300M
350M
Cou
destHostnameAttemptFailSuccess
Count of Destination Host Name for each Category Significance broken down by Category Outcome. The data is filtered on Destination Host Name, which excludes Null. The view is filtered on Exclusions (Category Outcome,Category Significance) and Category Outcome. The Exclusions (Category Outcome,Category Significance) filter keeps 35 members. TheCategory Outcome filter excludes Failure.
Successes
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 19
Trending success by hostname
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 20
Actual hostnames
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 21
Keep in mind this is demo data – however a quick internet search shows this domain has a reputation as a bullet proof server, delivering malware. Our visualization shows us it’s been accessed every day for the last 30 days.
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 22
Bullet proof servers
White spac e
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Hunt teams Use case 1
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 24
Device Receipt Time
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59
Cat Network
0K
10..
20..
30..
40..
50..
60..
70..
80..
90..
count of device severity 30 dall all scatter Device Severity
High
Medium
Unknown
Very-High
60 days IPS data
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 25
Jun 3 Jun 8 Jun 13 Jun 18 Jun 23 Jun 28 Jul 3 Jul 8 Jul 13 Jul 18Hour of Device Receipt Time [2014]
Cat Network
0K
20K
40K
60K
80K
100K
120K
140K
Co
IPS EventsDevice Severity
High
Medium
Unknown
Very-High
30 days
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 26
By technique
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 27
fromVictim
Jun 7 Jun 12 Jun 17 Jun 22 Jun 27 Jul 2 Jul 7 Jul 12Minute of Device Receipt Time [2014]
Network
0
100
200
300
400
500
600
Cou
tenacleCategory Technique
/Exploit/Vulnerability
/Policy/Breach/Traffic Anomaly/Network Layer
/Traffic Anomaly/Network Layer/Flow
/Traffic Anomaly/Network Layer/IP Fragments
Aggregate from victim a
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Hunt teams Use case 2
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 29
Sonar
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 30
Source addresses
Dest
inat
ions
Sonar trend
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Hunt teams Use case 3
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 32
VPN
Network-based IDS/IPS
Network Monitoring
Host-based IDS/IPS
FirewallApplications
fromAfar2infoOnlyCategory Device Type
ApplicationsContent Security
Database
Firewall
Host-based IDS/IPS
Mainframe
Network Monitoring
Network-based IDS/IPS
Operating System
Policy Management
Security MangementVPN
Bottom of the stack “Informational”
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 33
VPN logging
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 34
Who’s scanning via VPN?
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 35
For more information
After the event
• Contact your sales rep • Visit the HP Security Product Blog:
hp.com/go/securityproductsblog
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 36
Please fill out a survey. Hand it to the door monitor on your way out.
Thank you for providing your feedback, which helps us enhance content for future events.
Session TB3273 Speaker Joshua Stevens, Jeff McGee
Please give me your feedback
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you