39
Fix What Matters: Why CVSS Sucks And How To Do Better

Fix What Matters: BSidesDetroit 2014

Embed Size (px)

DESCRIPTION

Heartbleed has exposed a weakness in the way we assess risk in information security. We use archaic methods and ignore new data when assessing what to fix, and we rarely go back to see what new data is telling us. In this talk, we explore new, data-driven approaches to vulnerability management.

Citation preview

Page 1: Fix What Matters: BSidesDetroit 2014

Fix What Matters: !

Why CVSS Sucks And How To

Do Better

Page 2: Fix What Matters: BSidesDetroit 2014

Once Jailbroke an Iphone 3G

Michael Roytman

Proud Owner of Remote Controlled AirplaneRecently a Naive Grad Student

Data Scientist, Risk I/ODoes Not Wake Up Before 11 CST

qualifications:

Page 3: Fix What Matters: BSidesDetroit 2014

15x better than CVSS

Page 4: Fix What Matters: BSidesDetroit 2014

Probability A Vuln Having Property X Has Observed Breaches

Random Vuln

CVSS 10

Exploit DB

Metasploit

MSP+EDB

0.0 0.1 0.2 0.2 0.3

Page 5: Fix What Matters: BSidesDetroit 2014

PART 1: !

YOU SUCK AT YOUR JOB

!

(and don’t even know it yet)

Page 6: Fix What Matters: BSidesDetroit 2014

Why Are We Here?

Empirical Failures of CVSSProper Remediation Frameworks

CVSS SUCKS

Analytical Failures of CVSS

(+Data Driven Alternatives)

Page 7: Fix What Matters: BSidesDetroit 2014

Remove the Threat

RemediationAccept the Risk

Repair the Vulnerability

Page 8: Fix What Matters: BSidesDetroit 2014

C(ommon) V(ulnerability) S(coring) S(ystem)

“CVSS is designed to rank information system vulnerabilities”

Exploitability/Temporal (Likelihood)

Impact/Environmental (Severity)

The Good: Open, Standardized Scores

Page 9: Fix What Matters: BSidesDetroit 2014

“It is a capital mistake to theorize before one has data.

!

!

!

Insensibly, one begins to twist facts to suit theories, instead of

theories to suit facts.”

Page 10: Fix What Matters: BSidesDetroit 2014

FAIL 1: A Priori Modeling“Following up my previous email, I have tweaked my equation to try to achieve better separation between adjacent scores and to have CCC have a perfect (storm) 10 score...There is probably a way to optimize the problem numerically, but doing trial and error gives one plausible set of parameters...except that the scores of 9.21 and 9.54 are still too close together. I can adjust x.3 and x.7 to get a better separation . . .”

Page 11: Fix What Matters: BSidesDetroit 2014

F2: Data FundamentalismDon’t Ignore What a Vulnerability Is: Creation Bias http://blog.risk.io/2013/04/data-fundamentalism/ !

Jerico/Sushidude @ BlackHat https://www.blackhat.com/us-13/briefings.html#Martin

!

Luca Allodi - CVSS DDOS http://disi.unitn.it/~allodi/allodi-12-badgers.pdf

Page 12: Fix What Matters: BSidesDetroit 2014

F2: Data FundamentalismSince 2006 Vulnerabilities have declined by 26 percent.” http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf !

!

The total number of vulnerabilities in 2013 is up 16 percent so far when compared to what we saw in the same time period in 2012. ” http://www.symantec.com/content/en/us/enterprise/other_resources/b-intelligence_report_06-2013.en-us.pdf

Page 13: Fix What Matters: BSidesDetroit 2014

F3: Stochastic Ignorance

Attackers Change Tactics Daily

Page 14: Fix What Matters: BSidesDetroit 2014

F3: Stochastic Ignorance

Page 15: Fix What Matters: BSidesDetroit 2014

Empirical Failures of CVSSObjective: Remediate the riskiest vulnerabilities

Constraint: Can’t measure impact/priority

Need:

MOAR DATA!!!

Page 16: Fix What Matters: BSidesDetroit 2014

Repair the Vulnerability

Page 17: Fix What Matters: BSidesDetroit 2014

I Love It When You Call Me Big Data50,000,000 Live Vulnerabilities

1,500,000 Assets

2,000 Organizations

Page 18: Fix What Matters: BSidesDetroit 2014

I Love It When You Call Me Big Data

3,000,000 Breaches

Page 19: Fix What Matters: BSidesDetroit 2014

Baseline AllthethingsProbability (You Will Be Breached On A Particular Open Vulnerability)?

=(Open Vulnerabilities | Breaches Occurred On Their CVE) /(Total Open Vulnerabilities)

2%

Page 20: Fix What Matters: BSidesDetroit 2014

Probability A Vuln Having Property X Has Observed Breaches

RANDOM VULN

CVSS 10

CVSS 9

CVSS 8

CVSS 6

CVSS 7

CVSS 5

CVSS 4

Has Patch

0.000 0.010 0.020 0.030 0.040

Page 21: Fix What Matters: BSidesDetroit 2014

PART 2: !

FIX WHAT MATTERS

Page 22: Fix What Matters: BSidesDetroit 2014

Empirical Failures of CVSSObjective: Remediate the riskiest vulnerabilities

Constraint: Can’t measure impact/priority

Need:

MOAR DATA!!!

Page 23: Fix What Matters: BSidesDetroit 2014

Proper Framework

Know which vulnerabilities put you most at risk.

Page 24: Fix What Matters: BSidesDetroit 2014
Page 25: Fix What Matters: BSidesDetroit 2014
Page 26: Fix What Matters: BSidesDetroit 2014
Page 27: Fix What Matters: BSidesDetroit 2014
Page 28: Fix What Matters: BSidesDetroit 2014
Page 29: Fix What Matters: BSidesDetroit 2014
Page 30: Fix What Matters: BSidesDetroit 2014
Page 31: Fix What Matters: BSidesDetroit 2014

Uh, Sports?

Opposing Teams, Specific Players

Gameplay

Scouting Reports, Gametape

Roster, Player Skills

Learning from Losing

Page 32: Fix What Matters: BSidesDetroit 2014

InfoSec?

Page 33: Fix What Matters: BSidesDetroit 2014

Defend Like You’ve Done It Before

Groups, Motivations

Exploits

Vulnerability Definitions

Asset Topology, Actual Vulns on System

Learning from Breaches

Page 34: Fix What Matters: BSidesDetroit 2014

Work With What You’ve Got:

Akamai, Safenet

ExploitDB, Metasploit

NVD, MITRE

Page 35: Fix What Matters: BSidesDetroit 2014

Alternatives

Page 36: Fix What Matters: BSidesDetroit 2014

Probability A Vuln Having Property X Has Observed Breaches

Random Vuln

CVSS 10

Exploit DB

Metasploit

MSP+EDB

0.0 0.1 0.2 0.2 0.3

Page 37: Fix What Matters: BSidesDetroit 2014

Be Better Than The Gap

Page 38: Fix What Matters: BSidesDetroit 2014

I Love It When You Call Me Big Data

!

Spray and Pray => 2% !

CVSS 10 => 4% !

Metasploit + ExploitDB => 30%

Page 39: Fix What Matters: BSidesDetroit 2014

Holler!www.risk.io@mroytman