Secure360 on Risk

Challenging Conventional Wisdom: A New Approach to Risk ManagementAlex HuttonJay Jacobs

What’s this about?

We think you’re getting bad information!

We think our industry can do better!

We think this will make us “more secure!”

Security is now so essential a concern that we can no longer use adjectives and adverbs but must instead use numbers. – Dan Geer

How are you making decisions now?

What’s the quality of those decisions?

Effective Decisions need quality data, models, execution

Our vendors and standards aren’t

helping us:-(

hey, why are you getting lousy information from standards and vendors?

hey, why are you getting lousy information from standards and vendors?

The science of information security & risk management is hard

1. Pseudo Science & Proto Science

2. Models & Data

3. Complexity

hey, why are you getting lousy information from standards and vendors?

The science of information security & risk management is hard

1. Pseudo Science & Proto Science

2. Models & Data

3. Complexity

State of the Industry (a)(Thomas Kuhn is way smarter than we are)

proto-science

somewhat random fact gathering (mainly of readily accessible data)

a“morass”of interesting, trivial, irrelevant observations

a variety of theories (that are spawned from what he calls philosophical speculation) that provide little guidance to data gathering

State of the Industry (b)At our present skill in measurement of security, we generally have an ordinal scale at best, not an interval scale and certainly not a ratio scale. In plain terms, this means we can say whether X is better than Y but how much better and compared to what is not so easy. – More from Dan Geer

If Science is based on inductive observations to derive meaning and understanding and measurement on quality (ratio) scales, how about InfoSec?

Where do we sit in the family of sciences?

We’re the Crazy Uncle with tinfoil hat antennae used to talk to the space aliens of Regulus V, has 47 cats, and who too frequently (but benignly) forgets to wear pants.

Take, for example, CVSS

“the Base Equation multiplies Impact by 0.6 and Exploitability by 0.4”

= ShinyJet Engine X Peanut Butter

adding onewilly-nilly doesn’t suddenly transformordinal rankings into ratio values.

decimals aren’t magic.

hey, why are you getting lousy information from standards and vendors?

The science of information security & risk management is hard

1. Pseudo Science & Proto Science

2. Models & Data

3. Complexity

20

Data must exist in order to feed our models...

... but creating the right models are dependent on understanding what data is useful!

Data, Models, Execution: Garbage in-Garbage Out

Data, Models, Execution: Treat Data Poorly

Data, Models, Execution: Adapting to Situations

hey, why are you getting lousy information from standards and vendors?

The science of information security & risk management is hard

1. Pseudo Science & Proto Science

2. Models & Data

3. Complexity

These “risk” statements you’re making...

I don’t think you’re doing it right.

- (Chillin’ Friederich Hayek)

“Given Newton's laws and the current position

and velocity of every particle in the universe,

it was possible, in principle, to predict

everything for all time.”

-- Simon-Pierre LaPlace, 1814

A Comforting Thought...

8

4 4

2 2 2 2

Reductionism

8

4 4

2 2 2 2

Functionalism

?

?

Asset

Comp. Comp.

Sub. Sub.

Attribute

Attribute

Attribute

Attribute

Reductionism

Functionalism

Awww man......even if it were the case that the natural laws had no longer any secret for us, we could still only know the initial situation approximately. ... small differences in the initial conditions produce very great ones in the final phenomenon. A small error in the former will produce an enormous error in the latter. Prediction becomes impossible...

-- Henri Poincare, 1887

13

5 6

2 2 2 2

Holism

Complexity non-linear

Systems Approach

Complex systems contain changing mixtures of failures latent within them.

The complexity of these systems makes it impossible for them to run without multiple flaws being present.

... individually insufficient to cause failure

...failures change constantly because of changing technology, work organization, and efforts to eradicate failures.

Complex systems run in degraded mode.

“How Complex Systems Fail” - Richard Cook

Security is a characteristic of systems and not of their components

Security is an emergent property of systems; it does not reside in a person, device or department of an organization or system.

... it is not a feature that is separate from the other components of the system.

...the state of Security in any system isalways dynamic

“How Complex Systems Fail” - Richard Cook

We may want to rethink our approach.

36

Overcoming the problem

• Medicine uses an “Evidence-Based” approach to solving problems in the complex system that is the body.

• Dr. Peter Tippett (MD, PhD) applies Evidence-Based principles to Information Security.

threat landscape

asset landscape

impact landscape

controls landscape

risk

Suggested  context:Capability  to  manage(skills,  resources,  decision  quality…)

What to study: Sources of Knowledge

How: Data Quality in Evidence-Based Practice

Evidence  level  D Evidence  level  C Evidence  level  B Evidence  level  A

Evidence  level  A“Expert  opinion  without  explicit  cri8cal  appraisal,  or  based  on  physiology,  bench  research  or  first  principles.”

Case-­‐series  study  or  extrapola8ons  from  level  B  studies.

Consistent  Retrospec8ve  Cohort,  Exploratory  Cohort,  Ecological  Study,  Outcomes  Research,  case-­‐control  study;  or  extrapola8ons  from  level  A  studies.

Consistent  Randomized  Controlled  Clinical  Trial,  cohort  study,  all  or  none,  clinical  decision  rule  validated  in  different  popula8ons.

beNer

Evidence-Based Risk ManagementState of Nature State of Knowledge State of Wisdom

Evidence level D Lists Feeling like we’ve done something

Evidence level C Simple derived values with ad-hoc modeling

Outcomes with ad-hoc deductive selections

Evidence level B Formal Modeling Decision making constructs

Evidence level A

State of Nature State of Knowledge State of Wisdom

Evidence level D Lists Feeling like we’ve done something

Evidence level C Simple derived values with ad-hoc modeling

Outcomes with ad-hoc deductive selections

Evidence level B Formal Modeling Decision making constructs

Evidence level A

Evidence-Based Risk Management

State of Nature State of Knowledge State of Wisdom

Evidence level D Lists Feeling like we’ve done something

Evidence level C Simple derived values with ad-hoc modeling

Outcomes with ad-hoc deductive selections

Evidence level B Formal Modeling Decision making constructs

Evidence level A

You  are  here

Evidence-Based Risk Management

So  How  Do  We  Change?

DataModels…

Standards

START  WITH  THE  

OUTCOMES!

Two True Security Outcomes:

Success and Failure

Knowing Success in InfoSec is hard

- Known Success (anti-Threat ops)- Unknown success (controls work

without us knowing)- Dumb luck (We’re not targeted, but our

neighbor is)

Getting the outcomes:Success

Getting the outcomes:Success

stronger processes result in fewer availability incidents

Getting the outcomes- Successes:

- Existences of processes- Operational (performance) metrics- Maturity ratings

WHAT WE WANT ARE PATTERNS!

Knowing Failure is (somewhat) easier

Getting The Outcomes: Failures

VERIS | Verizon Enterprise Risk and Information Sharing

VERIS takes the incident narrative and creates metrics (risk determinants)

A  free  (as  in  beer*)  framework  created  for  metrics,  modeling,  and  compara8ve  analy8cs.

A  security  incident  (or  threat  scenario)  is  modeled  as  a  series  of  events.  Every  event  is  comprised  of  the  following  4  A’s:

Agent:  Whose  acLons  affected  the  asset

AcLon:  What  acLons  affected  the  asset

Asset:  Which  assets  were  affected  

AOribute:  How  the  asset  was  affected

VERIS | Verizon Enterprise Risk and Information Sharing

INCIDENT REPORT“An attacker from a Russian IP address

initiated multiple SQL injection attacks against a public-facing web application. They were able to introduce keyloggers and network sniffers onto internal systems. The keyloggers captured several domain credentials which the attackers used to further infiltrate the corporate network. The packet sniffers captured data for several months which the attacker periodically returned to collect…”

VERIS takes this :

and…

…and translates it to this…Event 1Agent: External (Org crime)Action: Hacking (SQLi)Asset: Server (Web server, Database)Attribute: IntegrityEvent 2Agent: External (Org crime)Action: Malware (Keylogger)Asset: Server (Web server)Attribute: ConfidentialityEvent 3Agent: External (Org crime)Action: Hacking (Use of stolen creds)Asset: Server, Network (multiple)Attribute: Confidentiality, IntegrityEvent 4…

1 2 3 4> > > >

patterns!

√∫∑

Framework

Models Data=

∩

Framework

Data

√∫∑Models=

∩

Framework

Data

Data √∫∑Models=

∩Process

ProcessProcess

Process

Using your metrics program

- Identify & Measure your processes- Identify & Measure your failures- Get into loss factors (ABC)- Share data- Support data sharing efforts

Bring it Home: your metrics program

Bring it Home: your metrics programor

Bring it Home: your metrics programorThe Amazing Technicolor Scorecard

Priority #1: no more surrogate data

Priority #1: (meaning) no more risk analysts*

Priority #1: (really) create data analysts

Data analysts need to focus on quality data, models, execution

State of Nature State of Knowledge State of Wisdom

Evidence level D Lists Feeling like we’ve done something

Evidence level C Simple derived values with ad-hoc modeling

Outcomes with ad-hoc deductive selections

Evidence level B Formal Modeling Decision making constructs

Evidence level A

Evidence-Based Risk Management

threat landscape

risk

A balanced scorecard of sorts

asset landscape

impact landscape

controls landscape

Where to look? The Two True Security Outcomes:

Success and Failure

Failures:threat landscape

asset landscape

impact landscape

controls landscape

incidents, red/blue team

vulnerabilities, misconfigurations, unknowns...

gaps in coverage, known lack of effectiveness, known underskilled/utilized...

Cost-Based Accounting around incidents, cost of operations, etc...

Successes:threat landscape

asset landscape

impact landscape

controls landscape

intel, red/blue teams, SIEM

vulnerabilities, misconfigurations, unknowns, skills, training

positive threat outcomes (tOps), skills, training

ROI? ROSI? (ducks to avoid tomatoes)

What to look? Two types of data to find:

Focus initially on Visibility, then look to find Variability.

How to look? The GQM Approach:

For each “where” for each “what” use the following “how”

How to look? The GQM Approach:

For each “where” for each “what”, start by using GQM as “how.”

Goal, Question, Metric

Conceptual level (goal) goals defined for an object for a variety of reasons, with respect to various models, from various points of view.

Operational level (question)

questions are used to define models of the object of study and then focuses on that object to characterize the assessment or achievement of a specific goal.

Quantitative level (metric)

metrics, based on the models, is associated with every question in order to answer it in a measurable way.

Victor Basili

The Book You Should Buy(Jay & Alex aren’t getting a kickback, in case you’re wondering)

GQM for Fun & Profit

Goals establishwhat we want to accomplish.

Questions help us understand how to meet the goal. They address context.

Metrics identify the measurements that are needed to answer the questions.

Goal 1 Goal 2

Q1 Q2 Q3 Q4 Q5

M1 M2 M3 M4 M5 M6 M7

GQM for Fun & Profit

Execution

Models

Data

Goal 1 Goal 2

Q1 Q2 Q3 Q4 Q5

M1 M2 M3 M4 M5 M6 M7

data about defined success and failures

models of assets, controls, threats contributing to impact

execution by data analysts...Feeding standards, audits and governance

Using your metrics program

- Identify & Measure your processes- Identify & Measure your failures- Get into loss factors (ABC)- Share data- Support data sharing efforts

Using your metrics program

- Identify & Measure your processes- Identify & Measure your failures- Get into loss factors (ABC)- Share data- Support data sharing efforts

Security is now so essential a concern that we can no longer use adjectives and adverbs but must instead use numbers. – Dan Geer

Questions?Jay Jacobs@jayjacobsjay@beechplane.com

Alex Hutton@alexhuttonalex@alexhutton.com

threat landscape

asset landscape

impact landscape

controls landscape

risk

Prioritize

De-prioritize

Approaching the system as a system

threat landscape

asset landscape

impact landscape

controls landscape

risk

Suggested context:Capability to manage(skills, resources, decision quality…)

Data Sharing:

- Sources:- Qualify this Intel according to

framework- Treat with appropriate data quality

listings (let models shape the certainty)

Get Into Accounting

- Use existing models that take advantage of accounting concepts (ABC) to Talk to the LOBs

Using your metrics program

- Identify & Measure your processes- Identify & Measure your failures- Share data- Support data sharing efforts- Get into loss factors (ABC)

Challenging Conventional Wisdom

Conventional Wisdom may not be wrong- Question current practices - Seek Evidence and Feedback