Download pdf - Secure360 on Risk

Challenging Conventional Wisdom: A New Approach to Risk ManagementAlex HuttonJay Jacobs

What’s this about?

We think you’re getting bad information!

We think our industry can do better!

We think this will make us “more secure!”

Security is now so essential a concern that we can no longer use adjectives and adverbs but must instead use numbers. – Dan Geer

How are you making decisions now?

What’s the quality of those decisions?

Effective Decisions need quality data, models, execution

Our vendors and standards aren’t

helping us:-(

hey, why are you getting lousy information from standards and vendors?


The science of information security & risk management is hard

1. Pseudo Science & Proto Science

2. Models & Data

3. Complexity




2. Models & Data

3. Complexity

State of the Industry (a)(Thomas Kuhn is way smarter than we are)

proto-science

somewhat random fact gathering (mainly of readily accessible data)

a“morass”of interesting, trivial, irrelevant observations

a variety of theories (that are spawned from what he calls philosophical speculation) that provide little guidance to data gathering

State of the Industry (b)At our present skill in measurement of security, we generally have an ordinal scale at best, not an interval scale and certainly not a ratio scale. In plain terms, this means we can say whether X is better than Y but how much better and compared to what is not so easy. – More from Dan Geer

If Science is based on inductive observations to derive meaning and understanding and measurement on quality (ratio) scales, how about InfoSec?

Where do we sit in the family of sciences?

We’re the Crazy Uncle with tinfoil hat antennae used to talk to the space aliens of Regulus V, has 47 cats, and who too frequently (but benignly) forgets to wear pants.

Take, for example, CVSS

“the Base Equation multiplies Impact by 0.6 and Exploitability by 0.4”

= ShinyJet Engine X Peanut Butter

adding onewilly-nilly doesn’t suddenly transformordinal rankings into ratio values.

decimals aren’t magic.




2. Models & Data

3. Complexity

20

Data must exist in order to feed our models...

... but creating the right models are dependent on understanding what data is useful!

Data, Models, Execution: Garbage in-Garbage Out

Data, Models, Execution: Treat Data Poorly

Data, Models, Execution: Adapting to Situations




2. Models & Data

3. Complexity

These “risk” statements you’re making...

I don’t think you’re doing it right.

- (Chillin’ Friederich Hayek)

“Given Newton's laws and the current position

and velocity of every particle in the universe,

it was possible, in principle, to predict

everything for all time.”

-- Simon-Pierre LaPlace, 1814

A Comforting Thought...

8

4 4

2 2 2 2

Reductionism

8

4 4

2 2 2 2

Functionalism

?

?

Asset

Comp. Comp.

Sub. Sub.

Attribute

Attribute

Attribute

Attribute

Reductionism

Functionalism

Awww man......even if it were the case that the natural laws had no longer any secret for us, we could still only know the initial situation approximately. ... small differences in the initial conditions produce very great ones in the final phenomenon. A small error in the former will produce an enormous error in the latter. Prediction becomes impossible...

-- Henri Poincare, 1887

13

5 6

2 2 2 2

Holism

Complexity non-linear

Systems Approach

Complex systems contain changing mixtures of failures latent within them.

The complexity of these systems makes it impossible for them to run without multiple flaws being present.

... individually insufficient to cause failure

...failures change constantly because of changing technology, work organization, and efforts to eradicate failures.

Complex systems run in degraded mode.

“How Complex Systems Fail” - Richard Cook

Security is a characteristic of systems and not of their components

Security is an emergent property of systems; it does not reside in a person, device or department of an organization or system.

... it is not a feature that is separate from the other components of the system.

...the state of Security in any system isalways dynamic

“How Complex Systems Fail” - Richard Cook

We may want to rethink our approach.

36

Overcoming the problem

• Medicine uses an “Evidence-Based” approach to solving problems in the complex system that is the body.

• Dr. Peter Tippett (MD, PhD) applies Evidence-Based principles to Information Security.

threat landscape

asset landscape

impact landscape

controls landscape

risk

Suggested context:Capability to manage(skills, resources, decision quality…)

What to study: Sources of Knowledge

How: Data Quality in Evidence-Based Practice

Evidence level D Evidence level C Evidence level B Evidence level A

Evidence level A“Expert opinion without explicit cri8cal appraisal, or based on physiology, bench research or first principles.”

Case-‐series study or extrapola8ons from level B studies.

Consistent Retrospec8ve Cohort, Exploratory Cohort, Ecological Study, Outcomes Research, case-‐control study; or extrapola8ons from level A studies.

Consistent Randomized Controlled Clinical Trial, cohort study, all or none, clinical decision rule validated in different popula8ons.

beNer

Evidence-Based Risk ManagementState of Nature State of Knowledge State of Wisdom

Evidence level D Lists Feeling like we’ve done something

Evidence level C Simple derived values with ad-hoc modeling

Outcomes with ad-hoc deductive selections

Evidence level B Formal Modeling Decision making constructs

Evidence level A

State of Nature State of Knowledge State of Wisdom





Evidence level A

Evidence-Based Risk Management






Evidence level A

You are here


So How Do We Change?

DataModels…

Standards

START WITH THE

OUTCOMES!

Two True Security Outcomes:

Success and Failure

Knowing Success in InfoSec is hard

- Known Success (anti-Threat ops)- Unknown success (controls work

without us knowing)- Dumb luck (We’re not targeted, but our

neighbor is)

Getting the outcomes:Success

Getting the outcomes:Success

stronger processes result in fewer availability incidents

Getting the outcomes- Successes:

- Existences of processes- Operational (performance) metrics- Maturity ratings

WHAT WE WANT ARE PATTERNS!

Knowing Failure is (somewhat) easier

Getting The Outcomes: Failures

VERIS | Verizon Enterprise Risk and Information Sharing

VERIS takes the incident narrative and creates metrics (risk determinants)

A free (as in beer*) framework created for metrics, modeling, and compara8ve analy8cs.

A security incident (or threat scenario) is modeled as a series of events. Every event is comprised of the following 4 A’s:

Agent: Whose acLons affected the asset

AcLon: What acLons affected the asset

Asset: Which assets were affected

AOribute: How the asset was affected

VERIS | Verizon Enterprise Risk and Information Sharing

INCIDENT REPORT“An attacker from a Russian IP address

initiated multiple SQL injection attacks against a public-facing web application. They were able to introduce keyloggers and network sniffers onto internal systems. The keyloggers captured several domain credentials which the attackers used to further infiltrate the corporate network. The packet sniffers captured data for several months which the attacker periodically returned to collect…”

VERIS takes this :

and…

…and translates it to this…Event 1Agent: External (Org crime)Action: Hacking (SQLi)Asset: Server (Web server, Database)Attribute: IntegrityEvent 2Agent: External (Org crime)Action: Malware (Keylogger)Asset: Server (Web server)Attribute: ConfidentialityEvent 3Agent: External (Org crime)Action: Hacking (Use of stolen creds)Asset: Server, Network (multiple)Attribute: Confidentiality, IntegrityEvent 4…

1 2 3 4> > > >

patterns!

√∫∑

Framework

Models Data=

∩

Framework

Data

√∫∑Models=

∩

Framework

Data

Data √∫∑Models=

∩Process

ProcessProcess

Process

Using your metrics program

- Identify & Measure your processes- Identify & Measure your failures- Get into loss factors (ABC)- Share data- Support data sharing efforts

Bring it Home: your metrics program

Bring it Home: your metrics programor

Bring it Home: your metrics programorThe Amazing Technicolor Scorecard

Priority #1: no more surrogate data

Priority #1: (meaning) no more risk analysts*

Priority #1: (really) create data analysts

Data analysts need to focus on quality data, models, execution






Evidence level A


threat landscape

risk

A balanced scorecard of sorts

asset landscape

impact landscape

controls landscape

Where to look? The Two True Security Outcomes:

Success and Failure

Failures:threat landscape

asset landscape

impact landscape

controls landscape

incidents, red/blue team

vulnerabilities, misconfigurations, unknowns...

gaps in coverage, known lack of effectiveness, known underskilled/utilized...

Cost-Based Accounting around incidents, cost of operations, etc...

Successes:threat landscape

asset landscape

impact landscape

controls landscape

intel, red/blue teams, SIEM

vulnerabilities, misconfigurations, unknowns, skills, training

positive threat outcomes (tOps), skills, training

ROI? ROSI? (ducks to avoid tomatoes)

What to look? Two types of data to find:

Focus initially on Visibility, then look to find Variability.

How to look? The GQM Approach:

For each “where” for each “what” use the following “how”

How to look? The GQM Approach:

For each “where” for each “what”, start by using GQM as “how.”

Goal, Question, Metric

Conceptual level (goal) goals defined for an object for a variety of reasons, with respect to various models, from various points of view.

Operational level (question)

questions are used to define models of the object of study and then focuses on that object to characterize the assessment or achievement of a specific goal.

Quantitative level (metric)

metrics, based on the models, is associated with every question in order to answer it in a measurable way.

Victor Basili

The Book You Should Buy(Jay & Alex aren’t getting a kickback, in case you’re wondering)

GQM for Fun & Profit

Goals establishwhat we want to accomplish.

Questions help us understand how to meet the goal. They address context.

Metrics identify the measurements that are needed to answer the questions.

Goal 1 Goal 2

Q1 Q2 Q3 Q4 Q5

M1 M2 M3 M4 M5 M6 M7

GQM for Fun & Profit

Execution

Models

Data

Goal 1 Goal 2

Q1 Q2 Q3 Q4 Q5

M1 M2 M3 M4 M5 M6 M7

data about defined success and failures

models of assets, controls, threats contributing to impact

execution by data analysts...Feeding standards, audits and governance





Security is now so essential a concern that we can no longer use adjectives and adverbs but must instead use numbers. – Dan Geer

Questions?Jay Jacobs@[email protected]

Alex Hutton@[email protected]

mailto:[email protected]




threat landscape

asset landscape

impact landscape

controls landscape

risk

Prioritize

De-prioritize

Approaching the system as a system

threat landscape

asset landscape

impact landscape

controls landscape

risk

Suggested context:Capability to manage(skills, resources, decision quality…)

Data Sharing:

- Sources:- Qualify this Intel according to

framework- Treat with appropriate data quality

listings (let models shape the certainty)

Get Into Accounting

- Use existing models that take advantage of accounting concepts (ABC) to Talk to the LOBs


- Identify & Measure your processes- Identify & Measure your failures- Share data- Support data sharing efforts- Get into loss factors (ABC)

Challenging Conventional Wisdom

Conventional Wisdom may not be wrong- Question current practices - Seek Evidence and Feedback