Crawl, Walk… Run!

A CASE STUDY IN IMPROVING SECURITY PERFORMANCE THROUGH METRICS

2

Problem Statements Large Number of Stalled Security Findings

Ineffective at Influencing Behavior

Limited Leadership Visibility

Constrained Resources◦ Limited staff time◦ Immature culture of operational measurement within IT

3

Agenda About Seattle Children’s

Understanding the Environment

Drowning in Findings

Designing a New Approach

Adoption and Current State

Future Plans

Thoughts Inspired by Haruki Murakami

4

About Seattle Children’s“We believe all children have unique needs and should grow up without illness or injury. With the support of the community and through our spirit of inquiry, we will prevent, treat and eliminate pediatric disease.”

5

Commodity Tools• PowerShell and PythonETL language

• Access, SharePoint, SQL Server and MongoDBData storage

• TableauVisualization

• InfoPath, Access and SharePointStructured data entry

• Blood, sweat and tearsEverything else

6

Generally Available Data Sources

CMDBVulnerability Data• Nessus & Web

Network Configurations

Active Directory

Mail• Exchange, TMG

& BlackBerry

Incidents• Security & DLP

Security Findings

7

Exploring the Problem Space Vulnerability Management

Demo◦ Vulnerability Management Dashboard

8

Drowning in Findings Remediation activities persistently stalled

Demo◦ Security Findings Dashboard

9

Designing a New Approach Principles

◦ The program is more important than the process◦ Focus on the outcome◦ Data should be low cost to acquire

◦ Frequent gathering allows frequent reporting

Goals◦ Make performance transparent◦ Provide owners the freedom to act

10

A New Hope Solution: Shift from compliance-based finding to outcomes-based measurement

24 month program with 6-month performance objectives

Results to be reported to executive oversight committee

Permit any exceptions needed so long as overall program targets are met

◦ Exception to be reviewed at end of 24 month effort

11

Defined Measures• Reduce the number of severe vulnerabilities• Reduce the total number of vulnerabilities

Goal 1: Address High Risk Applications

• Allow no hosts to become worse than these• Reduce vulnerabilities on these systems

Goal 2: Address Top Risk Systems

• Reduce overall time to patch• Improve scan frequency

Goal 3: Improve Overall Patching

Program

12

Current State Performance Measure rolled out to IS leadership

◦ Co-presented by CISO and CIO◦ Reporting to board-level committee

Automated monitoring implemented

Demo◦ Vulnerability Performance Measures

13

Future Plans Risk Identification Process

◦ Migrate additional findings to performance based measures

◦ Statistical clustering of applications to identify common risk factors

Incident Reporting◦ Root cause and impact analysis◦ Mini-DBIR creation (VERIS, Python, MongoDB)

Metrics Catalog◦ Target setting◦ Receive data rather than querying directly

14

Lessons from RunningWhat I talk about when I talk about metrics.

15

Discussion

16

Backup SlidesDISCUSSION

17

Questions1. Top-down vs. bottom-up metrics design

2. What items should be date driven vs. objective driven

3. Describing approach to auditors and regulators

4. Prioritizing areas for treatment