Cyber Measurement Campaign

Cyber Measurement Campaign

Charles Wright, Lee Rossey

Presented at the ITEA Technology ReviewPresented at the ITEA Technology Review

25-27 July 2012

This work is sponsored by the Assistant Secretary of Defense for Research and Engineering under Air Force Contract FA8721-05-C-0002. Opinions, interpretations, conclusions and recommendations are those of the author and are not necessarily endorsed by the United States Government.

Goals

• Meaningfully track progress in research to improve cyber security

• Challenge assumptions and encourage new ideas

• Quantitatively measure key aspects of cyber systems– Resiliencyy– Agility– Trustworthiness– Mission Effectiveness

CMC Overview - 2CVW 07/2012

Outline

• Survey of cyber testbeds and ranges

• Emerging cyber range architecture

• Initial experiments with cyber resilient systems


Approaches to Cyber Assessment

Modeling & Simulation Cyber Range Prototype DeploymentAnalysis

• Real code, real apps, • Validates modeling,• Based on first principles • Fidelity/complexity/time

A l i Modeling & C b R Prototype

Real code, real apps,emulated environment

• Repeatable• Provide users with a real time

implementation for evaluation

Validates modeling, simulation, and emulation

• Difficult to repeat• Difficult to obtain ground

truth

• Develops global performance intuition

• Provides bounds that serve as implementation goals

• Provides corner cases to validate modeling, simulation, and

trade-off• Repeatable • Easiest transfer

across organizations

Analysis Modeling & Simulation Cyber Range Prototype

Deployment

Fidelity Low Low Moderate to High High

Scalability High High Moderate Low

Cost Low Low Moderate High

emulation

Cost Low Low Moderate High

Repeatability N/A High Moderate to High Low

Program Phase Early Early Mid-term Mid-term to Late

Selecting an appropriate combination of assessment approaches is critical to a


• Selecting an appropriate combination of assessment approaches is critical to a successful quantitative evaluation

• Study focused on Cyber Range

Core Cyber Range Infrastructure Elements

Sit /R

• What is the site/range purpose, mission?• Who do they primarily support?

Site/Range • Do they have a robust security infrastructure?

• Trained range personnel?• Trained operators (e.g. mission)?• Can they support open-air?

Range AssetsRange Tools • What equipment to they have?

• What level of scale, fidelity?• Is it representative?• Is it realistic?• Is it unique?

• What range tools do they have?• What is their sophistication? • Is it unique?

• Is it cyber targetable?

• All three areas must be considered when assessing cyber ranges

• What is their sophistication?• Are they custom, validated?


All three areas must be considered when assessing cyber ranges• Strength in all areas increases the value of the range

Elements of a Cyber RangeTest Setup:

Specify Configure Validate Baselinep y• Network Layout• Background Traffic• Attack Scenario• Defensive Posture• Data Collection

g• Hosts• Network• Users• Internet

• Network Buildout• Host and site

Reachability

• Traffic generation• Defensive Tools• Data Collection

Test Scenario:

Red Mission Modeler

CommercialCafe Blue Mission Modeler

Traffic Generation

HQ

Protected

Internet

Att k

Traffic Generation

Att k

Situational AwarenessMonitor

MilitaryNet

Tactical Tactical

Military

Power

Gov’tCommercial

Attacks Attacks

AwarenessDefensive Tools

Verify, AssessTactical

Test Environment


Data Analysis:Collect

Data AnalyzeStore Test

Configuration, Data, Results

Report

Cyber Range Inventory Summary

• Dozens of cyber testbeds and ranges across the US– Many small, special-purpose labs– A few large cyber ranges available for general use

• Mostly run by government military and academiaMostly run by government, military, and academia

• Many different tool sets, missions, and focus areas


Overall Cyber Ranges Assessment

• Many ranges and testbeds are currently available to support pure cyber development, experimentation and testing

• Technology advancements are being made to improve range automation and sophistication (e.g. DARPA NCR)– Technology can be transitioned to other ranges

• Traditional kinetic ranges are very mature for traditional missions

• Secure wide area networks (e.g. Joint IO Range) are essential to provide connectivity between users and cyber and kinetic ranges

• No common standards or interfaces exist between cyber ranges orNo common standards or interfaces exist between cyber ranges or range capabilities– Incompatible and fragmented systems and tools

S f f


• Significant investments are still required to fill the capability gaps

Example Academic Testbed: Emulab

• Name: Emulab

• Host Org: University of Utah

• Purpose: Open research in networking and security

• Scale: 13 racks of equipment– 500+ PC servers – 12 network switches– 802.11 wireless– Software-defined radio (GNURadio, USRP)– Programmable network interfaces (NetFPGA)


Outline





Cyber Range Architecture Goals

From discussions with the cyber T&E community, it became clear that a vision is emerging for cyber ranges that offer:

• Full end-to-end integration of cyber and kinetic systems– Integrate military, civilian and critical infrastructure

Secure closed loop open air test environment– Secure, closed-loop, open-air test environment– Support integrated testing of cyber and traditional military systems

Support for experimentation development testing training and• Support for experimentation, development, testing, training and exercises across DoD and national agencies

• Cyber range technology that isy g gy– Open, standards based– Interoperable, extensible, robust, scalable– Support varying levels of fidelity

Deplo able to contractors and de elopment laboratories


– Deployable to contractors and development laboratories

Cyber Range Architecture Components

Users

National

Technology Development

Range Services

Army PrototypesAgencies CapabilitiesAir Force Navy

Core Principles: Open Standards

Network

… Field TestsConcepts

Traffic

DefensivePackages

AssetManagement

TestsScheduler

Health &Status

Range

ModelsRangeAutomation

ThreatRange

DistributedOperations

Command &

Visualization Data Collection

Metrics

Data Archival

Instrumentation

D A l i

EnvironmentEmulation

Mission TrafficEmulation

TrafficGeneration

RangeSanitization

EnvironmentRange Setup, Control, Operation

Network

ThreatPackages

RangeValidation

Command &Control

Core Services

Data Analytics

Resources

…

Adapter Adapter Adapter Adapter Adapter Adapter

TargetableAccessible, IntegratedAdapter


WebTransportationMaritimeSatelliteAirborneRadars SCADA Radars Ships/MissilesTelecom

Range Architecture Illustration

Establish a set of major cyber-capable ranges

(Locations shown below are for example only)

Key Technologies:• DARPA NCR• JIOR RSDP


* Notional site selection

• JIOR RSDP


Leverage secure wide-area range networks to connect geographically distributed sites for each experiment

Key technologies• Joint IO Range (JIOR)g ( )• Joint Mission Environment Test Capability (JMETC)

Key Technologies:• Joint IO Range• JMETC




• JIOR RSDP


Pool resources from all connected ranges to form a unified, distributed virtual testbed

Key technologies• DARPA NCR• TENA

VM Servers, Experiment Nodes Key Technologies:

Data Collection,Analysis, andVisualization

Range Automation

Range Adminand C2

Users

y g• DARPA NCR• TENAOPFOR





• JIOR RSDP


Key Technologies:• Traffic Generatio• Emulated

Execute experiments on the testbed

• Emulated Components

VM Servers, Experiment Nodes Key Technologies:

Data Collection,Analysis, andVisualization

Range Automation

Range Adminand C2

Users

y g• DARPA NCR• TENAOPFOR





• JIOR RSDP

Approaches to Connect Cyber with Kinetic Assets

Cyber Range(s)Targetable Kinetic Resources

Integration with Kinetic AssetsTargetable Kinetic Assets

Access to integrated and targetable

Access to integrated kinetic

Resources

Cyber Range(s)

IO Range JMETCIO

Range JMETC

Integrated Cyber Kinetic Cyber Range w/ Access to Kinetic

targetable kinetic resources

resources

Integrated Cyber-Kinetic Cyber Range w/ Access to Kinetic• Full control over use and scheduling of targetable

kinetic resources• Can support experimentation • Can support highly classified, contained cyber tests

on kinetic assets

• Expertise and operations maintained on the kinetic ranges• Targetable assets can be fully integrated with other weapon

systemsPros

on kinetic assets

• Requires experts from many domains to operate suite of systems

• Competition with other ranges for FME/FMA assets

• Traditional ranges may not have cyber expertise to support integrated testing

• Potentially difficult to schedule use of kinetic assets (competing priorities)

• Potentially difficult to integrate targetable kinetic systems

Cons


y g g y• Potential reluctance by traditional ranges to support

experimentation or destructive testing on kinetic assets• Must be able to execute secure distributed testing (e.g. range

control, instrumentation, analysis) across systems

Outline





Initial Experiments

• Primary Goals– Demonstrate experiments to measure and quantify cyber resiliency

with mature research prototypes– Measure resulting improvement to cyber security

• Availability, confidentiality, integrity

• Focus is on experimentation, not test & evaluation– This is not a test! No pass or failp– Not (yet) intended to be comprehensive or complete– No attempt (yet) to assess the overall security of the system– No attempt (yet) to assess impact to real missionsp (y ) p


Three Technologies for Initial Experiments

• ARCSYNE• IP hopping IPSec VPN gateway• Protects a closed community of interest from external threats• Protects a closed community of interest from external threats

• LPS: Lightweight Portable SecurityB t bl Li li CD f ti it f tiF f • Bootable Linux live CD for continuity of operations

• Approved by DOD for use in case of pandemic flu, etc.Focus ofthis talk

• TALENT• Dynamically composable platforms• Enables applications to seamlessly migrate from one hardware-

software platform to anothersoftware platform to another


Lightweight Portable Security

• Linux-based OS on a bootable CD-ROM– Includes web browser, VPN, office

software suite– Approved by DOD for use in maintaining

continuity of operations

• Improved security over standard desktop systems• Improved security over standard desktop systems

– Minimal software included

– No persistent storage


Cyber Kill Chain

Attacker seeks to carry out a sequence to steps to achieve his goal

Develop Attack

Recon

to steps to achieve his goal

Launch Attack

Persist & Establish

C2D5 Effects

or Exfil

Assess Eff t

Defender attempts to block his progress at every step

Effects


Cyber Kill Chain

Attacker seeks to carry out a sequence to steps to achieve his goal

Develop Attack

Recon

to steps to achieve his goal

Launch Attack

Persist & Establish

C2D5 Effects

or Exfil

Assess Eff t

LPS makes Attack Development and Persistence more difficult for the EffectsPersistence more difficult for the

attacker


Cyber Kill Chain

Develop Attack

Recon

Launch Attack

Persist & Establish Improvement is very

diffi lt t C2D5 Effects

or Exfil

Assess Eff t

difficult to measure

Underlying science: Theory of Computation

Challenge: EffectsChallenge: Analysis of software is still an unsolved problem Improvement is much

easier to measure

Underlying science:


Underlying science:Queuing Theory, Stochastic Processes

LPS Experiment

Hypothesis: Increasing recovery rate increases the time required for an attacker to penetrate the network

Experiment Outline: Multiple LPS hosts communicate with a remote network, while an attacker attempts to gain a point of presence on a fixed percentage of the LPS clients

Threat model: We assume that the attacker– Can gain a point of presence on a LPS machine though a remote software exploit

– Cannot persist in hardware (BIOS, flash, etc.)


LPS ExperimentVariables of Interest

– Session Length (influences recovery-rate)– Percentage of hosts required for coordinated attack (influences workload)

Measurements: – Workload: Time to gain a point of presence on a fixed percentage of hosts– Resilience: Rate of recovery

acke

r Del

ayIn

crea

se in

Atta


Resilience: Recovery Rate

LPS Experiment Configuration

Custom Firefox extension module simulates

Hypothesis: Increasing resilience (recovery rate) increases attacker’s delay

55 Virtual machines running LPS CD-ROM image40 Web sites on an emulated Internet

Custom Firefox extension module simulates a security vulnerability in the browser

1 Malicious site serving “drive-by” malware downloads

Malicious site includes a special “exploit” header that causes the simulated vulnerability to download and run a program (the “payload”) from the attacker’s site.

4 Dell PowerEdge R410 VM servers

1 D ll k t ti i l ti 55 i t l

Simulated exploit payload spawns a new process on the victim machine and sends a constant stream of packets until the machine is rebooted.

1 Dell workstation simulating 55 virtual users

Each “user” browses the web continuously, andreboots his VM intermittently with Poisson rate R.

Users select web pages uniformly at random. Each user views his selected web page in its

Experimental Methodology• Vary reboot rate R

• Average session length: 2 hr, 1 hr, 30 min, 10 min• Let each scenario run once for 22 hours

• Observe impact on attacker delay time


Each user views his selected web page in its entirety, and stays on a single page for no more than 4 minutes at a time.

• Observe impact on attacker delay time• Observe impact on attacker’s success rate

Results: Increase in Attacker Delay

Average S i

Resulting AvgI f ti

Recovery Rate ( i

16

18

s)Session Length

Infection Length

(recoveries per machine per hour)

2 hr 83 min 0.725

1 hr 49 min 1.22 8

10

12

14

To infect 10%

To infect 20%

er D

elay

(hou

rs

30 min 24 min 2.51

10 min 8 min 7.49

0

2

4

6 To infect 30%

Atta

cke

Increased recovery rate due to frequent reboots does not prevent th tt k f i i i iti l f th ld i t th t k

02 hr 1 hr 30 min 10 min

Average Session Length

the attacker from gaining an initial foothold into the network. (No increase in time required to infect 10%.)

Recovery rate has a profound effect on the workload required to achieve substantial penetration of the network.


p(22x increase in time required to infect 30%.)

Experimental Results (2)

Rebooting more frequently reduced the attacker’s average penetration of the network by a factor of 4.


Summary

• Surveyed national cyber ranges and testbeds

• Articulated an emerging vision for the nation’s cyber ranges

• Demonstrated real experiments to measure resiliency and security in cyber


Documents

Cyber Measurement Campaign