Time Traveling: Adapting Techniques from the Future to Improve Reliability, Jacob Kitchel of Exelon

Time Traveling: Adapting Techniques from the Future to Improve ReliabilityJacob KitchelJanuary 14, 2014

Presentation Title2

Bio

Present:

Security Architect at Exelon

Past:

Security & Compliance at Industrial Defender

ICS Risk Assessment (PT, VA, etc.)

Application Security research (Project Basecamp)

Enterprise Security Operations & Monitoring

Speaker (S4, EnergySec, ISA, API IT Security)

Hilarious LinkedIn Endorsements

Presentation Title3

Abstract

Technology in ICS environments lags the Enterprise by 10-15yr. This often leads to ICS companies having to stand by while other more nimble institutions are able to take advantage of new technology. What few people realize, is that our industry gets to watch the future happen out on the Internet and then pick and choose the best techniques to adapt and bring back in time.

Presentation Title4

How far have we come?

We have:• Compliance

• Incidents?

• Specialization

• Conferences

• Big Headlines?

• A LOT of vulnerabilities

Presentation Title5

Where has it gotten us?

Here we are:

• Multiple revisions of compliance requirements

• Basic improvements in security monitoring

• SOME patching happens

Presentation Title6

What is working against us?

Mountains or mole hills?• Refresh cycles

• “If it isn’t broken, don’t fix it”

• Skill set(s)

• Unknown unknowns

• Security v. Operations

• Budgets & time

Presentation Title7

Progress is sloooooowwwww….

Presentation Title8

What to do?

• Where do operations goals and security goals intersect?

• What is the lowest common denominator?

• What can have an impact?

It’s all about the customer…If you aren’t solving customer pain, then you aren’t doing anything

Presentation Title9

It’s about the customer

Operations

• Safety

• Reliability

• Uptime

Security

• Security

• Compliance

• Vulnerabilities

Where do these two areas intersect?

Customer

Presentation Title10

Where do Security and Operations Intersect?

• Patching

• Change Management

• Configuration Management

In other words…• Time-intensive

• Error-prone

• High-risk activities


Solving “Customer” problems lets you solve security

How can we do that?


Take a step back…to the future!


How?

Is there anyone that “looks” like us?Has anyone solved this problem before?

How can we:

• Reduce time commitments required

• Reduce errors

• Reduce risk


Know any of these names?


Internet-scale companies

• Millions of customers, world-wide

• High-availability, (near) zero downtime

• Complacency is death

• Some of the brightest minds >40

• Solving scale and complexity problems that we can barely imagine

• Leveraging software and hardware to dynamically define environments

• Have to be reliable and fast


How are they doing this?

They are doing it CONTINUALLY.

Continuous Delivery:

Changes to your environment are

proven to be deployable with predictable results


But you say, “There’s a catch!”

Continuous Delivery was popularized by Internet companies!

Internet companies deliver software and/or services as their products!

They’re not like us! We have a physical process!

Etc, etc, etc…

Guess What?Continuous Delivery is a collection of tools and processes – tools and processes that you use to focus your ability to deliver your physical process

Hint: You’re not getting off that easy! ;)


What does this mean to us?

• Major reduction in time and effort to push changes

What would a major time/effort reduction mean to your operations?• 500hr task takes 5 hours or 5 minutes?

• 40hr task takes 4hr or 4 minutes?

• How many times do all of your tasks gets repeated annually?

• What if you could save half of that time and effort?


How do we get there?

Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation


Continuous Delivery in Practice

How do you move a mountain?


First steps first

• Follow your build/development process & write it all down• What takes the most time?

• What tasks are the most error-prone?

• What tasks require the most human intervention?– Automate these tasks FIRST!

• What tasks cause headaches or are time sinks?– Automate these next!


Facilitate Adoption

• Put everything into version control

• Add tests to verify that changes work

• Manage servers with configuration management tools

• Monitor EVERYTHING


Tools

• Software-defined infrastructure

• Monitoring

• Continuous Integration

• Version Control

• Code Review

• Configuration Management

• Orchestration

• Dashboards

End Goal• Quality

• Reliability

• Speed


Tool Specific Information


Software-defined Infrastructure

Tool example:

• Quali Systems TestShell

How to apply:

• Define common network architecture and system objects

• Create test topology

• Run tests and see what breaks, verify what works


Version Control

Tool examples:

• Git

• SVN

• CVS

How to apply:

• Track versions of clear-text configuration files

• Firewall, switch, router configuration files

• Application configuration files


Configuration Management

Tool examples:

• Puppet

• Chef

• Ansible

• Salt

• Microsoft SCCM

How to apply:

• Store all configurations in management tool

• As machines run, configuration management tool ensures declared configuration


Orchestration

Tool examples:

• Puppet

• Chef

• Mcollective

• Ansible

• Capistrano

• WinRM

How to apply:

• Determine order of components

• Leverage tools to operate, deploy, and automatically configure systems in proper order


Virtualization

Tool examples:

• Most common tool here is VMWare and is likely your vendor’s approved virtualization provider

How to apply:

• Mirror Dev, Test, and Production environments

• Bonus: backup/redundant assets

• Can begin to act as a “do over” button


Metrics & Dashboards

Tool examples:

• Logstash

• Graphite

• Nagios

• Cactii

How to apply:

MONITOR EVERYTHING


Continuous Delivery tool

Tool example:

• Thoughtworks Go

How to apply:

• Automate and streamline the build-test-release cycle


Automated Testing

Tool examples:

• Thoughtworks Twist

• BDD/TDD tools

How to apply:

• Write tests to verify functionality

• Run tests automatically every time new code, features, or configuration changes are made