Fixing Security by Fixing Software Development Using Continuous Deployment Nick Galbreath, VP Eng IPONWEB, http://www.iponweb.com/

@ngalbreath http://www.client9.com/

May 14-15, San Francisco, California

Follow Along Latest version posted online:  

http://slidesha.re/144OIv3http://www.client9.com/

Nick Galbreath (@ngalbreath) ! Spoken at Black Hat, DEFCON, OWASP ! Author of libinjection: advanced SQLi

detection used in > 2 WAFs, 1 Honeypot ! Book on cryptography ! but really... ! Engineering Management and Software

Development for high growth startups. ! Personal site http://www.client9.com/

IPONWEB ! customized online advertising

infrastructure and exchanges ! engineering offices in Moscow, with

business offices in London, New York and Tokyo.

Original Abstract Fixing Security by Fixing Software Development Using Continuous Deployment Do you have an effective release cycle? Is your process long and archaic? Long release cycle are typically based on assumptions we haven't seen since the 1980s and require very mature organizations to implement successfully. They can also disenfranchise developers from caring or even knowing about security or operational issues. Attend this session to learn more about an alternative approach to managing deployments through Continuous Deployment, otherwise known as Continuous Delivery. Find out how small, but frequent changes to the production environment can transform an organization’s development process to truly integrate security. Learn how to get started with continuous deployment and what tools and process are needed to make implementation within your organization a (security) success.

Well that's a bold statement...

"Fixing Security by Fixing Development Using Continuous Deployment"

and here's another

For web applications, our release-based software development lifecycle is still based on a pre-Internet model and is harmful to organizations andparticularly harmful for security.

What needs fixing? ! SQLi dropped from #8 to #14 in the latest

White Hat "The State of Web Security" report. Good news, right?

! This means SQLi is only 7% of websites. That's 1 in 15. And this is #14 vulnerability!

! And time to fix was on average 196 days. That's embarrassing.

Veracode claims 32% of incoming web applications have SQLihttps://info.veracode.com/state-of-software-security-report-volume5.html

https://reg.whitehatsec.com/WPstats0513

Even worse... ! Number 1 driver to fix security problems...

compliance. ! Number 1 reason to not fix security is...

compliance. ! Not.. ! keeping our employees and customers safe ! protecting corporate interests. ! improving quality ! being good at what we do.

Security Products #1 .. in security bugs VeraCode: State of Software Security, V4 December 2011

Let's Just Give Up ! “You could spend all your resources chasing such

things as this,” William Ribich, the former president of Technology Solutions Group [ QintelIQ ], said in an interview in January. Ribich, who retired in November 2009, shortly after the discovery of a major data theft, said he needed to balance the uncertain risk that the hackers could use what they stole against a growing shopping list of security products and consulting fees.

! "You finally have to reach a point where you say ’let’s move on,’” he said.

http://www.bloomberg.com/news/2013-05-01/china-cyberspies-outwit-u-s-stealing-military-secrets.html

I would call that broken

! But preventing SQLi isn't a technically hard problem.

! And most security patches are very small. ! How did we get here?

Software Product Model ! Code flows between functional groups ! Product Managers spec code ! Engineers write code ! QA engineers tests code ! Release engineers package code ! Operations runs code ! ... and Security does something too.

High Distribution Cost ! The Software Product Model is designed

for software where the cost of distribution is high. "High" might be financial, risk, time, resources, customer annoyance. ! Retail, physical product, CD/DVD ! Embedded of Exotic Hardware ! Safety, Medical or Defense Systems ! Operating Systems (desk or phone) ! Homework (1-time deploy)

Software Product Lifecycle

Change to Production

Web Applications Year 2000 ! Mostly followed Software Product Model

since that's all we knew. ! High barrier to entry ! Specialized Hardware, Software and

People needed to get started. ! Lots of engineering needed to keep things

running. ! (side note: CERT/CVE started in 1999)

True Story #1 ! "Can't push out the spelling error fix – it's

too risky" ! "That code as already been through QA,

it's locked down." ! "Product has to prioritize that change, else

we aren't touching it."

True Story #2 ! We'll do an iteration, where we try to fix as

many things as possible. ! This won't be a scheduled iteration, it will

be done because things are so bad ! So the spelling error will get fixed...

uhh, who knows when.

Web Applications 2013 ! Almost no barrier to entry ! Commodity hardware ! Programming not that hard ! Scaling problems can

be mostly outsourced (mostly)

Cost of Distribution 2013 ! Frequently no compile step

or it's very fast. ! Moving to production a few kilobytes or

megabytes of code over 1Gbps, 10Gbps link.

! In other words... free

Failure is very different however ! Most web applications are data-driven. ! Frequently have social features, APIs,

user-generated content. ! Failures might be due to algorithmic

problems... but... ! Most likely to due to user input, bad data

in database or operational load. ! this means data in past can cause

problems in the future.

Releases and Problems ! When a web-release goes out, and has

problems.... ! Next week is spent tracking down who

changed what, where. ! Re-QA ! Re-Push ! meanwhile new code is piling up.

When SPM meets Web Apps ! A long time between code being written

and code being released. ! Might be weeks or months ! Feedback loop between code-in-dev and

code-in-production is broken ! When security or bug reports come in, the

author is likely on a different project.

Hypothesis ! It is impossible to simulate the production

environment in development, either due to operational differences or data differences.

! No amount of QA or Security Testing can prove you don't have bugs, vulnerabilities, or cause severe operational problems.

! You have bugs and vulnerabilities,right now, in your application.

Impedance Mismatch ! Easy to write code, + ! Long release cycles + ! Security as an end-of-line or out of band

process == ! no one cares

So the Answer is... ! Going slower? I'm sure your boss will love

that suggestion ! More steps and process? In other words,

slower. ! Asking for more people? Sure but good

luck hiring them. Doesn't scale. ! Asking for more products? Since the

others have worked so well.

Continuous Deployment ! Also known as Continuous Delivery. ! A System of Software Production

Characterized by Numerous Small Changes the Production Environment, initiated by the author of the change.

You change it, you push it to prod.

Deployment != Feature Release

"Writing Software" ! Software Developers think their job is

writing software. ! And so, they love to make things perfect

before anyone else sees it. ! Impolite: "data hiding" ! code is hiding on developer's computer ! or on some branch ! in other words invisible until it's ready.

Actually ! The software engineer's job is actually

writing running software, that works well. ! This idea is so alien, that companies have

to remind the engineers of this.

Rackspace Haiku

writing code is hardif you cannot deploy it does not matter

@paulvx from DevOpsDays Austin 2013

Facebook

"Move Fast and Break Things....

Except "Push" (deployment system)

via http://mitadmissions.org/blogs/entry/move_fast_and_break_things

Etsy Mantra Just Ship

Today's goal ! but for today the goal is getting the

developer to care about their code in production.

! If you don't have that, I don't think you can really solve security problems.

How does this work? ! Really? Developers push their own code

out? ! How is this not a disaster. ! How is this not a security disaster?

The Deploy Button ! What is you had a button that said

"DEPLOY" ! That pushed to production, whatever is

current in your source control system. ! And took about a minute ! The change and who pressed the button is

logged, but that's it.

Part 1: Fear ! No one is going to push it ;-) ! Meanwhile code is piling up

Real example: A new hire I had at Etsy was afraid of deploying an HTML change that they made. "But I don't want to break the site!"

Part 2: First Push ! Someone brave will press the button ! And very likely the site will explode, and a

rollback will need to be done. ! They'll know since someone else will have

told them.

Part 3: With Graphs ! Let's get all those operational graphs out

in the open. And put them right next to the button.

Part 4: Push #2 ! Repush ! Site might still explode ! But the developer is aware and can

rollback.

Take 5: Isolation ! Hmmm, the developer notices that in the

change set, a million things are going out. ! Maybe just pushing out a smaller change

will help isolate the issue.

Take 6: Success! ! Yes, the developer just pushed out some

code and made the site better. ! The secret about continuous deployment

is small changes that can be easily understood.

Take 7: Dark Pushes ! Now we got some bugs fixed, let's push a

feature. ! First let's push out all the supporting files.

Since they aren't being called, they do nothing and are safe to push out.

! Now everyone can see them

Take 8: Getting the feature live ! Instead of "all at once", we slowly ramp up

a feature. ! if (user_id % 20 == 0): do new feature;

! we change change the percentage easily with another code push.

! or turn it down. Much nicer change log. ! While the site didn't explode, it's hard to

see if the feature is being used or not.

Take 9: Application Level Graphs ! Allow developers to instrument their code

so they can see what is happening in production.

! Enter StatsD and other UDP-based tools ! Enter centralized logging and in-

application method to make it easy to log problems.

Take 10: Communication ! So far good for one developer. ! To scale up, you'll need a system to allow

developers. ! IRC-like tools work well (e.g. "the push

channel") – skype, jabber, hangouts, etc

Along the way ! Expose production logs to developers ! Add in a staging-step where the code

goes to faction of the cluster, so developers can test with real traffic

! Try to make development closer to prod. ! Make "smoke tests" to catch basic errors ! Add syntax checkers to eliminate obvious

issues. ! Use static analysis to find bugs

Mistakes will happen ! Do postmortem analysis ! Everyone thought they were doing the

right thing at the time. ! "How can the environment be changed to

prevent this" and build tools to enforce it. ! (Rarely can you truly change people)

Sounds good, but what about...

That guy who pushes at 3am ! Courtesy and convention will converge

very quickly when the site goes down at 3am and the developer starts getting calls ;-)

! Of hours pushes of course can happen, when they notify operations.

What About Code Reviews? ! Yes, please do them. ! Nothing here prevents code reviews. ! In fact code reviews are easier since ! they are small ! they are in mainline not some branch

What about Security Reviews ! Please do them. ! Nothing here eliminates architectural

planning or review. ! This actually doesn't change the SDLC

very much.

What about Agile Methods ! (everyone seems to have a different idea of

what Agile is but..) ! Agile methodologies typically work to

improve the business spec / development cycle. (are you building what the customer wants)

! But doesn't address code deployment. ! They are complimentary practices.

What about Customer Service? ! "Don't they freak out with all the

changes?" ! Remember: deployment != feature release ! Most deployments do very little from the

customer point of view ! Feature releases (frequently controlled by

ramp-ups or flags) always needs to be coordinated with product and customer service.

What about Compliance? PCI? ! Let me tell you about compliance... ! mechanism not policy

Back to Security

Obvious Benefit to Security ! Security patches can go out quickly ! You know this since they are now just part

of a normal development cycle.

More Importantly ! That Engineer who previously didn’t push

code is now sensitized that their code has consequences and are responsive and empowered to fix it.

! It’s amazing how interested engineers become in security when you find problems with their code when they are able to fix quickly themselves.

Hack The Stack ! A side effect of this you now have tools to

repurpose for security and monitoring of production

! Note that most changes are not security problems.

Logging ! Due to allow developers to see application

logging, it's now very easy to instrument the application to log security events.

! Or add logs to times when you are under attack.

Graphing ! Make dashboards of ! SQLi and XSS attacks ! Every type of log-in failure ! Core Dumps ! Database Syntax Errors

Static Analysis ! You now have a place to insert them. ! Work with QA group to add more code

quality tests.

Post-Commit Checks ! Alert on when sensitive areas of the code

are changed (auth) ! Alert on crypto usage (why is developer

using MD5.. hmmm) ! Alert your programming languages

"dangerous functions"

This allows you to engage the developer at the start of the cycle.

Faster is Better ! You could do most of this in a normal

release-cycle software lifecycle. ! The difference is you are finding problems

at the start instead of 10m before the launch and telling everyone to stop.

! The feedback loop works.

New Roles, Less Silos ! Developers: works with operations ! QA: works on building systems for testing,

to empower others to write better tests ! Release Engineering: tools to enable code

to flow faster ! Security: in-house consultancy, secure-by-

default architecture, monitoring

Getting Started

Goal: 50% reduction in deploy times times ! Whatever your state of deployment is, no

matter how many people are involved, no matter how long it currently takes, make a goal of cutting it in half.

! This is an easy sell to management just on cost basis.

! Everything else flows from this.

Mechanism not Policy ! Strive for the fastest deployment mechanism for

possible ! But you define the "continuous" in Continuous

Deployment ! Yes, Etsy was 60+ deploys per day, with each having

multiple authors. ! Current gig? we have rules of no more than 3 per week

since our customer have asked for that, and only deployed at "low-tide"

In Other Contexts

In other contexts: Operations ! How fast can you deploy OS changes to

you production environment?

In other contexts: IT ! How fast can you deploy desktop

changes?

In other contexts: software product ! here "production" might be getting code

into the main branch and running automated build / test.

! It's the flow of code: little changes vs big.

In other contexts: silicon ! Continuous deployment already done for

silicon! wut? ! Only small changes, with tests are allowed

to be committed! ! Big changes are rejected.

Learned the hard way that big changes are completely unmanageable

One more thing...

Your Attackers Use Continuous Deployment

Continuous Deployment Start with 50% Reduction Build the Deploy Button

Nick Galbreath ★ @ngalbreath ★ nickg@client9.com ★ http://www.client9.com/ http://slidesha.re/144OIv3

© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it

should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Fixing Security by Fixing Software Development Using Continuous Deployment Nick Galbreath ★ @ngalbreath ★ http://www.client9.com ★ nickg@client9.com