Perforce on Tour 2015 - Grab Testing By the Horns and Move

Grab testing by the horns and move

September 22, 2015Steve Howard, Safety & Security

Engineer

Agenda

• A holistic approach to security

• Blending DevOps and Agile for security

• Examples of security defects

• Best practices for Agile teams

• Q&A

2© 2015 Rogue Wave Software, Inc. All Rights Reserved

A holistic approach to security

4

Security information overload

NewsBlogs, social media

conferences

Security standardsOWASP, CWE, CERT, etc.Senator Markey report

NVD, White Hat, Black Hat OEMs, internal

Media More and more software running inside your carStandards and legislation

Research Requirements

Developers don’t know security

(80% failed security knowledge survey)

© 2015 Rogue Wave Software, Inc. All Rights Reserved

A holistic approach to security

5

Information overload Develop an adaptive process

Process External Data

Internal Threat Metrics

Action


7

Developing a threat metric

Build score

• Automated and functional testing can give you a pass fail metric on every run of the test suite

• A metric can be generated from penetration testing based on the number of exploitable paths in your code base

• Software quality tools can give you a count of critical static analysis and compiler warnings

• A metric can be developed based on the presence of snippets of open source code previously undetected or open source with new known vulnerabilities

• All of these metrics can be generated on every build of your software


Blending DevOps and Agile for security

Traditional development: Security as a service

9

Adaptive

Separation of duties for testing and auditing

Separate testing tools, results fed to development

Traditional secure development lifecycle activities

Design

• Establish design Requirements

• Analyze attack surface

• Threat modeling

Build

• Use approved tools

• Deprecate unsafe functions

Test

• Static analysis• Dynamic

analysis• Fuzz testing• Attack surface

review• Open source

review

Deploy

• Incident response plan

• Final security review

• Release archive

Development, compliance, and security are independent functions

Req's

• Establish security requirements

• Create quality gates

• Risk assessments


Consequences of security as a service

© 2015 Rogue Wave Software, Inc. All Rights Reserved.

10

Adaptive

Cost of Remediation Source: Barry Boehm, “Equity Keynote Address” March 19, 2007

Cost of Remediation

Increased remediation

costsDelayed releases

Security and development become adversarial

5x

Design

• Establish design requirements

• Analyze attack surface

• Threat modeling

10x

Build

• Use approved tools

• Deprecate unsafe functions

20x50x

Test

• Static analysis• Dynamic

analysis• Fuzz testing• Attack surface

review• Open source

review

150x

Deploy

• Incident response plan

• Final security review

• Release archive

1x

Reqs

• Establish security requirements

• Create quality gates

• Risk assessments

Agile development: Integrated security


11

Adaptive

AcceptSprint 1

Sprint 2

Sprint nRelease

ChangeAdjust and Track

FeedbackReview

Next Iteration

No!

Yes!

Release to

Market

Integrate and Test

Integrate and TestIntegrate

and Test

Multiple testing points

Rapid feedback required

“Outside” testing does

not meet Agile needs

DevOps SDLC

12

Continuous Integration

SDLC Step

UAT/exploratory

testing

Functional testing

Performance load security

Release Deploy

Metric

UnderstandNeedsInvent

Solution

DevelopBuild

CommitIdea


Security example

14

Load, performance, security…testing phase

Load, Performance, Security, … Testing


15

Develop, commit & build


16

Develop, commit & build


Best practices for Agile teams

Best practices for Agile teams


18

Adaptive

Context for remediation

Integrate security and compliance testing 1 Enforce standards that relate to the project 2 Context for remediation 3 Continuous improvement 4 Reporting for all stakeholders5

Best Practice 1.

Integrate security and compliance testing


19

AdaptiveGive Agile teams tools & responsibility for testing

Self-sufficiency is required for rapid

reaction

Run tests on development

schedule

Embed security with Agile team for triage and

assistance


20

Adaptive

Best Practice 1.

Integrate at IDE and Build Server

Do what works best

for each team

Run separately

Integrate at IDEIntegrate at build server

Testing and remediation on the

fly

Testing at the end of each

sprint

Testing with each sprint test build


21

Adaptive

Best Practice 2.

Enforce standards that relate to the project

Understand the

objectives

Risk varies with

application deployments

Use flexible rule sets

Compliance rules (e.g.,

PCI)

Language and framework

specific rules

Custom rules for custom frameworks

High/low security

requirements


22

Adaptive

Best Practice 2.

Compliance rule sets

PCI-DSS v 3The vulnerabilities listed at 6.5.1 through 6.5.10 were current with industry best practices when this version of PCI DSS was published. However, as industry best practices for vulnerability management are updated (for example, the OWASP Guide, SANS CWE Top 25, CERT Secure Coding, etc.), the current best practices must be used for these requirements.

Specific rule sets

SANS Top 25

Reporting for regulatory audits

OWASP Top 10

Provide information needed to act


23

Best Practice 3.

Context for remediation

What needs architectural

review?

Provide actionable results

Prioritize results to accelerate triage

Eliminate “noise” from reporting

What can I fix quickly?


24

Adaptive

Best Practice 3.

Minimize code changes after code check-in

Trace errors to root causes

Input validation

Manifests itself when

tainted data is used

A single error can result in 10’s or 100’s of issues


25

Adaptive

Best Practice 4.

Continuous improvement

Help developers learn on the job

Move from training “events” to a training

“process”Source: https://uwaterloo.ca/counselling-services/curve-forgetting

Push remediation advice to the IDE


26

Adaptive

Best Practice 5.

Enterprise reporting

Development

Security reporting

Compliance reporting

Legal reporting

Identify security training needs

Maintain independence of audits

Testing for OWASP/SANS bugs

Audits and reporting for OSSTraceability for security risks

28

Conclusions

The application security world is fluid Create concrete,

actionable strategies(Threat Metric, analysis & scanning)

Delivery cycles are short Update regularly with well-defined process

(Agile, CI)


Technology

Perforce on Tour 2015 - Grab Testing By the Horns and Move