Download pdf - AppSec Survey 2.0 Fine-Tuning an AppSec Training Program Based on Data

AppSec USA 2014 Denver, Colorado

AppSec Survey 2.0: Fine-Tuning an AppSec Training Program Based on

Data

John B. Dickson, CISSP @johnbdickson

September 18, 2014

John B. Dickson, CISSP

•  Application Security Enthusiast •  Ex-AF Guy & ISSA Distinguished Fellow

•  Serial Entrepreneur & MBA Type

•  Dad

Introduction

When Not Thinking about AppSec…

I am Snake Hunting on a Ranch in South Texas

Snake Hunting Essentials

Cooler Hat Cool Hat

Snake Guards Common Gardening Tools Machete

Guy who has a machete and who is actually good at “catching” snakes

OWASP AppSec 2011 t-‐shirt

© Copyright 2014 Denim Group - All Rights Reserved

•  Background •  Premise •  AppSec Study 1.0 Results – What We Learned •  Approach and Survey ParKcipants •  Key Results •  What We Can Put To Work •  Conclusions and QuesKons & Answers

Overview

•  Things we Knew Last Year

•  Key Findings of Last Year’s Study

•  AddiKonal Stuff We Learned Along the Way

•  Development training is hard

•  Results are rarely measured for ROI

•  Training is typically part of any AppSec program

AppSec Study 1.0 Results



•  AddiKonal Stuff We Learned Long the Way

•  25% retenKon aXer training

•  QA did worse than architects and soXware developers

•  Respondents answered basic awareness quesKons but not coding pracKces




•  AddiConal Stuff We Learned Long the Way

•  SoXware developers learn differently than companies teach

•  IncenKves ma[er •  Surveys are hard!


Overview of 2014 “2.0” Study

•  600 respondents •  Represents mulKple industries •  Asked the same applicaKon security quesKons as

2013 survey •  Expanded to include training method quesKons •  No “before” and “aXer” analysis •  No classroom training opportuniKes •  Used more social media •  Data collecKon ongoing

Approach and Survey Participants

Sample QuesCons

QuesKons that tested basic knowledge of applicaKon security:

•  ApplicaKon security is best defined as… •  Threat Modeling is… •  Input ValidaKon is…


Sample QuesCons

QuesKons that tested understanding of defensive coding:

•  Marking a cookie as “secure” will… •  Which of the following will help protect against XSS…

•  Which of the following is NOT an example of good session policy…

Delivery Means •  Direct Delivery of Customized Links via E-‐mail

•  Survey Monkey paid •  Social Media

– Facebook – Linkedin

Targets •  SoXware Developers •  Architects •  Quality Assurance


Demographic Questions Asked

•  What is your primary job funcKon?

•  What is your company's size?

•  How many years of soXware development experience do you have?

•  How much previous applicaKon security training have you received?

2014 Study Demographics

Less than a Year 18%

1-‐2 Years 9%

2-‐4 Years 10%

4-‐7 Years 13%

7-‐12 Years 16%

More than 12 Years 34%

How many years of soMware development experience do you

have?


Other 35%

SoXware Developer

53%

Quality Assurance

6%

Architect 6%

What is your primary job funcCon?


8% 8%

29%

8% 10%

37%

What is your company size?

1-‐24 Employees

25-‐99 Employees

100-‐499 Employees



10,000 or more Employees


None 31%

Less than a Day 19%

At least 1 day, but less than 2 days

17%

At least 2 days, but less than 3 days

8%

More than 3 days 25%

How much previous applicaCon security training experience have

you received?

Key Survey Results

•  Data shows soXware developers posiKvely answer quesKons about applicaKon security 56% of the Kme

•  2013 Denim Group study results: 58% •  2014 Aspect Study: 60%

Change Implementation

Yes 33%

No 25%

I don't know 42%

Did your organizaCon implement any SDLC or process improvement steps to formalize concepts learned in training?

Types of Training Received

0 50 100 150 200 250

Instructor-‐Led PresentaKons

e-‐Learning, CBT

Social Media

Social Learning Plaqorms

Developer E-‐mail Lists or RSS feeds

Crowdsourcing Sites

Websites

Webinars or Videos

1-‐on-‐1 Coaching

Wri[en Materials

Other


E-Learning & Instructor-Led Training

0 100 200 300


e-‐Learning, CBT

Social Media



Crowdsourcing Sites

Websites

Webinars or Videos


Wri[en Materials

Other


E-‐Learning & Instructor-‐led Training are SKll the Primary ApplicaKon Security Training Approach

Perceived Effectiveness of Training

0 50 100 150 200 250 300 350 400 450 500


e-‐Learning, CBT

Social Media



Crowdsourcing Sites

Websites

Webinars or Videos


Wri[en Materials

1: Not EffecKve

2: Somewhat EffecKve

3: Very EffecKve

Question Types

41%

59%

0% 10% 20% 30% 40% 50% 60% 70%

PrescripKve QuesKons

Awareness QuesKons

% of QuesKons Answered Correctly

Respondents Fared Far Worse on QuesKons Involving Secure Coding PracKces versus ApplicaKon Security Awareness QuesKons

Pass Rate by Job Function

Average Pass Rate

0%

5%

10%

15%

20%

25%

Other SoXware Developer Quality Assurance Architect

70% or more quesKons answered correctly

Quality Assurance respondents Fared 50% worse than soXware developers and architects

Pass Rate by Previous Training

Average Pass Rate

0%

5%

10%

15%

20%

25%

30%

Less than a Day or None At least 1 day, but less than 3 days More than 3 days

70% or more correct

The Pass Rate More Than Doubled for Respondents Who Had More Than Three Days ApplicaKon Security Training

Pass Rate by Job Function: Security

Average Pass Rate

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Security-‐Related Everyone Else

70% or more quesKons answered correctly

Respondents that worked for security organizaKons or vendors DID fare well compared to other respondents

What we Can Put to Work

•  Refresher training is criCcal •  Even with 3+ days of appsec training, most

respondents did not have a “passing” grade of 70%

•  Like any other training topic, leX unreinforced, what learned will be forgo[en over Kme •  ParKcularly given the lack of SDLC changes

•  Likely an area for addiKonal study for 2015 appsec training study


•  Training without SDLC changes likely will produce the same results •  33% of the respondents said their organizaKon

implemented some security SDLC improvements •  67% either answered “no” or “don’t know” •  OrganizaKons cannot rely exclusively on

developers retenKon and iniKaKve to produce long-‐term decline in applicaKon vulnerabiliKes


•  Augment QA with Focused AppSec Training •  QA has consistently responded poorly relaKve to

developers and architects •  Many organizaKon put their most junior

developers in QA to start •  QA is where appsec “lives” in many

organizaKons •  OrganizaKons might considering “doubling

down” on appsec training for QA staff to compensate for this fact


•  IncenCves Ma`er When Working with Developers

•  We used incenKves throughout the study to collect responses -‐ #Success!

•  SoXware developers have infinite reasons to ignore engagement by the AppSec team

•  Rewards help nudge soXware developers


•  Training programs must be tailored to be effecCve

•  Formal programs like classroom training and e-‐Learning are sKll the bread and bu[er of appsec training programs

•  ConsumpKon rates of e-‐Learning sKll abysmal without incenKves or internal markeKng

•  Add newer ways of learning to reinforce certain key points and to serve AppSec corner cases

•  Leverage current events to reinforce other key points

Conclusions

•  Data shows soXware developers posiKvely answer quesKons about applicaKon security 56% of the Kme

•  Data-‐driven applicaKon security programs will likely be more successful and chart improvement

•  SophisKcated security managers use incenKves and tailor programs to improve appsec IQ

White Paper? MenCon it on Twi`er

John B. Dickson, CISSP @johnbdickson #appsecstudy

Questions and Answers