Embed Size (px)
DESCRIPTION
Measuring the effectiveness of any security activity is widely discussed – security leaders debate the topic with a religious fervor rivaling that of any other hot button issue. Virtually every organization has some sort of application security training effort, but data on training effectiveness remains scarce. Last year our research team delivered the first-ever survey that captured developer awareness of secure coding concepts and the impact of formal application security training on a developer’s ability to write secure code. We learned that most software developer were aware of certain application security concepts, yet when asked how to write more secure code, they faired poorly. This year’s 600-developer survey provides more quantitative data on what software developers understand about application security, both concepts and practices. It dives most deeply into awareness of defensive coding practices, which most developers largely did not grasp in the 2013 survey. It also is separates respondents by roles, so we can better understand how architects, developers, and QA staff grasp key application security concepts and put them to work. It better captures how software developers learn in general, so one can tailor any security training effort to how software developers, in practice, actually learn. This information will provide data to application security managers responsible for corporate security training that should allow them them to make more fact-based decisions about security training.
Citation preview
AppSec USA 2014 Denver, Colorado
AppSec Survey 2.0: Fine-Tuning an AppSec Training Program Based on
Data
John B. Dickson, CISSP @johnbdickson
September 18, 2014
John B. Dickson, CISSP
• Application Security Enthusiast • Ex-AF Guy & ISSA Distinguished Fellow
• Serial Entrepreneur & MBA Type
• Dad
Introduction
When Not Thinking about AppSec…
I am Snake Hunting on a Ranch in South Texas
Snake Hunting Essentials
Cooler Hat Cool Hat
Snake Guards Common Gardening Tools Machete
Guy who has a machete and who is actually good at “catching” snakes
OWASP AppSec 2011 t-‐shirt
© Copyright 2014 Denim Group - All Rights Reserved
• Background • Premise • AppSec Study 1.0 Results – What We Learned • Approach and Survey ParKcipants • Key Results • What We Can Put To Work • Conclusions and QuesKons & Answers
Overview
• Things we Knew Last Year
• Key Findings of Last Year’s Study
• AddiKonal Stuff We Learned Along the Way
• Development training is hard
• Results are rarely measured for ROI
• Training is typically part of any AppSec program
AppSec Study 1.0 Results
• Things we Knew Last Year
• Key Findings of Last Year’s Study
• AddiKonal Stuff We Learned Long the Way
• 25% retenKon aXer training
• QA did worse than architects and soXware developers
• Respondents answered basic awareness quesKons but not coding pracKces
AppSec Study 1.0 Results
• Things we Knew Last Year
• Key Findings of Last Year’s Study
• AddiConal Stuff We Learned Long the Way
• SoXware developers learn differently than companies teach
• IncenKves ma[er • Surveys are hard!
AppSec Study 1.0 Results
Overview of 2014 “2.0” Study
• 600 respondents • Represents mulKple industries • Asked the same applicaKon security quesKons as
2013 survey • Expanded to include training method quesKons • No “before” and “aXer” analysis • No classroom training opportuniKes • Used more social media • Data collecKon ongoing
Approach and Survey Participants
Sample QuesCons
QuesKons that tested basic knowledge of applicaKon security:
• ApplicaKon security is best defined as… • Threat Modeling is… • Input ValidaKon is…
Approach and Survey Participants
Sample QuesCons
QuesKons that tested understanding of defensive coding:
• Marking a cookie as “secure” will… • Which of the following will help protect against XSS…
• Which of the following is NOT an example of good session policy…
Delivery Means • Direct Delivery of Customized Links via E-‐mail
• Survey Monkey paid • Social Media
– Facebook – Linkedin
Targets • SoXware Developers • Architects • Quality Assurance
Approach and Survey Participants
Demographic Questions Asked
• What is your primary job funcKon?
• What is your company's size?
• How many years of soXware development experience do you have?
• How much previous applicaKon security training have you received?
2014 Study Demographics
Less than a Year 18%
1-‐2 Years 9%
2-‐4 Years 10%
4-‐7 Years 13%
7-‐12 Years 16%
More than 12 Years 34%
How many years of soMware development experience do you
have?
2014 Study Demographics
Other 35%
SoXware Developer
53%
Quality Assurance
6%
Architect 6%
What is your primary job funcCon?
2014 Study Demographics
8% 8%
29%
8% 10%
37%
What is your company size?
1-‐24 Employees
25-‐99 Employees
100-‐499 Employees
500-‐2499 Employees
2500-‐9999 Employees
10,000 or more Employees
2014 Study Demographics
None 31%
Less than a Day 19%
At least 1 day, but less than 2 days
17%
At least 2 days, but less than 3 days
8%
More than 3 days 25%
How much previous applicaCon security training experience have
you received?
Key Survey Results
• Data shows soXware developers posiKvely answer quesKons about applicaKon security 56% of the Kme
• 2013 Denim Group study results: 58% • 2014 Aspect Study: 60%
Change Implementation
Yes 33%
No 25%
I don't know 42%
Did your organizaCon implement any SDLC or process improvement steps to formalize concepts learned in training?
Types of Training Received
0 50 100 150 200 250
Instructor-‐Led PresentaKons
e-‐Learning, CBT
Social Media
Social Learning Plaqorms
Developer E-‐mail Lists or RSS feeds
Crowdsourcing Sites
Websites
Webinars or Videos
1-‐on-‐1 Coaching
Wri[en Materials
Other
Types of Training Received
E-Learning & Instructor-Led Training
0 100 200 300
Instructor-‐Led PresentaKons
e-‐Learning, CBT
Social Media
Social Learning Plaqorms
Developer E-‐mail Lists or RSS feeds
Crowdsourcing Sites
Websites
Webinars or Videos
1-‐on-‐1 Coaching
Wri[en Materials
Other
Types of Training Received
E-‐Learning & Instructor-‐led Training are SKll the Primary ApplicaKon Security Training Approach
Perceived Effectiveness of Training
0 50 100 150 200 250 300 350 400 450 500
Instructor-‐Led PresentaKons
e-‐Learning, CBT
Social Media
Social Learning Plaqorms
Developer E-‐mail Lists or RSS feeds
Crowdsourcing Sites
Websites
Webinars or Videos
1-‐on-‐1 Coaching
Wri[en Materials
1: Not EffecKve
2: Somewhat EffecKve
3: Very EffecKve
Question Types
41%
59%
0% 10% 20% 30% 40% 50% 60% 70%
PrescripKve QuesKons
Awareness QuesKons
% of QuesKons Answered Correctly
Respondents Fared Far Worse on QuesKons Involving Secure Coding PracKces versus ApplicaKon Security Awareness QuesKons
Pass Rate by Job Function
Average Pass Rate
0%
5%
10%
15%
20%
25%
Other SoXware Developer Quality Assurance Architect
70% or more quesKons answered correctly
Quality Assurance respondents Fared 50% worse than soXware developers and architects
Pass Rate by Previous Training
Average Pass Rate
0%
5%
10%
15%
20%
25%
30%
Less than a Day or None At least 1 day, but less than 3 days More than 3 days
70% or more correct
The Pass Rate More Than Doubled for Respondents Who Had More Than Three Days ApplicaKon Security Training
Pass Rate by Job Function: Security
Average Pass Rate
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Security-‐Related Everyone Else
70% or more quesKons answered correctly
Respondents that worked for security organizaKons or vendors DID fare well compared to other respondents
What we Can Put to Work
• Refresher training is criCcal • Even with 3+ days of appsec training, most
respondents did not have a “passing” grade of 70%
• Like any other training topic, leX unreinforced, what learned will be forgo[en over Kme • ParKcularly given the lack of SDLC changes
• Likely an area for addiKonal study for 2015 appsec training study
What we Can Put to Work
• Training without SDLC changes likely will produce the same results • 33% of the respondents said their organizaKon
implemented some security SDLC improvements • 67% either answered “no” or “don’t know” • OrganizaKons cannot rely exclusively on
developers retenKon and iniKaKve to produce long-‐term decline in applicaKon vulnerabiliKes
What we Can Put to Work
• Augment QA with Focused AppSec Training • QA has consistently responded poorly relaKve to
developers and architects • Many organizaKon put their most junior
developers in QA to start • QA is where appsec “lives” in many
organizaKons • OrganizaKons might considering “doubling
down” on appsec training for QA staff to compensate for this fact
What we Can Put to Work
• IncenCves Ma`er When Working with Developers
• We used incenKves throughout the study to collect responses -‐ #Success!
• SoXware developers have infinite reasons to ignore engagement by the AppSec team
• Rewards help nudge soXware developers
What we Can Put to Work
• Training programs must be tailored to be effecCve
• Formal programs like classroom training and e-‐Learning are sKll the bread and bu[er of appsec training programs
• ConsumpKon rates of e-‐Learning sKll abysmal without incenKves or internal markeKng
• Add newer ways of learning to reinforce certain key points and to serve AppSec corner cases
• Leverage current events to reinforce other key points
Conclusions
• Data shows soXware developers posiKvely answer quesKons about applicaKon security 56% of the Kme
• Data-‐driven applicaKon security programs will likely be more successful and chart improvement
• SophisKcated security managers use incenKves and tailor programs to improve appsec IQ
White Paper? MenCon it on Twi`er
John B. Dickson, CISSP @johnbdickson #appsecstudy
Questions and Answers
