Embed Size (px)
DESCRIPTION
Presentation by Yvette du Toit at ISSA in 2011. This presentation is about application assessment metrics and their challenges. Examples of Sensepost metrics are given.
Citation preview
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
PRESENTED BY: Yvette du Toit
Agenda!
• Background!• Approach!• Examples!• Challenges with Application Security Metrics!• Q&A!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
Background!
• As Security Consultants we write reports!– Test, analyse, write up findings, submit to client!
• Issues still remain open – why?!– Reports not say enough!– Question value report offer!
• Solution – metrics / visualisation!– Graphs, colour, size etc!
• First – letʼs take a look at what reports say…!– Qualitative ratings!– Best practice!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
What do Reports Say?!
• 2007 - 2011!• Many words….!• Content (Exec Summary, Technical Summary, Conclusion)!• Are actions effective?!• What would be more valuable – comparison (time & peers)!• How do we use metrics?!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
Pages Words Assessments 638 224587 Re-Tests 137 28164 Total 775 252751
Approach!
• Metrics – definition!– Quantifiable!– Characteristics!
• 3 Metric Veterans:!– Jacquith - “those that support decision making about risk for the
purpose of managing that risk” !– Marty – “a picture paints a thousand log records”!– Godin: “just because something is easy to measure doesnʼt mean
itʼs important”!• NB: To measure what is important & that will yield “useful”
information!– Examples of metrics not necessarily useful!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!
• Definition!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
Useful?!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!
• Example!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
• Metrics can be misleading!
Useful?!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!
• Example!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
• Metrics are not always 100% useful!
Approach!
• Why?: illustrate useful information!– Recurring issues!– Time required to compromise!– Top 10 list!– Effectiveness of remediation!– Benchmarking!
• Who? 7 organisations in financial sector!• When? 3 ½ years!• How? Data capture process!
– Marco Slaviero (Head of R&D)!– Spreadsheet for data capture!– Report meta-data (project length, frameworks, dates etc.)!– Findings categorised (pre-defined list of vulns)!– Findings ranked (Impact, EoE, Threat metric)!
• Normalisation !– Allows for comparison across time and peers !
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!
• Introduction!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
Annual Distribution of Project (Days)!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
SensePost Metrics Proposal!
• Metrics extracted from report data:!– Timelines (plotting projects on timeline)!– Basic counts and statistics (uncover counts)!
• Number of projects!• Number of days!• Number of words and pages in report!
– Threat metrics (Findings per threat level)!– Bug class metrics (Findings across categories) !– Top 10 list !– Re-Test Metrics!– Benchmarks (comparison to peers)!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!
• Our Metrics!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
SensePost Metrics in Action: Timelines!
• Useful?!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
! "#$%&! '()*&!
!"#$%&'()'*++%++#%,-+' ./0' 112304'
!"#$%&'()'5%67%+-+' 8/4' 108.2'
7(-9:' 443' 131438'
SensePost Metrics in Action: Threat Metrics!
• Useful?!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
SensePost Metrics in Action: Bug Classes!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Useful?!• See 56% of findings occur in Top 11 bug classes!• 2008 Anomaly (No Re-Tests) !
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
SensePost Metrics in Action: Top 10!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Useful? !• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
SensePost Metrics in Action: Re-Test!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Useful?!• 29% Critical and 42% High-risk issues remain open !
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
SensePost Metrics in Action: Benchmarks!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Useful?!• Our client positioned 3rd (not highlighted here)!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
Challenges!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
• Bug counts vs bug classes!– Bug counts – number of findings!– Bug classes – categories!– 2 applications scenario (10 findings 1 bug class vs 1 finding in 10 bug classes)!
• Depth vs breadth!– Each occurrence – depth!– Each bug class - breadth!
Q&A!
ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
Agenda!
• Background!• Approach!• Examples!• Challenges with
Application Security Metrics!
• Q&A!
• Thank you!• Longer paper – mail me!• Email: [email protected]!• Contact: +27 79 509 8913!
