ATM: Adaptive Testing Methodology
Daniel MiesslerDirector of Advisory ServicesIOActive
Web hacking in pictures
Image from stopherdingcats.com
Concepts
“ I used to think we had security problems, and then we figured out how to integrate the security solution.
Actually, the security basics are long figured out, it’s the integration that's killing us. We don't have a security problem with integration requirements. We have an integration problem with security requirements.
~ Gunnar Peterson
http://1raindrop.typepad.com/1_raindrop/2013/11/there-are-no-security-problems.html
My take on Gunnar’s thought
1. Security is an integration problem
2. It’s not that we don’t know what to do
3. It’s that we don’t know how to integrate what we know (or learn) into what we do
Security is an integration problem
Two ways to learn: Osmotic vs. Algorithmic
VS
Osmotic learning
1. Consume a talk/book/video about testing SAP2. Don’t fall asleep3. Mostly pay attention to the content4. Say, “hmm…” to yourself 1-3 times5. Maybe jot something down on a piece of paper
you’ll never see again6. Don’t remember any/most of it when you do
the task next
Algorithmic learning
1. You already care about testing SAP a lot2. For this reason, you already have an algorithm for
doing so3. You also like to learn more about it (seminars/etc.)4. When you learn something new, you immediately
update your methodology with anything legit5. The very next time you test SAP, you have
directly benefitted from the talk/video/book you consumed
Algorithmic vs. Osmotic learning
Web methodologies are monolithic
199 Pages
94 Pages
=
Web methodologies lack context
Web methodologies lack empathy
“The customer wants you to find everything you can in 13 minutes.”
Methodologies are hard to update
Review– Security is an integration problem
Review– Security is an integration problem– Algorithmic learning is better for improving methodologies
Review– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic
Review– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic– Methodologies are not context-sensitive
Review– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic– Methodologies are not context-sensitive– Methodologies don’t know how much time you have
Review– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic– Methodologies are not context-sensitive– Methodologies don’t know how much time you have– Methodologies are hard to update
Review (propositions, challenges)– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic– Methodologies are not context-sensitive– Methodologies don’t know how much time you have– Methodologies are hard to update
Adaptive Testing Methodology (ATM)
Methodology
Methodology (project)
https://github.com/danielmiessler/ATM
Methodology (content)
1. WAHHhttp://www.amazon.com/The-Web-Application-Hackers-Handbook/dp/1118026470
Methodology (content)
1. WAHHhttp://www.amazon.com/The-Web-Application-Hackers-Handbook/dp/1118026470
2. OWASP ASVShttps://www.owasp.org/images/6/67/OWASPApplicationSecurityVerificationStandard3.0.pdf
Methodology (content)
1. WAHHhttp://www.amazon.com/The-Web-Application-Hackers-Handbook/dp/1118026470
2. OWASP ASVShttps://www.owasp.org/images/6/67/OWASPApplicationSecurityVerificationStandard3.0.pdf
3. OWASP Web Testing Guidehttps://www.owasp.org/images/5/52/OWASP_Testing_Guide_v4.pdf
Methodology (content)
1. WAHHhttp://www.amazon.com/The-Web-Application-Hackers-Handbook/dp/1118026470
2. OWASP ASVShttps://www.owasp.org/images/6/67/OWASPApplicationSecurityVerificationStandard3.0.pdf
3. OWASP Web Testing Guidehttps://www.owasp.org/images/5/52/OWASP_Testing_Guide_v4.pdf
4. Jason Haddix’s Bughunter Methodologyhttps://appsecusa2015.sched.org/event/3kXN/the-bug-hunters-methodology
** initial compilation / curation done by me
ATM Concepts
ATM Concepts
– BJJ vs. Praying Mantis (efficacy)
Image by knotlikeyou2 of Deviant Art
ATM Concepts
– BJJ vs. Praying Mantis (efficacy)– Willingness to use other tools
ATM Concepts
– BJJ vs. Praying Mantis (efficacy)– Willingness to use other tools– Heavy focus on OSINT
ATM Concepts
– BJJ vs. Praying Mantis (efficacy)– Willingness to use other tools– Heavy focus on OSINT– Flexibility based on conditions
ATM Concepts
– BJJ vs. Praying Mantis (efficacy)– Willingness to use other tools– Heavy focus on OSINT– Flexibility based on conditions– Transparency
Methodology (structure)
Methodology (structure)
Methodology (structure) [ technology ]
Universal | 30M | 1H | 1D | 2D | UL | Check text here.
Methodology (structure) [ technology ]
Universal | 30M | 1H | 1D | 2D | UL | Check text here.Apache | 30M | 1H | 1D | 2D | UL | Check text here.
Methodology (structure) [ technology ]
Universal | 30M | 1H | 1D | 2D | UL | Check text here.Apache | 30M | 1H | 1D | 2D | UL | Check text here.Wordpress PHP | 30M | 1H | 1D | 2D | UL | Check text here.
Methodology (structure) [ time ]
Universal | 30M | 1H | 1D | 2D | UL | Check text here.
Methodology (structure) [ time ]
Universal | 30M | 1H | 1D | 2D | UL | Check text here.PHP | 30M | 1H | 1D | 2D | UL | Check text here.
Methodology (structure) [ time ]
Universal | 30M | 1H | 1D | 2D | UL | Check text here.PHP | 30M | 1H | 1D | 2D | UL | Check text here.Express | 30M | 1H | 1D | 2D | UL | Check text here.
Execution
Execution
1. Client makes a request to ATM service
Execution
1. Client makes a request to ATM service2. Client sends two (2) things
- DOMAIN- TIME SCOPE
Execution
1. Client makes a request to ATM service2. Client sends two (2) things
- DOMAIN- TIME SCOPE
3. ATM service tests the domain for its stack
Execution
1. Client makes a request to ATM service2. Client sends two (2) things
- DOMAIN- TIME SCOPE
3. ATM service tests the domain for its stack4. ATM service receives stack information
Execution
1. Client makes a request to ATM service2. Client sends two (2) things
- DOMAIN- TIME SCOPE
3. ATM service tests the domain for its stack4. ATM service receives stack information5. ATM service parses the current
methodology for rules that match the stack and time combination given
Execution
1. Client makes a request to ATM service2. Client sends two (2) things
- DOMAIN- TIME SCOPE
3. ATM service tests the domain for its stack4. ATM service receives stack information5. ATM service parses the current
methodology for rules that match the stack and time combination given
6. ATM service returns the custom set of methodology checks to the client
Execution (visual)
SITE
CLIENT ATM
Execution (visual)
SITE
CLIENT ATM(send domain/time)
[1]
Execution (visual)
SITE
CLIENT ATM(send domain/time)
[1][2]
(checks site stack)
Execution (visual)
SITE
CLIENT ATM(send domain/time)
[1][2]
(checks site stack)
[3] ATM parses checks
Execution (visual)
SITE
CLIENT ATM(send domain/time)
[1][2]
(checks site stack)
[3] ATM parses checks
(returns checks to client)
[4]
Demo
- Contextual security testing- Crowdsourced updates via Github- Adjusts to technology stack- Adjusts to your time constraints- Produces customized testing for your app
Next steps
Improve the methodologies (currently alpha, way more to add) Add additional factors (continuous monitoring via proxy logs) Add other types of context (besides stack and time) Add other types of testing (network/forensics/etc) Determine best time increments (community) Improve performance of the stack detection (multithreading) Create this is public service infrastructure that can be used with
various methodologies and clients Determine if I should do anything with the domains (stack-
check.com / adaptivetestingmethodology.com) (meh) Explore local implementations (non-service-based) for product
integrations
Announcement: Portswigger and ATM
Coming to Burpsuite Soon!
** Ask me about RobotsDisallowed and SecLists integration as well
Thanks
Daffyd Stuttard and the WAHH OWASP for ASVS and Web Testing Methodology Jason Haddix for the Bug Hunter’s Methodology Nestor Mata Cuthbert for help with Wordpress IOActive for being a phenomenal place to work
- [ PROJECT ] github.com/danielmiessler/ATM- [ SERVICE ] danielmiessler.com/services/atm
- [ TWITTER ] twitter.com/danielmiessler- [ MAIL ] [email protected] [ MAIL ] [email protected]