ATM: Adaptive Testing Methodology

Daniel MiesslerDirector of Advisory ServicesIOActive

Web hacking in pictures

Image from stopherdingcats.com

Concepts

“ I used to think we had security problems, and then we figured out how to integrate the security solution.

Actually, the security basics are long figured out, it’s the integration that's killing us. We don't have a security problem with integration requirements. We have an integration problem with security requirements.

~ Gunnar Peterson

http://1raindrop.typepad.com/1_raindrop/2013/11/there-are-no-security-problems.html

My take on Gunnar’s thought

1. Security is an integration problem

2. It’s not that we don’t know what to do

3. It’s that we don’t know how to integrate what we know (or learn) into what we do

Security is an integration problem

Two ways to learn: Osmotic vs. Algorithmic

VS

Osmotic learning

1. Consume a talk/book/video about testing SAP2. Don’t fall asleep3. Mostly pay attention to the content4. Say, “hmm…” to yourself 1-3 times5. Maybe jot something down on a piece of paper

you’ll never see again6. Don’t remember any/most of it when you do

the task next

Algorithmic learning

1. You already care about testing SAP a lot2. For this reason, you already have an algorithm for

doing so3. You also like to learn more about it (seminars/etc.)4. When you learn something new, you immediately

update your methodology with anything legit5. The very next time you test SAP, you have

directly benefitted from the talk/video/book you consumed

Algorithmic vs. Osmotic learning

Web methodologies are monolithic

199 Pages

94 Pages

=

Web methodologies lack context

Web methodologies lack empathy

“The customer wants you to find everything you can in 13 minutes.”

Methodologies are hard to update

Review– Security is an integration problem

Review– Security is an integration problem– Algorithmic learning is better for improving methodologies

Review– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic

Review– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic– Methodologies are not context-sensitive

Review– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic– Methodologies are not context-sensitive– Methodologies don’t know how much time you have

Review– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic– Methodologies are not context-sensitive– Methodologies don’t know how much time you have– Methodologies are hard to update

Review (propositions, challenges)– Security is an integration problem– Algorithmic learning is better for improving methodologies– Most web methodologies are monolithic– Methodologies are not context-sensitive– Methodologies don’t know how much time you have– Methodologies are hard to update

Adaptive Testing Methodology (ATM)

Methodology

Methodology (project)

https://github.com/danielmiessler/ATM

Methodology (content)

1. WAHHhttp://www.amazon.com/The-Web-Application-Hackers-Handbook/dp/1118026470

Methodology (content)

1. WAHHhttp://www.amazon.com/The-Web-Application-Hackers-Handbook/dp/1118026470

2. OWASP ASVShttps://www.owasp.org/images/6/67/OWASPApplicationSecurityVerificationStandard3.0.pdf

Methodology (content)

1. WAHHhttp://www.amazon.com/The-Web-Application-Hackers-Handbook/dp/1118026470

2. OWASP ASVShttps://www.owasp.org/images/6/67/OWASPApplicationSecurityVerificationStandard3.0.pdf

3. OWASP Web Testing Guidehttps://www.owasp.org/images/5/52/OWASP_Testing_Guide_v4.pdf

Methodology (content)

1. WAHHhttp://www.amazon.com/The-Web-Application-Hackers-Handbook/dp/1118026470

2. OWASP ASVShttps://www.owasp.org/images/6/67/OWASPApplicationSecurityVerificationStandard3.0.pdf

3. OWASP Web Testing Guidehttps://www.owasp.org/images/5/52/OWASP_Testing_Guide_v4.pdf

4. Jason Haddix’s Bughunter Methodologyhttps://appsecusa2015.sched.org/event/3kXN/the-bug-hunters-methodology

** initial compilation / curation done by me

ATM Concepts

ATM Concepts

– BJJ vs. Praying Mantis (efficacy)

Image by knotlikeyou2 of Deviant Art

ATM Concepts

– BJJ vs. Praying Mantis (efficacy)– Willingness to use other tools

ATM Concepts

– BJJ vs. Praying Mantis (efficacy)– Willingness to use other tools– Heavy focus on OSINT

ATM Concepts

– BJJ vs. Praying Mantis (efficacy)– Willingness to use other tools– Heavy focus on OSINT– Flexibility based on conditions

ATM Concepts

– BJJ vs. Praying Mantis (efficacy)– Willingness to use other tools– Heavy focus on OSINT– Flexibility based on conditions– Transparency

Methodology (structure)

Methodology (structure)

Methodology (structure) [ technology ]

Universal | 30M | 1H | 1D | 2D | UL | Check text here.

Methodology (structure) [ technology ]

Universal | 30M | 1H | 1D | 2D | UL | Check text here.Apache | 30M | 1H | 1D | 2D | UL | Check text here.

Methodology (structure) [ technology ]

Universal | 30M | 1H | 1D | 2D | UL | Check text here.Apache | 30M | 1H | 1D | 2D | UL | Check text here.Wordpress PHP | 30M | 1H | 1D | 2D | UL | Check text here.

Methodology (structure) [ time ]

Universal | 30M | 1H | 1D | 2D | UL | Check text here.

Methodology (structure) [ time ]

Universal | 30M | 1H | 1D | 2D | UL | Check text here.PHP | 30M | 1H | 1D | 2D | UL | Check text here.

Methodology (structure) [ time ]

Universal | 30M | 1H | 1D | 2D | UL | Check text here.PHP | 30M | 1H | 1D | 2D | UL | Check text here.Express | 30M | 1H | 1D | 2D | UL | Check text here.

Execution

Execution

1. Client makes a request to ATM service

Execution

1. Client makes a request to ATM service2. Client sends two (2) things

- DOMAIN- TIME SCOPE

Execution

1. Client makes a request to ATM service2. Client sends two (2) things

- DOMAIN- TIME SCOPE

3. ATM service tests the domain for its stack

Execution

1. Client makes a request to ATM service2. Client sends two (2) things

- DOMAIN- TIME SCOPE

3. ATM service tests the domain for its stack4. ATM service receives stack information

Execution

1. Client makes a request to ATM service2. Client sends two (2) things

- DOMAIN- TIME SCOPE

3. ATM service tests the domain for its stack4. ATM service receives stack information5. ATM service parses the current

methodology for rules that match the stack and time combination given

Execution

1. Client makes a request to ATM service2. Client sends two (2) things

- DOMAIN- TIME SCOPE

3. ATM service tests the domain for its stack4. ATM service receives stack information5. ATM service parses the current

methodology for rules that match the stack and time combination given

6. ATM service returns the custom set of methodology checks to the client

Execution (visual)

SITE

CLIENT ATM

Execution (visual)

SITE

CLIENT ATM(send domain/time)

[1]

Execution (visual)

SITE

CLIENT ATM(send domain/time)

[1][2]

(checks site stack)

Execution (visual)

SITE

CLIENT ATM(send domain/time)

[1][2]

(checks site stack)

[3] ATM parses checks

Execution (visual)

SITE

CLIENT ATM(send domain/time)

[1][2]

(checks site stack)

[3] ATM parses checks

(returns checks to client)

[4]

Demo

- Contextual security testing- Crowdsourced updates via Github- Adjusts to technology stack- Adjusts to your time constraints- Produces customized testing for your app

Next steps

Improve the methodologies (currently alpha, way more to add) Add additional factors (continuous monitoring via proxy logs) Add other types of context (besides stack and time) Add other types of testing (network/forensics/etc) Determine best time increments (community) Improve performance of the stack detection (multithreading) Create this is public service infrastructure that can be used with

various methodologies and clients Determine if I should do anything with the domains (stack-

check.com / adaptivetestingmethodology.com) (meh) Explore local implementations (non-service-based) for product

integrations

Announcement: Portswigger and ATM

Coming to Burpsuite Soon!

** Ask me about RobotsDisallowed and SecLists integration as well

Thanks

Daffyd Stuttard and the WAHH OWASP for ASVS and Web Testing Methodology Jason Haddix for the Bug Hunter’s Methodology Nestor Mata Cuthbert for help with Wordpress IOActive for being a phenomenal place to work

- [ PROJECT ] github.com/danielmiessler/ATM- [ SERVICE ] danielmiessler.com/services/atm

- [ TWITTER ] twitter.com/danielmiessler- [ MAIL ] daniel@danielmiessler.com- [ MAIL ] daniel.miessler@ioactive.com