Upload
codenomicon
View
446
Download
1
Embed Size (px)
Citation preview
WWW.CODENOMICON.COM2
Fuzzing 101:
• The webcast series for fuzzing industry
• Vendor neutral presentations on fuzzing technologies and use-cases
• Includes invited speakers from the industry
Codenomicon:
• Fuzzing research since 1996
• 2001, Spinoff from University of Oulu
• 50-100% annual growth in number of customers and revenues in fuzzing industry
About Fuzzing 101 and Codenomicon
WWW.CODENOMICON.COM3
About Ari Takanen
The Past: Researcher and Lecturer• 1998-2002• University of Oulu• OUSPG/PROTOS research group• Software Quality related lectures
The Present: Entrepreneur and Evangelist• 2001-today• CTO of Codenomicon• Evangelist: 10 conference talks every year • Author of two books:
• VoIP Security• Fuzzing
WWW.CODENOMICON.COM4
Agenda
Intro: • Fuzzing Overview• Fuzzing Performance
Demo in Theory• Multi-threaded execution• Test execution optimization• Command-line use
And if time allows: Demo in Practice for the interested
WWW.CODENOMICON.COM6
What is Testing - Three Flavors
Feature/Conformance tests validate positive requirements with use cases and test cases
Performance/Load testing repeats one of the feature tests to validate performance limits
Robustness/Fuzz testing uses misuse cases to verify software reliability with unexpected inputs
WWW.CODENOMICON.COM10
Some Helpful Definitions
Vulnerability – a weakness in software, a bug
Threat/Attack – exploit/worm/virus against a specific vulnerability
Protocol Modeling – Technique for explaining interface message sequences and message structures
Fuzzing – process and technique for security testing
Anomaly – abnormal or unexpected input
Failure – crash, busy-loop, memory corruption, or other indication of a bug in software
WWW.CODENOMICON.COM11
Fuzzing In Short
Fuzzing means crash-testing Also called:
• Negative testing• Robustness testing• Grammar testing
Based on sending systematically broken (rarely random) inputs to a software, in order to crash it
We will ignore random mutator fuzzers for now Two techniques of smart model-based fuzzers:
• Template-based• Specification-based
WWW.CODENOMICON.COM12
Model Based Fuzzing Techniques
Template Based Fuzzing• Quality of tests is based on the used seed and
modeling technique• Very quick to develop, but slow to run• Editing requires deep protocol know-how• Good for testing around known vulnerabilities
Specification Based Fuzzing• Full test coverage• Always repeatable• Short test cycle, more optimized tests• Easy to edit and add tests
WWW.CODENOMICON.COM13
Precision is about attack surface/protocol coverage
All interfaces/protocols tested?
All message sequences tested?
All message structures tested?
All data definitions tested?
All “tags” (values) tested?
Accuracy is about anomaly coverage
Anomaly categories? SQL? Buffer overflow?
All values: 0..65k, a..z, 0x00..0x255 ?
Combinations of anomalies?
Coverage
WWW.CODENOMICON.COM17
Key Questions in regards to Fuzzing
Test Coverage:• Q: Which interfaces and protocols are tested?• A: Understanding of threat models and attack surface
are the most critical starting point to fuzzing
• Q: How well something is tested?• A1: Specification coverage explains how deep
knowledge the fuzzer has on a tested interface, and is typically represented by protocol models or protocol templates
• A2: Anomaly coverage explains how well each protocol element is tested, and is the hardest metric to measure
WWW.CODENOMICON.COM
Fuzzing Scalability
Software-based Fuzzers scale for all testing needs• The throughput depends only on the available hardware• The entire Network Under Test (NUT) can be virtualized• Software fuzzers can attack every interface, and against all
protocol layers• XML• HTTP• TLS• TCP/IP
WWW.CODENOMICON.COM20
Why Fast Speed Fuzzing?
“One of the most important aspects of fuzzing is how fast you can execute test cases. The faster you can execute test cases, the more test cases you can run and the more vulnerabilities you will find.” - Dr. Charlie Miller from Independent Security Evaluators. "
WWW.CODENOMICON.COM
Generating Load with Defensics
Full model-based message sequences
Options for monitoring, instrumentation, fuzzing, …
Amount of available CPUs and Logging Level impact performance
Some speed records (test cases per second):
• TLS: 2.400 tc/s• IPv6: 4.500 tc/s• HTTP: 16.000 tc/s• DNS: 41.000 tc/s (with capture replay)
21
WWW.CODENOMICON.COM22
Motivation for our Performance Study
Test our test generation efficiency for different type of protocol fuzzers using high-end servers
Analyze benefits (test efficiency, ROI) of software-based fuzzers to appliances
WWW.CODENOMICON.COM23
Raw Fuzzing Performance Metrics
Packets or bytes per second does not really work as each fuzz test case can consist of several packets back and forth
Test cases, or full protocol transactions, generated by each suite provides more insight
Performance is generated by having fuzzers running in parallel
These result in “test cases per second” or tc/s
WWW.CODENOMICON.COM24
Defensics Benefits
No hardware constraints
No operating system constraints
Parallel processes and threads
WWW.CODENOMICON.COM25
Performance Bottlenecks and Metrics
Network bandwidth: • Can be measured with number of full protocol
transactions, packets, and data volume CPU usage by different components of
Defensics• Test generator(s)• Test driver(s)
Memory usage of Defensics components
Most interesting metric: • New type of bugs related to performance?
WWW.CODENOMICON.COM29
Test Plans Using Command-Line
With some fuzzers, you can run the tools faster on command-line
With Defensics, the GUI is sometimes even faster
E.g. with a laptop with dual-core Intel T7500 processor (with default tool settings):
• 10.000 x valid-case in loop• using GUI: 30 seconds• using command-line: 35 seconds
WWW.CODENOMICON.COM30
Running Fuzzers in Parallel
Defensics GUI has options to run several suites in parallel
• 2 x 5.000 x valid-case in loop• using GUI: 22 seconds (about 50-100% faster)
WWW.CODENOMICON.COM31
Using Multi-threaded Mode
When CPU resources are available, multi-treaded mode is often best way to speed up testing
Uses less memory
Note that all parallel execution creates more false positives
Let’s look at that and other speed optimization hints now in live demo...
WWW.CODENOMICON.COM33
Performance Value
Users can now…• Increase test coverage through parallel tests• Speed up testing• Find performance bottle-necks• Integrate Defensics also in other functional tests
Remember that Fuzzing ROI is measured in: • TCO vs. Value of Security Findings / Cost of Security
Compromise• Reduced maintenance costs, failure reproduction costs• Better customer confidence, higher retention rate of
customers• In performance: software-based tools can be significantly
cheaper (and faster) than appliances
WWW.CODENOMICON.COM34
More News from Codenomicon
Facebook:• Become fan of Codenomicon and Fuzzing
Twitter:• CodenomiconLTD
Codenomicon Website:• Newsletter every second month
Customer portal• The Backstage! Contact us for access!
34
WWW.CODENOMICON.COM35
Our Book On Fuzzing!
http://www.fuzz-test.com/book/
Takanen, DeMott and Miller: “Fuzzing for Software Security Testing and Quality Assurance”
Aimed at the general public, you do not need to be a security specialist to read this book
Purpose of the book is to teach next-gen testing approaches to:
• Software practitioners• Security engineers• Academics
WWW.CODENOMICON.COM
PROACTIVE SECURITY AND ROBUSTNESS SOLUTIONS
THANK YOU – QUESTIONS?
“Thrill to the excitement of the chase! Stalk bugs with care, methodology, and reason. Build traps for them.
....Testers!
Break that software (as you must) anddrive it to the ultimate
- but don’t enjoy the programmer’s pain.”
[from Boris Beizer]