OWASP AppSec 2004 Presentation

Copyright © 2005 - The OWASP FoundationPermission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License.

The OWASP Foundation

OWASP

AppSec

DCOctober

2005http://www.owasp.org/

::Software Security Quality::

Testing Taxonomy &Testing Tools Classification

Arian J. Evans, OWASP Tools ProjectRandom Security Title, FishNet [email protected] [mobile]

2OWASP AppSec DC 2005

OWASP AppSec DC \NIST 2005


Testing Taxonomy &Testing Tools Classification

Subtitle: What is Software Security, how do we evaluate it, and what do we

use to evaluate it?


--OR--


Why does every web-based Document Management Solution Suck so Badly?


Testing Taxonomy & Tools Project

This project, like many other OWASP projects (e.g.-WebScarab, .NET Tools), is:

Performed mostly nights and weekends A personal labor of love (or dislike of FUD) Performed by a team of one As objective and vendor-agnostic as possible Owes thanks to FishNet for letting me use their lab and

encouraging vendors to participate Results are free and worth every penny So don’t complain too much about the results. Emphasis has been on “web app scanners” since that’s

where the predominant market interest seems to lie right now.


Who

Arian J. EvansTester, Teacher, Researcher, Decoder

Application Security ServicesFishNet Security

FishNet SecuritySecurity Consulting Company focused only on

InfosecWork in AppSec Group

They pay me to test and help people fix things(blah, blah)


Outline:

I. Introduction (done)II. Chapter One: In the Beginning was FUD.

1) Problems2) About the OWASP Testing Taxonomy & Tools Project

III. Chapter Two: What is Application Security?1) Distinctions2) Definition

IV. Chapter Three: Clarification and Goals

V. Chapter Four: Taxonomy of Testing

VI. Chapter Five: Automation Tools

VII. Chapter Seven: Q&A (Shoot the Messenger)


Chapter One:

Problems with understanding application security, vendor

FUD, appsec testing tools, and testing techniques.

In the Beginning was the FUD


Webappsec Vendor Marketing Hype

I. The Problem of Particulars− XSS, XST, XDS, XDF, XBS: why particulars can hurt

more than they help (reference: Issues slide)− Selling managers/auditors bullet-point friendly

products− Classification of Threat/Risk/Issue: Lack of distinction

between Category/Class/Particular—Does anyone take formal logic classes anymore?

II. Immature Metrics− What is *really* being exploited? (This is not what

the OWASP Top-10 or WASC taxonomy addresses.)− *What* exploits result in real loss? (Not what the

threads on webappsec@sf are talking about.)− No reward in securing without demonstrable risk.


The Problems:

Different ways to “test” applications for security quality.

Different tools to “test” applications for security quality. No categorization of differences or metrics to measure

pluses and minuses of testing techniques/tools. No single source of information on what tools are

available for testing applications No independent source of information on testing tools

that are available. Some tool vendors do a terrible job of educating the

consumer on what they do and don’t do, and how they relate to their competition.


The Problems:

Unclear and immature terminology Few Taxonomies, Categorizations, Metrics

and Models on AppSec No Business Rule/Risk Taxonomy No mature Threat Models/Taxonomy No accurate Attack Models/Taxonomy Other Taxonomies needed We need a Taxonomy of Taxonomies


This Presentation is About:

The OWASP Tools Project Version 1.2 which aims to:

Categorize the different ways to “test” an application for security.

Categorize the variety of tools that may be used “test” applications for security.

Provide basic insight into features and functionality of “software security tools”.

Offset the active dissemination of misinformation about what tools can and cannot provide.

I bit off more than I can chew here (why are there so many tools out there for this this???)


This Project is Headed:

The OWASP Tools Project Version 2.0 which aims to: Categorize & Classify tools that may be used “test”

applications for security. Work with groups like OWASP, WASC, NIST, and Mitre/CVE Evaluate the value of & possibly provide checklist charts

of features and functionality in specific versions of appsec tools.

I have concerns about above because for example not all XSS checks are equal, and the quality can and does change overnight in current tools.

In non-COTS software metrics tend to lack pragmatic value as software continues to increase size, complexity, extensibility, and develop emergent behaviors as disparate technologies are bolted together and new software evolves.


Limited Information; Full-on Equivocation

I. Things Commercial Tool Vendors like to say:− Sanctum: “Oh, those are *network* scanners”− Compuware: “the only ones in the world who do

this”− Cenzic: “We’ve automated webapp pen testing”− Several: “Oh, no, that’s a CVE scanner. It

doesn’t find anything on our sample application which has “real” vulnerabilities. See here, we’ll just install our sample application we wrote right here and show you…”

− Tautology: The Scanner and Firewall VendorsII. Goings-on in the AppSec Industry:

− No Standards or Standards-bodies− No Metrics− No Watchdogs− Vendors provide the vast bulk of “Factual

Literature”


Chapter II: The Definition

What is Application Security?


Why do you care about AppSec Testing Tools?

When will the tools be mature enough that I just click

“Scan” and get my report?

Why and What is Application Security?:


Why do you care about AppSec Tools?

Because you lack expertiseBecause you lack subject information.Because you lack resources.Because you lack time.Because of the desire for automation.Because of the need for reliable,

repeatable tests that produce accurate quantifiable information about software defects with security implications.


Never.

There are things that we can automate and things that we cannot automate.

There are issues that are easy to automate reliable detection and issues that are not:

- business logic flaws- workflow bypasses- human eyeball parsable errors.

Terminology: Functional versus Technical Issues

When can I click ‘Scan’?


There are two main types of application security issues:

I. Technical Vulnerabilities1) Lack of Input Validation, resulting in:2) XSS3) SQL Injection4) Parameter Tampering, etc.

Functional versus Technical Issues

II. Functional or Logical Vulnerabilities1) Click two or more buttons at the same time

results in unexpected behavior.2) Password change form disclosing detailed

errors.3) Session-idle deconstruction not consistent with

Policies4) Spend deposit before deposit funds are

validated (B/L)


These two types of security issues must be tested differently:

I. Technical Vulnerabilities− Usually about Data Handling− Dealing with Strong Typing, Encoding, and

Validation of Data that the application handles− Can be tested fairly effectively by automated

tools (at least, in theory, as tools mature)

Functional versus Technical Issues

II. Functional or Logical Vulnerabilities− Deals with issues allowed by design, but not

foreseen by the designers (or understood to be a risk)

− Deals with eyeball issues—does the error message text provide a brain-decipherable “secret message”?


Restated:

I. Automation Tools− Testing: useful; use daily− Protection: maybe useful, maybe not (is your

issue XSS? Do you know what your issues are?)− Anecdote of client who solved 90% of their

issues even though they didn’t know what they were.

− Anecdote of WAF vendor who “successfully deployed in under an hour” for the java applet communicating on TCP/5000 with an encrypted binary protocol.

Two Items to Reiterate

II. Examples of Vendor Challenges− Sanctum Support Issues: XSS vs. Phishing

debate− Other vendors “Phishing” “XSS” “blahfoo”

feature FUD


What is “Application Security”?

I. Things that Application Security is not:− It is not a product− It is not a firewall with “AI” (app

intelligence)− It is not a NIDS/NIPS with app

signatures− It is not any “perimeter access

control”.− It is not a “web app firewall” widget.− For today and tomorrow’s web

applications the concept of a perimeter is dead.



II. Things that Application Security is not:− It is not a process (it is possible,

however unlikely, to have secure applications with or without a process)

− It is not “application hacking school” or “secure coding school” for your developers

− It is not the BPPT (Big Production Penetration Test) you have at the end of your BUFD app lifecycle.



III. Things that Application Security IS:− It is one possible emergent Quality of

a well designed application.− It is about predictability,

dependability, reliability. (Business speak !=“security”)

− Rob’s Report must Run Right (bugs are not difference of kind but difference of degree and the issue is always the same)


How do we evaluate applications for Security?

First and most important application security principle:

The Application Must Defend Itself


Presentation Outline Updated

I. Introduction (done)

II. Chapter One: The Problem (done)

III. Chapter Two: The Definition (done)

IV. Chapter Three: The Clarification and Goals

V. Chapter Four: Taxonomy of Testing

VI. Chapter Five: Tools Tools Tools of Automation

VII. Chapter Six: Q&A (Shoot the Messenger)


Chapter III

Clarification and Goals


The Categorization Basics:

Software flaws with Security Implications

Category Class Particular

-Example-

1. Category (Coding Error)2. Class (Output Encoding)3. Particular (XSS (Cross-Site Scripting))


Are we trying to Categorize and Classify?

Clarification of Terms Categorization of Issues Abstract fixation from particulars Needs: Category, Class, Particular Needs Risk, Threat, Attack, Weakness,

Vulnerability Needs account for Severity, Attackability

(Ease), Prevalence, Priority


The Classification Basics:

Risk ((Loss|Impact) : Financial, Legal, Market)

Threat (Likelihood, Severity, Ease, Implication)

Attack (Exploit Vector) Weakness (Fundamental Issue) Vulnerability (Particular Issue/Flaw

Instance)


The Classification Breakdown:

-Example One- Risk

Financial Loss, Repudiation, Market Goodwill Threat

Elevation of Privilege; Forged Transaction Attack (A: Spoofing of User Identity)

A1 Session Fixation A1-2 Session Token\Cookie set a priori user authentication

Weakness Insecure Authorization Mechanism; Insecure Session

Handling Vulnerability

Harvest Session Cookie on Login Form


The Classification Breakdown:

-Example Two- Risk

Financial Loss(???), Market Goodwill Threat

Elevation of Privilege; Sensitive Information Disclosure Attack (A: Arbitrary Script Injection)

A1 XSS/Cross-Site Scripting A1-2 Embedded/Persisted Malicious Javascript

Weakness Insecure Output Encoding

Vulnerability Support forum page allows embedded linking to external

javascript/flash/actionscript/etc.


Example Problems with Existing Threat Models:

Microsoft: STRIDE and DREAD Spoofing of User Identity (Threat or Attack?) Tampering with Data (Attack?) Repudiation (Risk or Threat?) Denial of Service (Threat or Attack?) Elevation of Privilege (Threat)

STRIDE, DREAD and others mix of Threats, Attacks, Vulnerabilities, disparate elements. New TRIKE methodology shows promise.


More on Threat Modeling:

Threat Modeling is very important; I use it extensively particularly in communicating to business owners or stakeholders of an application

For more see the OWASP Testing presentation The OWASP Testing GUIDE STRIDE & DREAD in MS Threat Modeling Book TRIKE is a new Threat Modeling Whitepaper Visit the Threats and Countermeasures

Website by Foundstone


How do we evaluate applications for Security?

First and most important application security principle:

The Application Must Defend Itself


Conclusion to Chapter III

The previous material is presented to provide a common set of terms and framework for understanding what we can and can’t do with automated tools that discover software flaws with security implications.

Security is not a “feature” you implement in code Security is an emergent Quality of well-designed and well-

built software, much like speed and reliability Categorical difference between technical and functional

issues and how you discover them Tools tend to flag everything as “vulnerabilities” Tools really find a mix of threats, attacks, and vulnerabilities

(XSS is an attack, not a vulnerability) Tools tend not to indicate systemic weaknesses, like

improper output encoding techniques, which is what we really need to be concerned with if our focus is *fixing* the application issues.


Chapter IV

Taxonomy of Testing


Taxonomy of Testing Categories

1. Vulnerability Scanning2. Fault-Injection Testing

(Blackboxing)3. Fault-Injection Testing with

Sandboxing4. Binary Analysis5. Source Code Analysis6. Threat Modeling/Architectural

Analysis7. Rootkit and Trojan Analysis8. (What about runtime analysis and

runtime patching?)



1. Vulnerability Scanning regex matches on version numbers.

2. Fault-Injection Testing (Blackboxing) Inject all possible faults and parse

responses.3. Fault-Injection Testing with Sandboxing

Inject all possible faults and analyze data storage, manipulation, and local entry and exit points, and fuzz/fault-inject those points.

4. Binary Analysis Analyze static binary or perform

runtime analysis and look for breakpoints.



5. Source Code Analysis Mostly static signature matching, but some tools claim

to walk codepath, follow nested loops and other regression through objects

6. Threat Modeling/Architectural Analysis Detailed, logical modeling. Time consuming, thorough;

avoid wasted man hours of broken feature-implementation up front. Or find obvious design flaws quickly.

7. Rootkit and Trojan Analysis Why not? The NCUA/OCCU mandates it.

8. What about runtime analysis and runtime patching? Another idea…


What Follows

The next series of slides is a short summary of common and well-known applications that fall into these categories.

It is not intended to be comprehensive (later slides will be more comprehensive).

It is not intended to indicate quality or imply any form of advocacy for the listed software security tools.

If you are looking for product advocacy there are plenty of sales people and “sales engineers” that can help you there.

Most importantly: These tools are constantly evolving. Consider that the data presented may already be outdated. Use as a guide only and evaluate any selection you make Yourself, on Your Applications.


Vulnerability Scanning

1. ISS Internet Security Scanner (the old basic VA tool)

2. eEye Retina (CGI Scanning?)3. Qualys Qualysguard (XSS, limited

SQLi)4. NGS Typhon (claims full JS Parsing)5. McAfee Foundscan (DNE for

webapps)6. nCircle IP360 (similar to

Qualysguard)7. Known-File Scanners e.g.-Whisker,

Nikto, Nessus+Nikto, Nstalker Nstealth

These tools do a regex version match (e.g. 12.8.1.0 diff vuln-database) or simply match a filename (e.g. “login.asp”…Oh No!, not a Login!)


Fault Injection Testing

1. SPI Dynamics WebInspect 5.52. Watchfire AppScan 5.53. Acunetix WVS 2.x4. Cenzic Hailstorm 2.65. NTO Spider x?6. Compuware Devpartner x?7. NGS Typhon x?8. Parasoft Webking X?9. SPIKE (while using binaries with attached debuggers to

monitor breakpoints like any traditional fuzzing)10. Peach Fuzzer

These tools inject a fault into the application and in some cases evaluate the response.


Fault Injection with Sandboxing

Sandboxing Tools:

• Security Innovation’s Holodeck• Sysinternals filemon, regmon,

procmon• Any sandboxing malware-analysis

tools


Binary Analysis

1. IDA Pro2. @stake SmartRisk Analyzers3. Fortify Application Risk Analyzer4. Halvar Flake’s Sabre-Security

projects– BinNavi– BinDiff– BinAudit

5. WiSA


Source Code Analysis

1. Compuware2. Coverity3. Fortify Software4. Ounce Labs5. Parasoft6. Secure Software7. Lots O’ Free & Open Source Stuff


Threat Modeling / Architectural Analysis

1. Microsoft Threat Modeling Tool2. SecuriTree3. PTA Technologies4. TRIKE TM tool (to be released)


Rootkit/Trojan/Backdoor Analysis

Checking for Rootkits, Trojans, and Backdoors is another interesting issue that is recommended in the “Secure Coding” pamphlet sold as a book by O’Reilly, and recommended by the NCUA 2004-03 Guidance letter “must review source code for backdoors, trojans, etc. etc.”

1. Rootkit and Binary analysis tools for Backdoors & Trojans


Runtime Analysis and Patching

Tools that are designed to analyze behavior and code at runtime and perform runtime limitation/security enforcement and/or “virtual” patching of userspace coded and kernel code.

1. Immunix kernel & stack protection.2. Some new virtual patching product

for Linux that I accidentally deleted the info on and can’t find the emails of the kind folks who contacted me originally.


Chapter V

Automation Tools(or manual human eyeball supplicants)

YMMVKey:MA (Mature)HP (Has Promise)DNE (Did Not Evaluate)RE (Refused Evaluation or ignored repeated requests)TOY (Amusing little widget)


Vulnerability Scanning

1. ISS Internet Security Scanner (???)2. eEye Retina (CGI Scanning?)3. Qualys Qualysguard (XSS, limited

SQLI)4. McAfee Foundstone Enterprise5. nCircle IP3606. Rapid 7 Nexus7. NGS Typhon (full JS Parsing; XSS &

SQLI)8. Saint9. Known-File Scanners e.g.-Whisker,

Nikto, Nessus+Nikto, Nstalker Nstealth


Commercial Fault Injection Test Tools

1. SPI Dynamics WebInspect (5.5, MA)2. Sanctum now Watchfire AppScan (5.5 MA but may become a

‘dashboard’)3. Cenzic Hailstorm (HP…ways to go, version 2.6 DNE)4. NT Objectives NTOSpider (HP, DNE new GUI version)5. Acunetix Web Vulnerability Scanner 2 (TOY, no scheduling, state, form-

fill, etc.)6. Compuware DevPartner Fault Simulator (RE)7. Whitehats Security (hosted application, ‘dashboard’ approach, think

‘Qualys’)8. Fortify Pen Testing Team Tool (RE, unknown; secretive)9. Burp Intruder (HP, nice fuzzer)10. MaxPatrol 7 (strange phone-home to Russia traffic observed, stopped

eval)11. Syhunt Sandcat Scanner & Miner (evolving)12. TrustSecurityConsulting HTTPExplorer (TOY, limited)13. Ecyware BlueGreen Inspector (TOY, limited)14. NGS Typhon (DNE, claims full parsing/testing capability)15. Parasoft WebKing (DNE, positive references from credible sources)16. Kavado Scando (dead, IP purchased, may show up again, CVE-type

scanner)17. AppSecInc AppDetective for Web Apps (dead, believed permanently)18. @stake Web Proxy 2.0 (dead or suffocating at Symantec)19. Sandsprite Web Sleuth (HP, but development End-of-Life?)



Data on FI Tools:Data includes version number as many of these tools have

new features/improvements since time of testing: Tools tested from March to August of 2005 Tools tested on sample applications like OWASP

WebGoat Tools tested on multiple real-world applications Tools tested on same versions of applications Tools tested on “default” settings and also attempted to

customize tools as best possible to provide optimal use-case scenario for final result data

Many tools were configured with help from or onsite support from the vendors, all installed on clean, new boxes, so should represent the best the vendor could provide at that time.

I have been guilty of RTFM errors before, but for this testing this is why vendor configuration support was solicited.



SPI Dynamics WebInspect 5.5Pros: Best manual testing tools of the FI market (Proxy, Cookie

Tester) Excellent javascript parsing Most powerful tools for writing custom checks Best API integration with other QA tools Have removed a lot of the marketing crap that other

tools still report like detecting “31 Flavors of XSS” and

Cons: No ability to look under the hood on adaptive agents No ability to customize default checks (to protect “IP”) Come on guys, I can run a sniffer and see what you are

doing anyway Not the fastest



Watchfire AppScan 5.5Pros: Fast on smaller applications Best XSS testing Most powerful right out of the box with no tweaking Excellent reporting New free manual testing toolkit; ‘PowerTools’

Cons: No ability to look under the hood or customize effectively Unnecessary XSS variants, and made-up vuln

terminology Watchfire is a dashboard company and this is not a

effective way to evaluate/measure application security Limited ability to tweak to handle dynamic session

affinity and complex authentication variants



Cenzic Hailstorm 2.3Pros: Allow you to get under the hood and customize

everything. Most potential for automating complex custom checks. Nice drag-and-drop configuration in GUI. Definitely a contender to keep an eye on.

Cons: Custom checks simply didn’t work right Slowest tool tested No ability to configure operational/engine parameters

like threading, bandwidth, scope, etc. At time of testing was long on hype, highest priced, low

on delivery of results



NT Objectives NTOSpider1.something CLI-only beta version

Pros: Fast. Wow, this thing is fast. Most effective XSS checks behind WI and AS. Only tool with effective automated session token testing. Auto-state detection and auto-login abilities. Small, lightweight, simple configuration.

Cons: No CVE-style checks. No ability to look under the hood or customize tests

easily. Roughly half the accuracy of WI or AS on XSS and SQLi. No GUI at time of testing.



Acunetix WVSPros: Nice, pretty GUI Appears fast, but it doesn’t do much so I guess it’s easy

to be fast at that.

Cons: Limited authentication support No scripting or scheduling capabilities Utter lack of session/state handling No manual analysis tools False positive centric Marketing appears targeted at sub-100 IQ demographic Included due to massive advertising blitz recently



Compuware DevPartner Fault SimulatorPros: Built with the Developer in mind IDE integration Integrates with Source-Code analysis May be one to keep an eye on but looks like not intended

to be “best of breed” solution.

Cons: Vendor has no clue about competitive landscape. Vendor appears to have limited security knowledge. Did Webex eval only. Got tool but not license key; vendor promised follow up

repeatedly but never did. Vendor to provide data for the OWASP Tools project but

never followed through in providing this.



WhiteHat Security (the Seattle Company)Hosted FI Tool; think Qualys of WebAppSec

Pros: DNE (Did Not Evaluate) Some smart people building this Hosting approach nice for repeated testing/change

management Cost effective

Cons: No replacement for manual testing DNE



NGS TyphonPros: DNE for Web Application Checks Experience with NGS tools is that they are very fast,

fairly accurate, lack ability to customize/configure much through the GUI and have terrible reporting.

Will include in next release of this project.

Cons: DNE.



Parasoft WebKingPros: DNE Have heard has is less mature but some nice features Will include in next version of project

Cons: DNE



SandSprite Web SleuthPros: Inexpensive Simple easy-to-use Windows GUI Nice code markup and views Custom scripts for crawling, cookie testing, etc.

Cons: Geared towards all manual testing No default pre-built checks Doesn’t appear under active development



OWASP WebScarab v.20051012 Pros: Many advanced testing features geared towards

experienced application security professionals. Free & Open Source. Session Token entropy analysis. Static string fuzzer. User-extensible with custom agents. Web Services testing interface for complex types

Cons: Non-intuitive, complex interface. No “scan” or pre-built checks like CVE issues. Tends to hang and need restarted. No vulnerability reporting mechanism or management

style reporting a’la Watchfire.


Fault Injection Test Tools Anecdotes

Similar to diet fads My Book “Body of Anecdotal Evidence:

Portion-sized Marketing” out soon That was a “Body for Life” joke for you

not from the US Again, there is no book for those of you

that asked me after the presentation. YMMV Anecdotes are subjective Anecdotes are not logically valid proofs Repeat Disclaimer: YMMV


Fault Injection Tool Experiences

The following metrics were gathered from real-world assessments performed by myself along with up to four other testers.

These were commercial assessments on a mix of COTS software and custom applications developed by clients ranging from Fortune 100 to smaller ISVs that authorized benchmarking of these tools during the course of software security assessment.



Application #1: 150+ pages, Complex workflow Dynamically built javascript menu/functions Application contained 1 trivial XSS, 1 Workflow Bypass

(WFB), 1 Brute Force (BF) The tools in this test were run with default

configurations; essentially clicked ‘scan’ as many people unfortunately use these tools in RealLife™.

WebInspect 5.0: 147 pages, no XSS, WFB, BFAppScan 5.5: 38 pages, 1 XSS, no WFB, BFAcunetix 2: 6 pages, no valid positive resultsNTO Spider: 80+ pages, no XSS, WFB, BFParos 2.something: 0 (zero) (loop)Cenzic Hailstorm: Couldn’t get working

(The security defects detected in the applications were discovered during the course of manual analysis)



Application #2: 30+ templates, Dynamically generated content 15k+ actual pages, Long forms, simple workflow 32 XSS from URL parameters to multi-form <--!--> Weak Session Handling; Auto-scripting attacks

(“Session Riding” or “Web Trojans”) Tests run as ‘scan’ with minimal tweaking

WebInspect 5.0: 31 XSS, no SH or SR issuesAppScan 5.5: 18k+ pages, slowed to 1 rec/6 secondsAcunetix 2: No Results (couldn’t automate effectively)NTO Spider: 18 XSS, weak SH, no SR, fast! (CLI version)Hailstorm 2.2: 3 days, 4 people, Cenzic onsite, nothingParos 3.0.2: 18 pages, no results, would hang on audit


Fault Injection Tool ExperiencesApplication #3: 100’s of pages/forms Strong UI, Input Validation, Output Encoding Web Server packages XML POST string Passes to routing servlet for authorization Function Servlet name/route in Hidden FF Create XML String and POST directly to DST servlet XML Abstraction Layer has no authentication/authorization ASMQ does not have crypto enabled; auth is function group code Point of this example is that there are holes you could drive a Mac Truck

through and every FI Scanner failed to detect them. Some things require humans.

WebInspect 5.5: NothingAppScan 5.5: Nothing; WF PowerTools Proxy: failedAcunetix 2: PHP issue, some other issues, false positives (200 codes)NTO Spider: Nothing (CLI version); no proxyParos 3.0.2: Nothing; Proxy-mode: failedProxies: SPI Proxy (only proxy to handle various cert-auth)


Fault Injection Tool ExperiencesApplication #4: AAWBDMS (Another Awful Web-Based DMS) XSS, Parameter Tampering, etc. (everything you can imagine

controlled in client-side hidden form-field parameters) URL Parameters passed directly to compiled C code (DLLs,

EXEs) running as Domain Admin with traditional buffer overflows.

Dual-Factor Authentication Authentication Two-step Workflow First page UserID & PRNG Token number in HTML Form Second Page identical UserID and Password in HTML Form All responses 200; everything returns one of two login forms

Every darn tool: NADT

NADT: Not a Darn Thing

Nobody could handle dual-auth with custom error pages identical to the login pages.


Fault Injection Tool ExperiencesXSS attacks should be fully automated, but as of yet are not. We find

XSS in almost every application we test that is not found by any automated tool; some are trivially devastating. Examples:

Form value expects email address ([email protected])

Form value validated by sloppy regex; Output not encoded Form error returns no data unless you bypass regex filter Tools submit the following types of checks:

<script>alert(‘somethingclever’);<script> <script>alert(‘somethingclever’);<script>[email protected] [email protected]<script>alert(‘somethingclever’);<script> some replace ‘silly’ and ‘com’ with <script>alert();<script> check some add escape characters like ‘>, ‘>>, ‘), and so on

Actual XSS by silly@<script>alert()<script>.com Other attack vectors tools fail include

encoding attack strings (reference OWASP Guide 2.0.1 or RSnake’s XSS page)

partial encoding of attack string elements using specific HTML tags like body elements and background


Open Source or Freeware Fault Injection Test Tools

1. WebScarab (DNE current version, re: HTTPush, Exodus)

2. Paros Proxy (HP)3. Watchfire PowerTools (HP, free, proxy nice GUI, rest

so-so)4. Burp Spider (HP, free)5. Burp Proxy (HP, free)6. Snark HTTP Proxy & Tabby Tunnel (DNE)7. Peach Fuzzer (HP, free, Python fuzzer)8. SPIKE Proxy (Horrible interface, free, limited fuzzing)9. SPIKE Fuzzer (dev EOL? Dave pointing people at

Peach)10. Achilles Proxy (TOY, first Windows proxy, free, old)11. Odysseus Proxy (some cool features, GUI needs work)12. Webstretch Proxy (DNE)13. Absinthe 1.1 (formerly SQLSqueal) (HP, free)14. Sensepost E-Or, Wikto (Google hacking utility), other

webappsec and pen testing tools(everything Sensepost is working on is worth looking at)


Open Source or Freeware Fault Injection Test Tools

And your Basic Web Browsers…don’t forget them:

15. Internet Explorer + HTMLBar DLL extension (buggy)16. Firefox + LiveHTTP Headers, Tamper Data,

Developer Tools,and many, many, more useful

extensions….however,keep in mind Firefox encoding is often different,

morerestrictive, sometimes more secure than IE so if

theapplication client population includes/is

predominantlyIE MAKE SURE you test encoding/format in IE as

well.

17. Foundstone Free Tools: SiteDigger (Google Hacking Utility) SSLDigger Probably others I haven’t looked at yet SiteGenerator (coming soon, may be useful for

testing Fault-Injection tools for javascript parsing, etc.)


Fault Injection with Sandboxing

Sandboxing Tools (Commercial & Free):

1. Security Innovation’s Holodeck (HP)

2. Sysinternals filemon, regmon, procmon can be combined with VMWare

3. Any sandboxing malware-analysis tools available from various forensics sites and possibly from the AV vendors.


Binary Analysis

1. IDA Pro + scripts from Halvar & iDefense2. Grammatech CodeSonar3. Paradyn Project (claims superiority to IDA Pro when C++ variables

present in object)

4. FX Dumbug5. @Stake SmartRisk Analyzers (gone/dead)6. Fortify Application Risk Analyzer (marketing freebie)7. WiSA (Wisconsin Safety Analyzer)8. SABRE BinDiff (IDA Pro extension)

9. SABRE BinNavi (Graphical flowgraphs, playback of codepaths, Win/Linux debuggers)

10. SABRE BinAudit (Full Binary Analysis; beta project; Halvar restarted this again)

11. Secure Software CodeAssure Auditor (Binary Analysis)

12. Brute Force Binary Checker (http://bfbtester.sourceforge.org)


Commercial Source Code Analysis

1. Compuware DevPartner SecurityChecker .NET Languages

2. Coverity SWAT C/C++

3. Escher Technologies Eschertech4. Fortify Software Suite (analysis, workbench, metrics &

trending console, customization module, won’t show me anything) C/C++,Java,JSP,HTML,Java bytecode,

(PL/SQL),C#,VB.NET,MSIL, …5. Gimple PC and Flexe-Lint C/C++ (we use, simple, cheap, functional)

6. Grammatech CodeSurfer C/C++

7. Ounce Labs Prexis C/C++, JSP, Java (nice, rough UI, recursion abilities?)

8. Parasoft JTest Java (? NE)

9. Reasoning SecurityInspect “Service”10. Secure Software CodeAssure Workbench C/C++, Java

11. Security-Assessments CodeScan ? (April 01 05) won’t give specifics12. Note if you are in sales: My name is not Ariel, Arial, Adrian, Aruin, etc

13. WiSA C/C++


Open Source, Free, or Bundled Source Code Analysis

1. OWASP .NET (Reference Dinis Cruz presentations on OWASP website about OWASP .NET tools)

2. Foundstone .NET Tools (same story as above; DNE but heard these were coming out soon)

The Next Slide has All The Other Stuff


Open Source, Free, or Bundled Source Code Analysis

1. Bunch http://serg.cs.drexel.edu/bunch2. Cigital ITS4 (no longer supported) C/C++3. CQual http://www.cs.berkeley.edu/~jfoster/cqual4. David Wagner’s BOON C

http://www.cs.berkeley.edu/~daw/boon/5. David Wheeler’s Flawfinder C/C++ www.dwheeler.com6. Reference/Credit David Wheeler’s Page7. Microsoft PREFix (full codepath execution, recursion), C families8. Microsoft PREFast (stripped down PREfix) only static signatures

for performance reasons, shipping with Visual Team Studio 2005, C families

9. Microsoft FXCop (even more limited signature tool) C/C++ only10. MOPS http://www.cs.berkeley.edu/~daw/mops11. PScan http://www.striker.ottawa.on.ca/~aland/pscan12. RAT C/C++, Java, Perl, Python, whatever other random

signatures written13. ShareFuzz14. SPLINT http://splint.org/


Threat Modeling / Architectural Analysis

1. Microsoft Threat Modeling Tool(non-intuitive for beginners)

2. SecuriTree(clunky Java tool, not appsec focused)

3. PTA Technologies(DNE, have had several references)

4. TRIKE Threat-Modeling Tool(DNE, in development stage, unsure

releasedate, but idea has promise)

Need more tools here!


Rootkit/Trojan/Backdoor Analysis

Checking for Rootkits, Trojans, and Backdoors is another interesting issue that is recommended in the “Secure Coding” pamphlet sold as a book by O’Reilly, and recommended by the NCUA 2004-03 Guidance letter “must review source code for backdoors, trojans, etc. etc.”

1. Rootkit and Binary analysis tools for Backdoors & Trojans

rootkits.org and rootkit.nl + many Forensic Distributions have these tools


Recommended Motto for Vendors

Ne Supra Crepidum Suter Judicaret


Chapter 6

Shoot the Messenger

(he may duck and run)


Arian J. EvansDoes Application Security Stuff

FishNet Security913.710.7085 [mobile]

[email protected]

OWASP Tools ProjectOWASP Chapter Leader, Kansas City

Feel free to contact me with questionsVendor updates, corrections, confusions, or complaints welcome

Email me repeatedly if I miss your first one

Documents

OWASP AppSec 2004 Presentation