Analyzing the Effectiveness and Coverage of Web AP

8/14/2019 Analyzing the Effectiveness and Coverage of Web AP

1/6


2/6


3/6

Links Crawled Surface Coverage

500

000

500

000

500

000

500

000

500

0

275

250

225

200

175

125

100

75

50

25

0

Vulnerability Findings % Findings That Are False Positiv

50

25

00

75

50

25

00

75

25

0

RWSS

App

Scan

Web

Inspect

0% 10% 20% 30% 40% 50% 6RWSS AppScan WebInspect

RWSS AppScan WebInspect RWSS AppScan WebInspect

Real Finding False Positive

Analyzing the Effectiveness and Coverageof Web Application Security Scanners

By Larry Suto Application Security Consultant San Francisco October, 2007


4/6



he basic idea of the Tracer tool is to place code into the application that will be able to analyze the web application scan

s it accesses security critical areas of the application. Code can be placed manually into the application source as it is

nd turned on and off as testing takes place. This is more difficult and time consuming and requires access to the source c

nd thus the technique most often used is byte code injection which only requires access to the compiled byte code.

oftware that uses bytecode in its operation (Java/.Net) can leverage techniques that come from the code coverage world

s instrumentation (byte code injection) in which code is inserted during build time or at run time using a custom class loa

nstrumentation adds the new code to the compiled code and thus the source is not necessary. Tracer employs byte c

nserted statically into the compiled application byte code.

ach scanner was run in default mode and not tuned in any capacity to the application. The importance of this lies in

ffective the default mode so that scalability of scanning is not limited by manual intervention and setup procedures w

an be very time consuming. Second, in most cases it is simply unrealistic to spend much time with many applications.

ach scanner was provided a basic username and password similar to what a basic user would receive. There were iss

ogging into some of the applications but calls to technical support quickly solved these problems. The Tracer applica

was configured to monitor each web scanner as it executed its tests. After completing each test run, the results were sa

nd the monitoring tool was reset for the next scanner.

All vulnerability findings were hand vetted to determine if they were true findings or false positives.

Testing

he following section reports the results of the tests. The analysis for this study was focused on: 1) determining how wel

canners covered security sensitive sections of code within an application under test, 2) number of true vulnerability find

nd 3) number of false positives. This study did not hand vet the applications to determine if there were any false negat

eyond those in the set discovered by any tool. A set of security sensitive coverage categories were selected from the re

et and the presented in the following tables:

Detailed Results

Closed Source - Internal Corporate Application

Methodology

SCANNER LINKSCRAWLED

DATABASEAPI

WEB API TOTAL API VULNERABILITYFINDINGS

FALSEPOSITIVES

RWSS 91

91

AppScan 113

20

17

17

35

23

24

55

40

41

0

0

0

0

0

0WebInspect


5/6



Roller - Open Source Blogging platform


DATABASEAPI


FALSEPOSITIVES

RWSS 736

663

AppScan 129

2

1

1

121

68

98

123

69

99

2

0

1

0

5

10WebInspect

OpenCMS - Open Source Customer Management application


DATABASEAPI


FALSEPOSITIVES

RWSS 3,380

1,687

AppScan 742

47

45

36

158

140

132

205

185

168

225

27

11

0

0

3WebInspect

Totals


DATABASEAPI


FALSEPOSITIVES

RWSS 4,207

2,441

AppScan 984

69

63

54

314

254

254

383

294

308

227

27

12

0

5

13WebInspect

he three applications chosen have increasing level of complexity and size. The three scanners all did a reasonable job on

maller, first application. The second application resulted in some false positives for Appscan and WebInspect.

he results for the last application are notable in that both AppScan and WebInspect severely underperformed RWSS in all t

ey areas of the test: 1) application coverage, 2) vulnerability findings and 3) avoidance of false positives. The fact that th

esults were most evident only in the most complex of these applications may indicate that for security groups to adequa

est scanners, they need to use more complex applications. It is not surprising that smaller, less complex applications show

ifference between the tools. One would expect fewer true findings and less complexity in crawling the applications.


6/6



n the aggregate, RWSS crawled 328% more links than AppScan and 72% more links than WebInspect; RWSS covered 24% m

f the total APIs than AppScan and 30% more than WebInspect. RWSS found 227 total vulnerabilities versus 27 for AppS

nd 12 for WebInspect. None of the findings by AppScan or WebInspect were missed by RWSS and AppScan missed 88%

WebInspect missed 95% of the legitimate vulnerabilities found by RWSS. RWSS had a 0% false positive rate. Appscan h

alse positives and a 16% false positive rate. WebInspect had 12 false positives and a 52% false positive rate.

he false positive findings were of interest because some appeared to be caused by custom 404 error handling routines in

web application, and some simply were based on faulty assumptions.

n addition the areas that coverage tool reported as missed were analyzed to determine if there were any security cri

ections and also to try and to determine whether it would actually be possible for http based requests to access that por

f the application.

he most surprising result is the discrepancy in the number of vulnerability findings between the three tools. AppScan

WebInpsect are market share leaders in the space and their companies were both recently purchased by large, sophistic

echnology companies (AppScan by IBM and WebInspect by HP). While security professionals testing small, highly sec

imple applications may achieve acceptable results from AppScan and WebInpsect, these results indicate that they may

ome concern with relying on the results of these tools for larger applications. The relatively large number of false posit

articularly for WebInspect, is also a matter of some concern. False positives can be difficult for all but the most experien

ecurity professional to identify. If they are not identified, they can cause difficulties by weakening the credibility of the sec

eam with application developers. Additionally, vetting false positives by hand, even by experienced security professiona

very time intensive process that will increase the cost of the program. While WebInspect has certain tools to reduce f

ositives, it would appear that this remedy is not necessary if using RWSS (and to a lesser extent AppScan). In any case, train

he tool to reduce false positives will need to be done by experienced personnel and will increase program costs.

Conclusion

arry Suto is an independent consultant which has consulted for companies such as Wells Fargo, Pepsico, Kaiser Permana

harles Schwab and Cisco during his time with Strategic Data Command Inc. based in Oakland, CA. He specializes in enterp

ecurity architecture, risk management, software quality analysis from a security perspective and RF security. Larry has b

ctive in the industry for over ten years and has worked with many fortune 500 companies around the country. Recently

esearch has included introducing software testing and quality assurance techniques into the security engineering proce

rder to understand how the effectiveness of security tools and processes can be measured with more accurate and quantifi

metrics. Larry can be reached at [email protected]

Biography

About eEye Digital Security

Eye Digital Security is pioneering a new class of security products integrated threat management. This next-generation of securi

etects vulnerabilities and threats, prevents intrusions, protects all of an enterprises key computing resources, from endpoints to

etwork assets to web sites and web applications, all while providing a centralized point of security management and network

isibility. eEyes research team is consistently the first to identify new threats in the wild, and our products leverage that research

eliver on the goal of making network security as easy to use and reliable as networking itself. Founded in 1998 and headquartere

n Orange County, California, eEye Digital Security protects more than 9,000 corporate and government organizations worldwide

ncluding half of the Fortune 100. For more information, please visit www.eEye.com

To learn more, please visit www.eeye.com

or call 866.282.8276

l

Documents

Analyzing the Effectiveness and Coverage of Web AP