Ha.ckers blog published Larry's new report: "Accuracy and Time Costs of Web Application Security Scanner Report".Unfortunately Larry never contacted us so we didn't know that he was doing such a test. However as soon as the report was out we conducted the very same test as methodology was straight forward.Also we sent an email to Larry and offered fully-functional trial version for him to conduct the test as well. Anyone with full or demo version of Netsparker can repeat these tests easily.

How did we conduct the test?

• We used the last public version of Netsparker 1.1.5.87
• We have not trained the scanner other than changing the start URL of the scans.
• We attached the save files, Netsparker database and XML reports. So you can see the results by yourself. If you don't have Netsparker than you can take a look at our XML reports. Download reports and save files.
• We considered [High Possible] vulnerabilities as vulnerable and if [High Possible] vulnerability was not correct then it considered as False Positive, although all high possible issues we correct.

This is the overall output:

Two charts exactly like Larry's report, we added Netsparker to the results.

As you can see after NTO Spider, Netsparker is the best scanner when "Trained" and the second best trainer in "Point and Shoot" right after NTO and IBM AppScan.

False Positive Free Scanning

We delivered what we claimed and the report was false-positive free.Although 1.1.5.87 release caused some LFI bugs which already has been addressed and will be deployed in the next release. This caused [High Possible] LFI vulnerabilities. None of them were confirmed (obviously!) but it was quite irritating, for full-disclosure I wanted to point out that clearly. We spotted some instances unconfirmed possible LFI issues in all Permanent XSS locations . Since we considered [High Possible] as vulnerability, we'll consider these as False Positives. We addressed this problem in v1.1.5.91, use "Help > Check Update" to update Netsparker.

Training Time

Actually Netsparker has not many training options because it doesn't require any. It picks up many URL Rewrites automatically, it detects Custom 404's on the fly, 99% of the time you don't need to tweak it. It just works. In this case all we did was changing the start URL of the scan. So our training time was something between a second and a minute. Depending on how fast we can copy & paste a URL.

Overall Human Time/Cost

Larry calculated the overall human time/cost with the following formula:

Training time + (# False Positives * 15min) + (# False Negatives * 15min)

This is the original chart (in minutes, lower is better):

However since none of the scanners in the test has a confirmation engine like Netsparker he excluded the fact that even though all of the issues are not false positives you still have to analyse them, otherwise you wouldn't know if they are false-positive or not.This is not an issue with Netsparker as we can confirm vulnerabilities. Out of 103 identified vulnerabilities we confirmed 87 of them, so we confirmed 84% of all identified issues. This could've been much higher if the test websites were using MySQL, ORACLE or MS SQL or even Postgres (we have limited support) instead of MS Access.I'll discuss this further at the end of this post.So I've revised Larry's function and made it more realistic by adding one more criteria, time to confirm that a vulnerability is not a false positive. 15 minutes would have been harsh as some issues could be really obvious so I used 3 minutes for per identified vulnerability which is quite naive.Revised Formula:

Training time + (# False Positives * 15min) + (# False Negatives * 15min) + ( # Identified none FP Vulnerabilities * 3 min )

Updated and more realistic results (in minutes, lower is better):

Netsparker identified 5 new vulnerabilities that other scanners missed

Netsparker identified 7 new vulnerabilities that all of the other scanners missed:

• NTO Webscantest
  • URL Based XSS in /datastore/search_get_by_id.php and in many other URLs
  • XSS in "method" parameter in /soap/wsdlclient12.php
• Acunetix testPHP
  • XSS in "uphone" parameter in "/userinfo.php"
• Cenzic Crackme
  • Permanent XSS in /kelev/php/loanrequestlist.php
  • Permanent XSS in /kelev/php/approveloanpage.php

UPDATE: 2 of the issues removed from zero.webappsecurity.com because they were duplicates, we didn't notice it in the first analysis.

A Funny Vulnerability(!)

We observed that Netsparker missed a remote code evaluation vulnerability according to the Larry's results.I don't know how Larry confirmed all vulnerabilities but this is certainly not exploitable :)From "http://testphp.acunetix.com/comment.php" and "phpaction" parameter.

if ($_POST["phpaction"] == "printf(md5(acunetix_wvs_security_test));exit;//") eval($_POST["phpaction"]);

So it's not actually a vulnerability it's just a PoC vulnerability to demonstrate Acunetix's related checks.

UPDATE: I want to make it clear that this is obviously not an intentional code to block other scanners. This is just a test page for Acunetix scanner for their customers and demo versions. Which make a lot of sense to do such a thing, all I wanted to point out that this issue should be excluded from the test and leaving such an issue in the report might raise questions about other issues. I'm not quite sure if there are any other issues like this in the report as we haven't investigated every single one of them. My apologies to our friends in Acunetix if I seemed like accusing them, that definitely wasn't my intention.

What and Why Netsparker Missed?

• Netsparker missed some XSS vulnerabilities in forms. Netsparker did fill up everything correctly which means it missed the cross-site scripting issues in validation pages. Because it hasn't seen the validation page at all. This issue was on our list and we'll try to address it as soon as possible.
• Netsparker missed some error message information disclosure, because they are so prone to false positives. In real world reporting such issues should cause a lot false positives. Netsparker missed about 6 error messages issues. Just to be clear Netsparker reports debugging related information but these pages were barely leaking any debug related information hence Netsparker didn't report.
• Couldn't confirm or missed MS Access SQL Injections. Netsparker engines are specifically designed for ORACLE, MS SQL and MySQL. Currently we are in the process of adding Postgres and will eventually add support for all popular DBMSes. Basically if you change MS Access backend of all these systems to ORACLE, MySQL or MS SQL Netsparker would find and confirm those vulnerabilities. Obviously this is not an excuse to miss a vulnerability and we're trying to do our best to add these new database engines. Netsparker especially was bad in IBM AppScan's testfire website due to this problem.
• Netsparker missed all attacks based on HTTP Header or Cookies because currently it doesn't support. This is known limitation, in our roadmap and will be addressed.

Conclusions

We were expecting good results and we got it. Although I still think Netsparker would have performed better in more realistic scenarios. For example I haven't notice any Full-Blind SQL Injection (time based) vulnerabilities in the whole test.One of the most unrealistic things about the report is the amount of false positives possibilities in the test websites. If you haven't use any of these scanners just ask anyone and they'll tell that they definitely report more than 2-3% false-positive issues in every scan.Report proves that Netsparker’s Confirmation and False-positive Free Scanning feature is a real time saver.

Download Test Files and XML Reports

• Netsparker Vulnerability Tables – VulnerabilityTables.pdf
• Overall, Chart Document – Charts-Overall.ods
• Netsparker "XML reports" and ".dlm" scan files. You need Netsparker 1.1.5.87 for opening ".dlm" files" – Reports.zip

Do you want to test it too?

You can request a demo and try Netsparker’s fully-functional evaluation version.