Firefox 2.0 is better than IE7 and IE7 is better than Firefox 2.0

Nov 15, 2006 11:16 GMT  ·  By

Microsoft and Mozilla both contracted independent market firms to perform effectiveness tests on the anti-phishing features included in Firefox 2.0 and Internet Explorer 7. Big surprise... both Microsoft Internet Explorer 7 and Mozilla Firefox 2.0 came out on top in each respective test.

3Sharp LLC, the technical service contracted by Microsoft indicated that Internet Explorer 7 (the test was carried out at the time on the Beta 3 build of the browser) produced a composite score of 172 out of 200 possible while Google Safe Browsing on Firefox only managed a score of 106. The tests involved a total of 100 known-phishing websites and 500 proved genuine URL.

In response, Mozilla turned to SmartWare to author a study of the anti-phishing performances of Firefox 2.0. Following various testing scenarios, SmartWare proposed a maximum 81.54% blocked phishing sites for Firefox 2.0 while Internet Explorer 7 only managed to amount to 66.35%.

So where is the truth? Criticism has already emerged targeting the Mozilla and SmartWare anti-phishing study. The first one of them is the fact that while Microsoft produced a composite score for Internet Explorer 7 and that SmartWare's study for Firefox had a different methodology. First off, the Firefox 2.0 anti-phishing performance test only compared Mozilla's browser to IE7. Microsoft's own study took into account not only Firefox and Internet Explorer but also additional browsers and anti-phishing technologies such as Netcraft Toolbar, eBay Toolbar, Earthlink ScamBlocker, GeoTrust TrustWatch, Netscape 8.1 and McAfee Site Advisor.

Another problem put forward is also related to methodology. The tests of the anti-phishing filters in Firefox 2.0 did not include a report on the false positives generated by the browser, an aspect that was included in the IE7 composite phishing score. "They didn't make any attempt to score false positives. This is a critical omission, because a filter that produces significant numbers of false positives will quickly train users to ignore its legitimate warnings," commented Paul Robichaux, from 3Sharp.

The websites used for the anti-phishing effectiveness test of Firefox 2.0, although an impressive 1040 sites, came all from a single source Phishtank, a community filtering system and were gathered over a two week period. "That's a good number of phish, but the study period was awfully short, and the phish all came from one source. We used multiple sources, including honeypots and user reports, to generate the phish list we used. Because they used a community-generated feed of phish, there's no way to tell which of the phish had also (or already) been reported to other systems that may have fed into the "Ask Google" or Microsoft data feeds. By contrast, we took great pains to try to find phish that we knew hadn't been submitted to Microsoft's URL reputation service," added Robichaux.

Additionally, SmartWare's study for Firefox and IE7 reports only information of the percentage of blocked phishing sited while 3Sharp provided data related to blocked, not blocked, warn and false positive percentages.