14 DAY TRIAL // A report from a company called Bit9, which counted the number of high-risk vulnerabilities reported in popular software, was misinterpreted by many to show that some applications, like Internet Explorer, are more secure than competing products.
Bit9 claims that its "Dirty Dozen" apps list is meant to raise awareness that popular programs are also the most vulnerable ones, a reason for which they require constant monitoring and patching.
The fact that widespread applications have the highest number of publicly reported vulnerabilities is nothing new and is actually to be expected.
Hackers want to compromise as many systems as possible and will therefore target those programs with the largest user base. In consequence, security researchers will focus their vulnerability finding efforts on such software in order to make it more secure.
Bit9's 2010 "Dirty Dozen" list reads: Google Chrome (76 vulnerabilities), Apple Safari (60), Microsoft Office (57), Adobe Reader and Acrobat (54), Mozilla Firefox (51), Sun Java Development Kit (36), Adobe Shockwave Player (35), Microsoft Internet Explorer (32), RealNetworks RealPlayer (14), Apple WebKit (9), Adobe Flash Player (8), Apple QuickTime (6) and Opera (6).
The company's methodology for this report involved counting vulnerabilities listed in the U.S. National Institute of Standards and Technology’s (NIST) vulnerability database, that had a high severity rating (between 7.0 and 10.0 CVSS base score).
Unfortunately, what some people, including journalists, understood from this list was that applications ranking lower were more secure than those at the top.
However, the number of publicly disclosed vulnerabilities is far from an indication of a program's state of security and this is even admitted by Bit9's Chief Technology Officer Harry Sverdlove.
"You can’t really compare who is #1 on our list to #10, for example, without further context," Sverdlove writes on the company's blog.
"[…] The products toward the top of our list may in fact be more secure or present less risk – IF you are keeping your applications up to date," he explains.
This is because a lot of crucial factors were not taken into account. From a security perspective the speed with which vulnerabilities get fixed is much more important than their number.
For example, at the end of October, Mozilla patched a zero-day Firefox vulnerability discovered in the wild in under 48 hours.
A similar vulnerability discovered in Internet Explorer at the beginning of November remains unpatched two weeks later, despite being actively exploited.
Furthermore, the underlying security features and architecture of a program are also critically important. Chrome comes with a sandbox, which severely restricts how its processes interact with the system.
Therefore, if one of those 76 critical vulnerabilities were to be exploited in Chrome, the attacker would have a very hard time executing arbitrary code to compromise the computer.
Google's browser also features silent automatic updates, which means that the vast majority (over 90%) of its users are always running the latest patched version.
In comparison, according to a recent study, only 80% of Firefox users run Firefox 3.6 and only 60% of IE users run Internet Explorer 8, the latest stable versions of those browsers.
This means that even though Chrome is at the top of Bit9's list, it is actually more secure, from a practical point of view, than its lower-ranking competitors.
In addition, Bit9's vulnerability counting methodology is lacking. It doesn't account for the fact that some vendors don't disclose vulnerabilities discovered in-house, while open source projects like Firefox or Chrome, do.
Then, some companies cover multiple vulnerabilities in a single security bulletin and don't always reveal how many, which can skew the final count.
Furthermore, organizations like Google and Mozilla pay researchers through vulnerability reward programs, which motivates them to more actively search for problems in their products than in those of other vendors.
In our opinion, however well intentioned Bit9's "Dirty Dozen" report might be, its benefits are seriously overshadowed by the confusion it generates every year when it is released.