14 DAY TRIAL // In late February, Secunia published its vulnerability report for 2013. The study should provide valuable information to organizations, but experts say that Secunia has not only missed some vulnerabilities, but it has also included questionable or incorrect statistics.
In a lengthy review of the Secunia Vulnerability Review 2014 on the blog of the Open Source Vulnerability Database (OSVDB), Jericho, a founding member of Atrrition.org and an officer in the Open Security Foundation (OSF), highlights the problems.
First of all, Jericho points to the methodology used to count vulnerabilities. Secunia doesn’t count CVE identifiers, but instead it relies on the number of vulnerabilities covered by the company’s advisories. The vulnerability count however is not part of public advisories, making it impossible to realistically duplicate the results.
Furthermore, as Secunia admits, one advisory may cover multiple products, and multiple advisories may cover the same vulnerability.
“This high rate of duplicates and lack of unique identifiers make the data set too convoluted for meaningful statistics,” Jericho wrote.
Another problem is the way that the vulnerabilities covered in the review are classified – all five classifications include the word “critical” and the distinction between them is minor. Moreover, this scoring system isn’t consistent with others.
As far as the figures in the report are concerned, Jericho highlights the fact that the categorization of non-Microsoft programs as third-party programs can be confusing.
“This completely discounts users of Apple, Linux, VMs (e.g. Oracle, VMWare, Citrix), and mobile devices among others. Such a Microsoft-centric report should clearly be labeled as such, not as a general vulnerability report,” the expert noted.
Secunia’s review shows that 13,073 vulnerabilities were found in 2,289 products from 539 vendors. This clearly shows that several of the security holes are counted multiple times.
For instance, four distinct advisories have been published for multiple Java vulnerabilities in IBM’s WebSphere Application Server. They each cover different versions of the software, but they cover mostly the same vulnerabilities. So instead of 27 flaws, IBM is shown as having 102 vulnerabilities in its product.
When it comes to the “Top 50” applications, it contains a wide range of software, including Windows components installed by default and product driver support tools.
“With approximately half of the Top 50 software having vulnerabilities, and mixing different types of software components, it causes summary put forth by Secunia to be misleading,” Jericho noted.
“Since they include Google Chrome on the list, by their current logic, they should also include WebKit which is a third-party library wrapped into Chrome, just as they include ‘Microsoft PowerPoint Viewer ’ (33) which is a component of ‘Microsoft Powerpoint’ (14) and does not install separately.”
Check out the full review of the Secunia Vulnerability Review 2014 on the OSVDB blog.
Update. Secunia says it's aware of OSVDB's report. The company has provided the following statement to Softpedia:
"Yes, Secunia is aware of the blog post. It is company policy not to comment or engage publicly in criticism/accusations with competitors. Secunia firmly stands behind the data and methodology of its Annual Vulnerability Review.”