14 DAY TRIAL // On April 2, NSS Labs published its Breach Detection Systems (BDS) Security Value Map after analyzing the solutions of six leading vendors: AhnLab, Fidelis, FireEye, Fortinet, Trend Micro and Sourcefire (Cisco). FireEye is not happy with the results of the analysis.
NSS Labs has found that the effectiveness of the breach detection systems ranges from 94.5% to 99.1%. Trend Micro scored the best in overall security effectiveness.
FireEye is displeased with the 94.5% rating, so the company has published a response to NSS Labs’ report. FireEye highlights three main issues with NSS’s analysis.
First of all, the security company claims that NSS mainly obtained the test payloads from VirusTotal, which means that the sample set doesn’t include APTs, or unknown and complex malware. Furthermore, FireEye claims that NSS hasn’t performed forensic analysis on the samples to see if they’re actually malicious, corrupt (can’t execute), or goodware.
FireEye also doesn’t like the testing methodology, which it calls “poor” due to the fact that no zero day exploits have been included in the test samples and their appliances were not connected to the Dynamic Threat Intelligence cloud from where customers get the latest updates.
“We respect NSS and the work they do—especially for IPS – and their testing methodology for BDS is also more suited to testing IPS products. However, we believe the issues we identified with their evaluation of advanced threats are indicative of the security industry’s broader lack of knowledge regarding sophisticated attacks,” FireEye noted in its response.
“FireEye is designed to supplement legacy signature and reputation based technologies to protect against advanced threats—and the NSS tests didn’t properly gauge our capabilities.”
In response to FireEye’s response, NSS Labs’ Bob Walder noted, “Not everyone can end up in the top right quadrant of the NSS Labs Security Value Map™ (SVM), so it is not unusual for someone to be unhappy. It is, however, unusual for someone to behave the way FireEye did in this instance. Normally we would not respond to such attacks, but there are a number of untruths and misdirections in their blog post that we feel we must address.”
Although it claimed that it had declined to take part in the test, NSS said that FireEye did willingly participate. Furthermore, NSS denied that the FireEye product they used in the study was not fully functional, and that samples were taken mostly from VirusTotal.
“In the grand scheme of things, FireEye’s results were not that bad. The real issue here is that FireEye now has credible competition in the BDS market place and the data from this NSS test shows it,” Walder said.
Check out FireEye’s blog post and NSS Labs’ response.