Useless MMX instructions added just to throw anti-virus engines off their track

Apr 15, 2009 08:37 GMT  ·  By

VXers are on a constant look-out for new programming techniques that would allow them to create malware that is more resilient to detection. Such is also the case with a new trick observed by virus analysts, which leverages on a rarely used instruction set, in order to make anti-virus emulation harder or impossible.

Today's malware comes encrypted with custom-built packers in order to achieve polymorphism, constantly mutated versions of the same code. This has caused classic definition-based detection to become increasingly obsolete during the past several years and to be replaced by what is known as the heuristic analysis.

Anti-virus heuristic engines can look at the programming instructions inside a file as well as analyze its behavior in a virtual environment in order to determine if the program is malicious. This latter approach is achieved through emulation. "It is no secret that antivirus engines use in-built emulators to safely observe how suspicious files behave when run," Matthew Asquith, malware analyst at Sophos, writes.

Mr. Asquith has recently discovered an interesting piece of code in WinPC Defender, a rogue security application particularly aimed at making emulation impossible. More specifically, this scareware program adds instructions from Intel's MMX instruction set to its packer. MMX is an extension to Intel's standard instruction set, which is "primarily developed to aid in highly computationally-expensive multimedia tasks, like gaming and video encoding."

Since there is no reason for malware to use such instructions, the AV emulators have not been designed to execute them. Therefore, behavioral analysis through emulation might not be possible for this scareware program, whose "MMX instructions […] are used to move numbers to and from the MMX processor without actually performing any operations to change their value."

Useless from a computational perspective, these instructions serve a much darker purpose and demonstrate the ingenuity of the cyber-criminals. However, in this particular case, the trick fails miserably, because another important aspect has not been considered. By using this technique, emulation might be thwarted, but it also makes it incredibly easy for the file-analysis heuristics to pick it up.

"This technique actually makes these samples much easier to detect, as a legitimate program would almost certainly never use MMX instructions in this way when first loaded," the Sophos security researcher points out. "Whether this technique will be used more sophisticatedly in future variants is yet to be seen," he concludes.