Different scanning technologies to check URLs

Dec 1, 2009 14:30 GMT  ·  By

The company operating bit.ly, the Internet's most popular URL shortening service, has enlisted the help of VeriSign, Websense and Sophos to protect its users from spam. Each of the vendors will provide bit.ly with fundamentally different URL checking technologies for what aims to be one of the most complete and reliable malicious URL protection solutions.

URL shortening is the practice of generating short aliases for long URLs. URL length might not count that much in the vast majority of cases on the Internet, but it makes a big difference for users of micro-blogging platforms where messages have a very restrictive character limit.

Twitter's explosion in popularity has exponentially increased the need for such services and soon enough multiple such websites sprung up to challenge the authority of TinyURL, one of the first widely used URL shorteners. One of those challengers, called bit.ly, rose up to take the leader position from TinyURL, partly because Twitter chose it as its default URL shortening service in May this year.

URL shortening might be useful for users, but it also poses serious security problems. This is because cybercriminals can hide malicious URLs behind these short aliases, which, due to their widespread use, are trusted by default by many users.

According to recent research by antivirus vendor Kaspersky Lab, which created a service that scans over half a million URLs posted daily on Twitter for malicious content, the current threat level is seven out of ten. The company also revealed that 99% of URLs posted on Twitter are shortened and that over 75% of them use bit.ly as shortener.

So far, bit.ly has responded to the problem by implementing its own spam checking solution and making it possible for users to easily expand short URLs and view their destination before visiting them. However, this is clearly not enough and the company has decided to take a more aggressive approach – partnering with security vendors.

"Spam sucks. That’s why we’ll be integrating three new services over the next few weeks, to extend the current spam and malware protection we offer to our users," the bit.ly team announces on its official blog. These services are VeriSign's iDefense, Websense's Threatseeker and Sophos' behavioral-analysis engine.

VeriSign iDefense is a reputation-based service, which provides a blacklist of IP addresses, URLs and domain names that are known to have been used by cybercriminals. Websense Threatseeker is a Security-as-a-Service solution that works by analyzing the code of websites hosted at the submitted URLs in real time. Finally, Sophos' technology will be used to determine if content served from a URL is malicious by analyzing its behavior in a sandbox.

"bit.ly is committed to protecting its users from spam and malware. Services like [these] are an important part of building trust," Andrew Cohen, bit.ly's general manager, commented.