When it comes down to search engines it's never actually a matter of crawling before walking. Crawlers are specific tools associated with search engines designed to explore and index websites. In this context, Microsoft announced that it has evolved its crawler for Live Search, in order to improve efficiency. MSNbot taken to version 1.1 is the way Microsoft will implement a more efficient crawl, effectively taking Live Search at the same level as Google and Yahoo. There are two improvements delivered with the new MSNbot, namely HTTP Compression and support for Conditional Get.

"HTTP compression allows faster transmission time by compressing static files and application responses, reducing network load between your servers and our crawler. We support the most common compression methods: gzip and deflate as defined by RFC 2616 (see sections 14.11 and 14.39). Compression is currently supported by all major browsers and search engines. Use this online tool to check your server for HTTP compression support," revealed Fabrice Canel, Live Search Crawling Team.

In addition, Microsoft now supports "conditional get. Generally we will not download the page unless it has changed since the last time we crawled it. As per the standard, our crawler will include the 'If-Modified-Since' header & time of last download in the GET request and when available, our crawler will include the 'If-None-Match' header and the ETag value in the GET request. If the content hasn't changed the web server will respond with a 304 HTTP response," Canel added.

Although the improvements to Live Search are a welcomed evolution to the search engine's crawler, the fact of the matter is that Google and Yahoo have been supporting this functionality for some time now. Back in 2005, Yahoo was the first to support both HTTP compression and conditional get with its crawlers and Google followed one year later in 2006. So in this regard, Microsoft is simply catching up with its main rivals.