Testing Is Ramping Up on Bing's New Crawler

Bing has brought hope to Microsoft's search division and, while the initial traffic numbers don't show much growth and are sometimes conflicting, overall the signs are positive and the product is seen as a success. But with all the focus on the user interface it's easy to forget that a search engine is only as good as its results so development is ramping up on a new crawler bot to handle the indexing.

“We are busy working on our improved crawler and plan to ramp up its workload as we move toward the goal of putting it into full production. As a result, webmasters may begin seeing an increased amount of traffic from the new bot on their sites,” Rick DeJarnette from the Bing Webmaster Center, wrote on the Bing blog. “We do not anticipate any problems related to our increasing emphasis on MSNBot 2, but the unexpected can’t always be avoided, no matter how hard you try!”

The new bot, dubbed MSNBot 2, was announced in December 2008 and has been popping occasionally in webmasters' logs during testing. Now with development approaching the finish line Microsoft is giving webmasters and website owners a heads up on the increased traffic they are likely to experience. The full user agent string will be “msnbot/2.0b (+http://search.msn.com/msnbot.htm)” and the crawler will keep its old name for consistency's sake.

However, since the new bot is still very much in development the older version is still the one being mainly used and will continue to be so for some time coming. The older version will show up as “msnbot/1.1 (+http://search.msn.com/msnbot.htm).” As testing is increasing on the new version some problems are bound to show up so the development team is advising those who may come across one to report it to the Bing support team or on the crawling/indexing forum.