14 DAY TRIAL // It turns out that Microsoft is indeed boosting the crawl rate of the bot associated with its search/decision engine.
Earlier this week, based on a Microsoft adCenter report, I told you that the Redmond company was kicking up a notch the speed at which the Bing crawler visited websites.
It was a small case of miscommunications, as it turns out that the software giant was actually talking about adCenter Ad Index (adidxbot) and not the Bing bot, MSNBot.
I’ve corrected the slip since then, and added an explanation from a member of Microsoft adCenter.
But now, Sasi Parthasarathy, Senior Program Manager, Bing Engineering, revealed that webmasters will see an increase of the Bing bot crawl rate.
“In our continuing efforts to provide great value to our publishers, the Bing Engineering team is constantly testing new features to improve crawl efficiency. Such experiments are vital to our ability to innovate and improve our end user and publisher experience,” Parthasarathy explained.
“Due to an upcoming experiment, a few sites may encounter an intermittent increase (anywhere from 5% to 25%) in crawl requests from MSNBot over the next month (specifically from August 25th through end of September).
“If you see an increased crawl rate on your site and want to throttle it down, please contact bwmc at microsoft.com as soon as possible, and we will address your concerns. Thanks for your understanding as we try to improve Bing for all of us,” Parthasarathy added.
MSNBot 1.1 will only survive until October 1st, 2010, at which time Microsoft will discontinue it and introduce the new Bingbot (MSNBot 2.0b), which will drop its current Beta label.
The adidxbot crawl rate increase is a tad more consistent compared to the Bing bot, up to twice as much.
“The crawl rate increase [for adidxbot is] only applicable to a very small subset of adCenter advertisers with very specific domains that were identified as being good candidates (e.g. able to handle the temporary crawl rate increase) as part of an ongoing effort to improve relevance,” revealed Tina Kelleher.
“Please note that if your site should be identified as fitting this same criteria, you can expect to hear directly from the adCenter Team with a request for permission to increase the adidxbot crawl rate before any action is taken.”