14 DAY TRIAL // Google has gotten rid of the dedicated Google News crawler and instead it is now using the main search one for News as well. While this means that Googlebot-News will no longer show up on your site, Google says that you can still treat the two types of visits differently and still block the news crawler while allowing the main search one.
Google says the move was necessary to consolidate its infrastructure, but that the change doesn't really affect most news websites.
"Google News recently updated our infrastructure to crawl with Google’s primary user-agent, Googlebot," David Smydra, Google News Product Specialist, announced.
"Any news organizations that wish to opt out of Google News can continue to do so: Google News will still respect the robots.txt entry for Googlebot-News, our former user-agent, if it is more restrictive than the robots.txt entry for Googlebot," he said.
Google wants to underline that this doesn't mean that it's all or nothing, you can still block only the news bot and be excluded from Google News and still show up in the organic search results.
It also underlines that the change does not affect regular user visits, analytics tools will know which users come from the search results and which from Google News.
"Sites that have implemented subscriptions using a metered model or who have implemented First Click Free will not experience any changes," further explains Google.
First Click Free is especially useful for Google News users, since it ensures that they can read articles they found via the aggregator, even if the site normally has a pay wall.
"For sites which require registration, payment or login prior to reading any full article, Google News will only be able to crawl and index the title and snippet that you show all users who visit your page," Google added.
This policy is consistent across Google, the company has very strict rules when it comes to 'cloaking' i.e. showing users one thing and the Googlebot another, even if it's for legitimate reasons.