Google Now Indexes Content Behind JavaScript and AJAX

Google is now confirming what people were already seeing, it is now indexing Facebook comments. In fact, that's not even half of it, it is now indexing a lot of content that is hidden away behind JavaScript and AJAX.

This means that Google is able to to index comments made via Facebook Comments, Disqus and other platforms.

But it means more than that, there are plenty of websites going for a full JavaScript implementation, often without proper justification, and plenty of sections in which content is generated via JavaScript.

Google should now do a much better job of discovering and indexing content in any of those cases.

"Googlebot keeps getting smarter. Now has the ability to execute AJAX/JS to index some dynamic comments," Google's Matt Cutts revealed on Twitter.

Google has also put up a blog post explaining the change in greater detail, albeit in rather technical terms.

"Now, especially with the growing popularity of JavaScript and, with it, AJAX, we’re finding more web pages requiring POST requests -- either for the entire content of the page or because the pages are missing information and/or look completely broken without the resources returned from POST," Google explained the problem.  

"For Google Search this is less than ideal, because when we’re not properly discovering and indexing content, searchers may not have access to the most comprehensive and relevant results," it said.

Google has devised a method for the GoogleBot to run some JavaScript code and make POST requests without affecting the site being crawled.

It says it only chooses to make POST requests when those requests are automated, i.e. they would run any time a user would visit the page, without any input from the user.

For many years, a big challenge for Google and other search engine was indexing Flash content. In recent years, this hasn't been a problem, with the help of Adobe and with better tools from Google, basically any Flash content can be indexed. But that doesn't mean there aren't any new challenges and no more content that Google can't get to.