Search engines have come a long way since the early days of AltaVista. The issues Google, Yahoo and Microsoft tackle these days are very complex in some instances, yet one problem has lingered as technology progressed. In fact, it's precisely because technology progresses that search engines have this problem. Even now, a significant portion of the web is inaccessible to search engine crawlers and it's not because webmasters don't want their pages to be seen. Flash content was completely obscure to search engines for many years and, despite this becoming less of a problem recently, there are still issues. Things are even worse for dynamic pages implemented using AJAX as search engines have problems properly reading the majority of them. This though is changing, soon if Google gets its way.

Last year, Google has come up with a proposal to make AJAX website search friendly. Google said that the aim was to make more AJAX content available to its crawler which, in the end, benefits the webmasters as much as it does the users and Google itself. Search Engine Land has a thorough article dealing with the problem of indexing AJAX code, but it also notes that Google may have gone ahead and implemented its proposal in the main search engine.

"We believe that making this content [from AJAX-based websites] available for crawling and indexing could significantly improve the web," Google wrote at the time. " While AJAX-based websites are popular with users, search engines traditionally are not able to access any of the content on them. The last time we checked, almost 70% of the websites we know about use JavaScript in some form or another. Of course, most of that JavaScript is not AJAX, but the better that search engines could crawl and index AJAX, the more that developers could add richer features to their websites and still show up in search engines."

The proposal itself isn't overly complicated and all it takes on behalf of the web developers is to modify the existing code a bit to Google's specification. It also proposes webmasters set up a server-side headless browser which would process the JavaScript code and render the HTML version. This version would then be crawled by the search engine giving it access to the full content on the page. Back in October, Google said it was still working on the proposal but also on a prototype implementation which is apparently live by now. Some web developers have already implemented the changes Google proposed, but it may be too early to start doing the same. Still, Google seems determined to go through with it.