In case you were born yesterday and have learned to use the computer since, I'll let you in on a little inside info: Google is the "lifeblood of the Internet," in the words of Barak Obama. 75% of the Internet searches are conducted through it, it's going into the mobile business and it's basically a moneymaker.
What else could go the
right way for the Mountain View based company? Erm… it could be favored by the web admins… and it is. (note to self: must think of something else that would benefit Google and then send them an email with the idea… possible money-making deal)
A team of Penn State researchers have created a search engine called BotSeer and used it to examine more than 7500 web sites. The results were of such a nature that a pro-Google bias was found in terms of which search-engine web crawlers were or were not allowed access. "We expected that robots.txt files would treat all search engines equally, or maybe disfavor certain obnoxious bots, so we were surprised to discover a strong correlation between the robots favored and the search engines' market share," said C. Lee Giles, the David Reese Professor of Information Sciences and Technology at Penn State who led the research team that developed BotSeer, in a statement. "Robots.txt files are written by Web policy-makers and administrators who have to intentionally specify Google as the favored search engine," Giles said.
In case you didn't know it, robots.txt files are used to regulate web crawlers and to help prevent servers from getting overloaded. The research found that about four sites in 10 use this type of file, as opposed to the previous data from 1996, when just one in ten did it. Another interesting result was that although Yahoo! and MSN were also given greater than average access to web sites, Google outnumbered any of the two by showing twice as many times.
(note to self: should send an email to Google about it running for President. No one would see that coming.)