Inside Info about Google's Test Results Quality

The one thing that Google really prides itself with, among all of the accomplishments it has achieved, is the relevance of its search results, the Search team always striving to hone the algorithm to fill the respective niche that might still be unattended on the off occasion.

This might sound like an easy job to do, but as a matter of fact it's a big pain in the behind for the people there, they always need the feedback, or they need a lot of people to check whether or not the results were the ones that suited the users by using the specific tools given to them by Google. Here's what Peter Norvig had to say on the matter:

"We test it in lots of ways. At the grossest level, we track what users are clicking on. If they click on the number-one result, and then they're done, that probably means they got what they wanted. If they're scrolling down, page after page, and reformulating the query, then we know the results aren't what they wanted."

"Another way we do it is to randomly select specific queries and hire people to say how good our results are. These are just contractors that we hire who give their judgment. We train them on how to identify spam and other bad sites, and then we record their judgments and track against that. It's more of a gold standard because it's someone giving a real opinion, but of course, since there's a human in the loop, we can't afford to do as much of it", he concluded.

That's a bummer, no way to continuously improve automatically, the algorithm can only help this much and tweaking it might make the search results more relevant for some, while others would still be confronted with the same problem of not finding what they were after when they clicked the "Search" button.