Twitter relies on Amazon's Mechanical Turk to crowdsource searches

Jan 9, 2013 09:25 GMT  ·  By

Twitter's technological prowess is miles ahead of what it was just a couple of years ago. But technology can only help you so much, things happen so fast on Twitter that there's no time for algorithms to recognize patterns and come up with appropriate responses.

Search is the perfect example of this, Twitter is home to the latest news and trends, things change fast and it's hard for a computer to make any sense of it.

"One of the magical things about Twitter is that it opens a window to the world in real-time. An event happens, and seconds later, people share it across the planet," Twitter wrote.

"The queries people perform have probably never before been seen, so it's impossible to know without very specific context what they mean," it underlined one of the big problems with trying to understand real-time trends.

"We’ve built a real-time human computation engine to help us identify search queries as soon as they're trending, send these queries to real humans to be judged, and then incorporate the human annotations into our back-end models," Twitter explained.

This is why Twitter employs humans to help algorithms understand. Specifically, Twitter has a system that monitors trends. When it sees a search term that it doesn't understand spiking, it taps into Amazon's Mechanical Turk, the crowdsourcing service.

Mechanical Turk enables anyone to lend the help of actual people in completing a task, usually a computer-related task, though not necessarily.

In Twitter's case, it pushes the spiking query to Mechanical Turk and asks contributors for context, what it means, what it's related to, who are the people or photos that are relevant and so on.

The benefit in all of this is more relevant ads next to these searches, which benefit advertisers, Twitter and to a degree, users as well.