Twitter Unveils Massive Overhaul of Its Search Backend

Twitter has revealed that it had been rolling out a new search infrastructure for the past weeks. The transition is now over and Twitter says that the new backend scales a lot better, is significantly faster and comes with a few enhancements for the users as well.

"If we have done a good job then most of you shouldn’t have noticed that we launched a new backend for search on twitter.com during the last few weeks!," Twitter's Michael Busch, one of the people involved in the project, announced.

"One of our main goals, but also biggest challenges, was a smooth switch from the old architecture to the new one, without any downtime or inconsistencies in search results," he explained.

The new infrastructure has been re-written from scratch. The old MySQL-based system, inherited from Twitter's Summize acquisition, just couldn't cope with the amount of requests it was receiving.

So the team set out to build a modern search engine using an inverted index, just like any major engine out there uses. This makes for faster lookups and the possibility to handle a lot more tweets and a lot more requests than Twitter sees today.

The new technology is based on the open-source Lucene Java library which received some major improvements in some areas which would make it more suited for real-time searches, what Twitter specializes on. The company said that it will be releasing all of those changes as open-source as well.

Twitter's search volume has increased exponentially, along with the number of people and apps using the service. It must be noted that most of the searches Twitter sees are automated, coming from applications through the APIs the company provides.

This is why Twitter is calling them queries and not searches, to differentiate between purely human interactions, and all search requests.

Twitter now gets about 1 billion queries, per day, one and a half more than what it saw in April and the numbers will keep on rising. There are over 1,000 tweets per second, but 12,000 queries per second.

The new infrastructure is built for scale and Twitter says that it could handle as much as 50 times more queries than what the service is seeing today. Just five percent of the resources available for search are used right now.