Twitter Turns to BitTorrent for 200 Times Faster Server Updates

As sites get larger, scalability becomes more of an issue. For sites with a modest audience, there are tried and tested tools, but for the biggest sites on the web, there are very few existing options. They’re venturing into unknown territory and have to come up with solutions for problems that few if any have faced before. Yet, there are times when all you need to do is figure out how to apply existing technology to solve your particular problem. This is what Twitter has done having adopted BitTorrent as a means of distributing software updates to its thousands of servers in mere seconds.

Twitter used to rely on a Git-based system. Updates would be pushed to the repository and the servers would then sync to that. The problem with this centralized approach is that it doesn’t scale, certainly not at the level that Twitter needed. And as more servers were added the problem got worse.

“It was time for something completely different, something decentralized, something more like.. BitTorrent.. running inside of our datacenter to quickly copy files around. Using the file-sharing protocol, we launched a side-project called Murder and after a few days (and especially nights) of nervous full-site tinkering, it turned a 40 minute deploy process into one that lasted just 12 seconds!,” Larry Gadea, a Twitter engineer, wrote.

A flock of crows is called Murder, which is where the collection of scripts got its name. Written in Python and Ruby, it leverages the BitTorrent protocol and optimizes it for the particularities of a data center, high-bandwidth, low-latency and so on. Twitter uses a BitTorrent client built on top of the open-source BitTornado. The company has released Murder under an open license as well.

For Twitter, the advantages are obvious and the speed of deployment speaks for itself, BitTorrent is 200 times faster than the previous solution. Twitter is not the only one to tap into the peer-2-peer protocol, Facebook is also using BitTorrent for the very same thing, distributing updates to its servers efficiently.