MySpace Open Sources Internal Distributed Computing Framework

It looks like the two biggest social networks in the US are getting into an open-source war as, just days after Facebook released the Tornado framework, MySpace is releasing its own piece of open software with the Qizmt framework, designed to handle large amounts of unstructured data, which is now powering the “People You May Know” feature.

“Today, we are open-sourcing Qizmt, an internally developed framework for distributed computation created by the Data Mining team here at MySpace. Qizmt can be used for many operations that require processing large amounts of data such as collaborative filtering for recommendations and analytics,” says the announcement that came from Mike Jones, MySpace's chief operating officer himself. “Qizmt is a powerful MapReduce-based environment that enables MySpace user recommendation engines to become smarter, faster and more reliable. [...] Qizmt is unique because it was developed using C#.NET specifically for Windows platforms.”

The technology is based on Google's own MapReduce distributed computing framework, which it maintains and uses extensively across several services including its search indexing. What makes Qimzt interesting is the fact that it’s written using Microsoft's .NET framework and is designed to be run on large clusters of Windows machines, something rather uncommon in distributed computing especially at this level.

Qizmt is currently used by the social network to power “People You May Know” but will begin to see a wider rollout in the future. MySpace hopes that opening the technology to outside developers will help it grow faster but the decision was surely helped by the fact that many similar technologies at this level are open-source. Also, by using .NET, MySpace believes that it should give budding developers a much easier way of understanding MapReduce and how to develop for it.

Interestingly, FriendFeed, or rather Facebook, also open-sourced a distributed computing framework earlier this week, the Tornado framework for Python, designed to handle massive amounts of data in real time. MySpace, though, claims the two technologies have very different scopes and, as such, don't make for a good comparison.