Yahoo has announced its plans to release an open-source version of its Hadoop implementation, called Yahoo! Distribution of Hadoop, the software it uses as a backbone for many of its products. Hadoop is an open-source framework based on Java and used for distributed computing for large-scale businesses. It is based on Google's computing infrastructure, used to power its indexing services, and has just been made publicly available.

The framework is developed under the guidance of the Apache Software Foundation and Yahoo has been using and developing its own internal version based on the software. The Internet company made the announcement at the Second Annual Hadoop Summit in Santa Clara, California.

“We know from our own experience serving half a billion users worldwide requires large-scale distributed systems, and a growing number of other companies and organizations are in need of similar capabilities,” said Shelton Shugar, senior vice president of cloud computing at Yahoo. “Yahoo! was a pioneer in developing the Hadoop technology and now Hadoop provides Yahoo! key software infrastructure running on tens of thousands of machines to process data critical to Yahoo!’s core business. By making the Yahoo! Distribution of Hadoop generally available, we are contributing back to the Apache Hadoop community so that the ecosystem can benefit from Yahoo!’s quality and scale investments.”

Yahoo has been a major backer of Hadoop since early in its development and in 2006 Hadoop founder Doug Cutting joined the giant company to further the development of the software. The company deploys the framework on 25,000 servers handling massive amounts of data every day with Yahoo! Search and Yahoo! Mail being the biggest products to use it.

“It’s exciting to see how the Apache Hadoop project has progressed in the last few years,” said Justin Erenkrantz, president of Apache Software Foundation. “We’re looking forward to seeing community growth accelerate with Yahoo!’s continued support and focus on quality, ultimately driving more contributors to the Apache Hadoop project.”