Plans to intensify development efforts for Hadoop

Oct 2, 2009 13:30 GMT  ·  By

Yahoo is a big company with a lot of interests and products, so many that it's been closing down many of them for the past year or so. But with the latest focus on a more unified vision and better integration between products Yahoo has been heading more towards becoming a media company. And with the recent sale of its search unit to Microsoft many are wondering if Yahoo hasn't lost its roots as a Silicon Valley startup. The company's Shelton Shugar though says that, despite appearances, it's still pretty much a technology company.

"What impact does the Microsoft search deal have on the Yahoo! cloud? It has no impact," Shugar, Yahoo's senior VP of cloud computing, told The Register. "All the services we're building, we will continue to build. All the roadmaps we have in place, we will continue to work towards."

The cloud has become somewhat of a buzz word lately despite being a very loosely defined concept with experts having different opinions on what exactly a 'cloud' is. For large companies like Yahoo, Google and recently Facebook, the cloud means a flexible infrastructure and common components that can be adapted to any number of services and products each with different computing needs.

Google has its own, well documented, but closed set of services to handle its cloud computing needs. A while back though, it made available the general outline of some of its core components, allowing others to begin implementing an open-source alternative that would eventually become Hadoop. In an ironic twist perhaps, Yahoo is a main driving force behind the open-source project while Google still keeps its technology to itself.

After the announcement that Yahoo is ditching its Search business many were left wondering what would be the fate of Hadoop, which was used extensively by Yahoo to power the search technology. But Shugar insists that Yahoo will not scale down its involvement; if anything it plans to hire even more people to work on the technology. "There will be no backing away from Hadoop. You may even see an acceleration," he added.

Currently Yahoo runs Hadoop on some 25,000 servers and Search, despite being a major component, “is not the majority user" as the company runs a variety of other services using the cloud technology. It is also planning to develop related technologies like an SQL-like language for Hadoop. Facebook, which also uses Hadoop extensively to power its rapidly increasing computing needs, has developed a similar language, Hive, but Yahoo believes that it needs to create one that better suits its needs. Still, while Yahoo seems determined to have a very powerful tech team, it's too early to tell how all this will play out in the long term.