14 DAY TRIAL // Microsoft is in the process of migrating hundreds of millions of Windows Live Hotmail users and their accounts to a new storage system delivering better performance, enhanced reliability, superior disaster recovery capabilities and more scalability than before.
Kristof Roomp, an architect in the Hotmail team explains that the Redmond company is implementing some changes to the infrastructure that powers Hotmail. Essentially, while the software giant’s technologies, such as Windows Server and SQL Server continue to be at the backbone of the email service, the new storage system no longer uses RAID (Redundant Array of Inexpensive Disks) hard drives.
This is only part of the evolution story of the new Hotmail storage system. Microsoft is also starting to implement newer SSDs, or Solid State Drives, in addition to the older HDDs.
The rollout has already received green light after the company tested the Hotmail upgrade on a cluster that housed the accounts of Microsoft volunteers.
The accounts of some 30 million Hotmail users have already been migrated to the new storage system, with over 100 million planned to join them in the next months.
“The new system ensures that the copies of data reside on independent hard drives, controllers, and machines. This kind of system is nicknamed “JBOD,” which stands for “Just a Bunch Of Disks.” In a JBOD system, the hard drive controller almost completely gets out of the way, which means that the software must now worry about all the failures that the controller previously handled,” Roomp explained.
“In addition, the software must now scrub the drives periodically to check the data for “bit rot” (i.e., data that has for some reason become unreadable or corrupt). So basically, we built a distributed "RAID" controller completely in software, which replaces the industry-standard firmware ones.”
JBOD storage makes it extremely unlikely that user data will ever be lost, since the system is aware of exactly how many copies of user emails exist, and can prioritize their recovery in the eventuality of a failure, through automatic repair processes.
Another benefit is related to data availability, with Hotmail being able to have the users’ content move to a location which allows the service to function under normal parameters, even if a potential issue has yet to be resolved.
While still relying on traditional HDDs, the Redmond company is also embracing Flash Storage. However, while not denying its superiority in terms of performance, Microsoft stresses that the cost associated with SSDs impede its adoption at a much larger scale.
“In addition to storing the email messages themselves, we also track information about these messages (called metadata), such as the list of messages in your inbox, read/unread status of your messages, conversation threading, mobile phone synchronization etc. This metadata takes up an extremely small fraction of our total storage space, but due to its constantly changing nature, it is responsible for most of the load on our hard drives,” Roomp added.
“By using SSDs for this small and rapidly changing set of data, and using the largest hard drives available for storing messages, we are able to take advantage of the trend in larger and cheaper hard drives without making any sacrifices in the performance of our system.”