14 DAY TRIAL // The inboxes for over 17,000 Windows Live Hotmail users were deleted on December 30th, 2010 and Microsoft is blaming an automated test for the issue, which, ironically enough, was designed to help the company monitor the health of the email service.
According to Mike Schackwitz, Windows Live Hotmail team, it’s standard practice at the Redmond company to create accounts, configure them and run automatic tests against them in order to simulate normal user behavior, in an effort to detect errors and fix bugs.
Obviously, employees use scripts as a part of the testing process, especially considering that Windows Live Hotmail accounts are tested in bulk.
The end of the process is synonymous with the removal of test accounts, a task which is done by deleting the records from a group of directory servers which serve to direct all users as well as incoming mail to the destination mailboxes.
It appears that one of the scripts backfired on Microsoft, and caused problems for a range of customers which logged in just to find all of their content missing from Hotmail inboxes.
“On December 30th, we had an error in a script that inadvertently removed the directory records of a small number of real user accounts along with a set of test accounts.
“Please note that the email messages and folders of impacted users were not deleted; only their inbox location in the directory servers was removed.
“Therefore when they logged in, a new mailbox was automatically created for them on a new storage server that didn’t contain their old messages and folders. This is why the accounts received the “Welcome to Hotmail” message,” Schackwitz explained.
Microsoft first became aware of the problem on December 30, but an initial analysis failed to get to the source of the issue.
Furthermore, the glitch was only flagged for investigation on December 31st, and because something similar had never happened, it was assigned for fixing to the wrong team.
Users must understand that 17,000 accounts is just a drop in the ocean for Windows Live Hotmail, which currently houses in excess of 1 billion inboxes.
This is why the issue received a low priority rating, which was only elevated on January 1st, 2011. Just a day later the issue had been fixed.
“Our first step was to restore these users’ entries in the directory servers, which we did by early on the morning of January 2 PST.
“We then merged their old email messages and folders with any new mail they’d received throughout the day on January 2nd. This required multiple passes to capture all the accounts and messages, so for some users, service wasn’t completely restored until January 5th.
“We completed the merge for 16,035 users on January 2nd and by January 5th had completed this for the remaining 1,320 users who were affected by this particular issue,” Schackwitz added.
Those affected now have all of their emails restored, with the software giant reporting that no data was lost. At the same time, all of the custom folders created by users have also been recuperated.
“The only unfortunate exception to this statement is that, if you were affected by this incident and you didn’t sign in to your account between the time of the incident and the time your account was restored, then any messages sent to your account during that time would have bounced,” Schackwitz said.