A very small number of Gmail users temporarily lost their Gmail accounts starting on Sunday. Most of them had their accounts back by mid-day Tuesday, the company said.
Ben Treynor, Google’s (News - Alert) vice president for engineering and site reliability czar, blamed the outage on a “bug.”
According to a blog post he made on Monday, 0.02 percent of Gmail users temporarily lost their accounts.
“The good news is that e-mail was never lost and we’ve restored access for many of those affected,” he said. “Though it may take longer than we originally expected, we're making good progress and things should be back to normal for everyone soon.”
As of 12:20 PM PST on Tuesday, data for the remaining 0.012 percent of affected users was restored from tapes and was being processed, according to Google.
“We plan to begin moving data into mailboxes in two hours, and in the hours that follow users will regain access to their data. Accounts with more mail will take more time,” Treynor said.
As a precaution, Google has multiple copies of data in multiple data centers. But that wasn’t enough in this case.
“In some rare instances software bugs can affect several copies of the data,” Treynor explains. “That’s what happened here. Some copies of mail were deleted, and we’ve been hard at work over the last 30 hours getting it back for the people affected by this issue.”
In anticipation of bugs, Google also backs data up to tapes.
“Since the tapes are offline, they’re protected from such software bugs. But restoring data from them also takes longer than transferring your requests to another data center, which is why it’s taken us hours to get the email back instead of milliseconds,” he explains.
The bug came after Google released a “storage software update.”
“When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version,” he said.
If this temporary outage affected a Gmail account, e-mails sent to the users’ inbox between 6:00 PM PST on Feb. 27 and 2:00 PM PST on Feb. 28 was likely not delivered to a mailbox. In addition, senders received a notice their messages weren’t delivered.
This is not the first outage for Gmail. Gmail saw a short outage in November that kept users out of their e-mail and Google Docs accounts for an unconfirmed period of time, according to a report on TMCnet.
There are about 170 million Gmail users worldwide.
Ed Silverstein is a TMCnet contributor. To read more of his articles, please visit his columnist page.
Edited by Janice McDuffee