On December 30th at approximately 12PM, fire alarm maintenance in the main networking data center resulted in an unexpected and unscheduled shutdown of all power to the datacenter including all backup power systems.
This resulted in a temporary complete loss of connectivity to the LBL site, outage to site telephone services, as well as outages to numerous servers/services provided from this datacenter.
Sitewide connectivity was restored within approximately 30 minutes, however, return of site services provided from this datacenter took several hours.
In addition, an intermittent problem with email delivery that resulted from hardware issues as a result of the outage was not fixed until January 3. The effect of this outage was that some outbound emails with attachments were delayed from Dec 30 till January 3. All queued email will be redelivered.
