Inbound and Outbound email from lbl.gov was out of service from approximately 2pm to 5.30pm. Email delivery was restored at approximately 5:15pm and the backlog of message received and sent during the outage is expected to be processed and delivered by 8pm.
All emails sent from LBL during the outage have been delivered. Inbound emails may be delayed depending on how other sites attempt to redeliver mail. Additional information about the outage is available on the IT Website. If you are experiencing missing or delayed messages as of 8AM on April 25th, please contact the IT Help Desk at help.lbl.gov
Technical Details:
Email flowing to and from Google transits the LBL network for cyber security and mail processing under normal mail flows. A datacenter switch in Building 50 experienced issues at 2:09PM. It was restored at 2:25pm.
At this point, the two mail handling and hygiene appliances that form the first step in LBL email handling did not return to service normally. Troubleshooting these devices began immediately. After rebooting the appliances they did not return to service as expected. At this point, the Email Team concluded that the queue needed to be rebuilt (to restore the state of these devices).
It took approximately 3 hours for the queue to be rebuilt, at which time mail began to flow. During the outage, a substantial backlog email built up (over 15,000 messages). IT anticipates that this queue will be delivered by 9pm at which point inbound and outbound mail should behave normally.
While most mail was queued during this period, there may have been a small number of lost messages. IT will be investigating and to the extent possible, will notify people about messages lost during this period.
