Inbound and Outbound email from lbl.gov was out of service from approximately 2pm to 5.30pm. Email delivery was restored at approximately 5:15pm, but there is a substantial backlog of messages to process. We anticipate that all email in the queue will be delivered by later this evening.
Technical Details:
Email flowing to and from Google transits the LBL network for cyber security and mail processing under normal mail flows. A datacenter switch in Building 50 experienced issues at 2:09PM. It was restored at 2:25pm.
At this point, the two mail handling and hygiene appliances that form the first step in LBL email handling did not return to service normally. Troubleshooting these devices began immediately. After rebooting the appliances they did not return to service as expected. At this point, the Email Team concluded that the queue needed to be rebuilt (to restore the state of these devices).
It took approximately 3 hours for the queue to be rebuilt, at which time mail began to flow. During the outage, a substantial backlog email built up (over 15,000 messages). IT anticipates that this queue will be delivered by 9pm at which point inbound and outbound mail should behave normally.
While most mail was queued during this period, there may have been a small number of lost messages. IT will be investigating and to the extent possible, will notify people about messages lost during this period.
