Inbound and Outbound email from lbl.gov was out of service from approximately 2pm to 5.30pm. Email delivery was restored at approximately 5:15pm and the backlog of message received and sent during the outage is expected to be processed and delivered by 8pm.
All emails sent from LBL during the outage have been delivered. Inbound emails may be delayed depending on how other sites attempt to redeliver mail. If you are experiencing missing or delayed messages as of 8AM on April 26th, please contact the IT Help Desk at help.lbl.gov
Technical Details:
Email flowing to and from Google transits the LBL network for cyber security and mail processing under normal mail flows. A datacenter switch in Building 50 experienced issues at 2:09PM. It was restored at 2:25pm.
At this point, the two mail handling and hygiene appliances that form the first step in LBL email handling did not return to service normally. Troubleshooting these devices began immediately. After rebooting the appliances they did not return to service as expected. At this point, the Email Team concluded that the queue needed to be rebuilt (to restore the state of these devices).
It took approximately 3 hours for the queue to be rebuilt, at which time mail began to flow. During the outage, a substantial backlog email built up (over 15,000 messages). IT anticipates that this queue will be delivered by 9pm at which point inbound and outbound mail should behave normally.
While most mail was queued during this period, there may have been a small number of lost messages. IT will be investigating and to the extent possible, will notify people about messages lost during this period.
Google is currently experiencing an issue which prevents users from login into Gmail, Calendar, Drive, etc. When users attempt to login, they are presented with a page asking them to clear cache/cookies. For progress on this issue, check this page or visit the Google Status Dashboard.
Some Windows users are reporting problems restarting their computers after applying the latest Windows updates (KB4493472 (Windows 7, Server 2008R2) and KB4493446 (Windows 8.1, Server 2012R2)).
If your Windows system is having trouble, here are the basic steps to fix the issue. Please note that you will need to have administrator rights on the computer and have working knowledge of the Windows operating system. Please feel free to Request Help if you would prefer to not fix the issue yourself:
Restart in safe mode, see https://go.lbl.gov/WindowsSafeModeBoot. Methods include:
Reboot and hold down the F8 key and select safe mode OR
Before reboot use the msconfig tool to set safe mode on startup OR
Before reboot use the command prompt and issue safe mode commands
Once in safe mode, disable the Sophos Anti-Virus service
Open the Control Panel
Choose System and Security
Choose Administrative Tools
Open the Service icon
Locate the Sophos service and open “Properties” by double-clicking
Choose “Disabled” for the Startup Type
Click OK to confirm your choice
Restart the computer normally (Boot into normal mode)
Uninstall the Windows KB4493472 or KB4493446 updates depending on your operating system
Open the Control Panel
Click “Uninstall a program” from the Programs icon
Select “View installed updates” on the left window pane
Scroll down the list and search for the update(s)
Highlight and click “Uninstall”
Confirm you want to uninstall the update by selecting “Yes” button
Click “Restart Now” button
Enable the Sophos Anti-Virus service
Open the Control Panel
Choose System and Security
Choose Administrative Tools
Open the Service icon
Locate the Sophos AntiVirus service and open “Properties” by double-clicking
Choose “Automatic” for the Startup Type
Click OK to confirm your choice
Restart the computer normally again for good measure
Users can minimize this sort of impact in the future by utilizing the IT User Support WSUS (Windows Server Update Services) server rather than Microsoft. Request this support via Request Help.
On April 29, 2019, the certificate for the LBL server that handles eduroam authentication will expire. Once this certificate expires, you will not be able to use eduroam. You will need to update your eduroam profile if you plan to continue using this service. After April 29, 2019, you will not be able to use eduroam until you update your profile.
What do I need to do?
To update your profile you must first remove your existing eduroam profile (if you have one) and download a new profile from http://cat.eduroam.org/.
If you have any issues with the installation or would like some assistance, please contact the IT Help Desk at [email protected] or (510) 486-4357.
