We experienced an unscheduled outage of all authentication services including login.lbl.gov, ldap, OTP/pledge and other directory services on Christmas morning. Network engineers resolved the outage.
Networking and Telecom will take numerous short (1-2 hour) outages December 29-30 and one longer outage. The longest of these will take place December 29th which will impact AT&T mobile phone access, and traditional telephones at offsite buildings and the guest house, as well as VPN. The outage window for this outage is based on the expected time to complete electrical upgrades by Facilities and may be extended if the work proves more challenging than anticipated.
In addition, there will be numerous after hours outages on Friday December 18th to facilitate the work on Dec 29-30.
Current scheduled work in the Infrastructure Services area for the Dec-2015 winter break includes:
Tue, 29-Dec (tentative), times TBA -- Shutdown of 50A-1156 e-power to install branch circuit metering. We expect an outage of up to 2 hours for UPS-backed power and 1 hour for circuits that are just generator-backed during this installation. Most IT systems in 50A-1156 should be on dual power feeds before this time, so we expect the outage to impact:
All AT&T mobile phone and wired circuits (including the Guest House and offsite buildings)
WiFi service to the Lab
Remote access VPN service
Some building networking, including Blackberry Gate, Bldg’s 4, 48, 75, 75a, 75b, 85, and Keck Obeservatory
Some limited CPP equipment
Tue, 29-Dec, times TBA -- maintenance to primary border router er1-n1 and ALS Science DMZ router sr1-als. We will have shifted traffic to the backup border router before this work, so the only outage impact should be for Science DMZ DTN’s (managed by HPCS and ALS)
Tue, 29-Dec, times TBA -- maintenance to DNS anycast service. This may cause short (< 1 minute) interruptions to DNS resolution using the 131.243.5.1 anycast address.
Tue, 29-Dec 0800 - 2200, and Wed, 30-Dec 0800 - 2200 -- maintenance to non-data center switches at LBL. This will usually mean a ~ 5 minute outage per switch as it reboots into the new operating system, although some switches can take up to 30 minutes to upgrade their firmware and return to normal operation.
Tue, 29-Dec, times TBA -- maintennace to ir1-n1 and ir3-n2 routers and new larger power supplies for ir1-n1. This will mean a 10-30 minute outage for all routing for all LBL networks other than those at the ALS, at JGI, or in Zone 3 (buildings 31, 62, 66, 67, 72, 74, 77, 83, 84, 85, 86, and Strawberry Canyon gate house).
Tue, 29-Dec or Wed, 30-Dec, times TBA -- maintenance to LBLnet services switch. This may cause short (usually < 5 minutes) outages for network services such as ntp, dns, onestop, iprequest, etc.
Tue, 29-Dec or Wed, 30-Dec, times TBA -- uplink migration from end-of-life equipment for Buildings 1, 4, 14, 31, 55, 56W, 64, 77, and 86. Expected 5-10 minutes outages for these buildings as their uplinks move from old end-of-life equipment to current equipment.
Tue, 29-Dec or Wed, 30-Dec, times TBA -- maintenance to Bldg 84 switch. Expected 10-30 minute outage while this equipment is replaced.
Tue, 29-Dec 0800 - 2200 -- replace primary Bldg 978 network switch. Expected impact is several hours of downtime for most building network connections.
Wed, 30-Dec 0800 - 2200 -- replace primary Bldg 977 network switch. Expected impact is several hours of downtime for most building network connections.
Wed, 30-Dec 1200 - 2200 -- replace primary JGI Bldg 100 non-sequencer network switch. Expected impact is 2-3 hours of downtime for impacted 128.3.89.0 subnet.
- Dates/times still being negotiated -- maintenance to IDM and TSC subnets.
The scheduled power work of December 18, 2015 will impact authentication to Lab services, email delivery and access to eRoom and Sympa lists starting at 5:30 PM. Authentication to Lab services and email delivery will be impacted for approximately 15 to 30 minutes; access to eRoom and Sympa lists will be impacted for approximately 30 to 60 minutes.
