Blog

Some of our institutional systems will be unavailable Saturday due to electrical work in the IT Data Center.

The planned outage window is from 12 noon to 5PM, although some applications will be impacted for a much shorter period of time.

At this point, the applications that will be impacted for all or part of the outage include:

Facilities Maximo application

• AMS (Sunflower Asset Management System)
• BRS/Cognos
• All centralized Windows file (CIFS)services (shorter down time and recover automatically)
• Commons ( ~ 1 hr) - includes a number of web sites and wiki's  (including facilities, HR, IT, the Lab RPM)
• eBuy
• IRIS
• LETS
• Maximo
• Peoplesoft HRIS
• Peoplesoft FMS
• Peoplesoft TREX
• Sophos Antivirus Server ( ~ 1 hr)
• Webspace
• Windows printing - LBLPrint1, 2, 3, and Engineering ( ~ 1 hr )

Background

On Thursday, July 11 at 2:30pm, the Lab conducted a real evacuation of the hill for selected buildings.  In parallel with this event, all lab employees were encouraged to check-in to report on their personal status as part of the drill.  Some groups used traditional phone trees, text messages or email to report  out, while others used a Google based application to record similar information (and some used a combination of both methods).  IT offered to develop the Google Application during the planning process for the exercise in the hopes of reducing the burden on Divisions.  

The Check-in Program

The application was designed to be a pilot, not the eventual application that the lab might make use of in a real emergency - it was designed to help us learn about how people would use the application in an emergency.  Because of the very short time between our offer to develop the application and the drill, we implemented the program in Google Apps Script, a scripting language that is provided as part of the Google Apps Suite which generally provides a reasonably quick platform for simple development projects.  

The application we implemented allowed people to to check themselves in, and also provided supervisors with the ability to report on members of their group. In addition, an effort was made to provide feedback on progress by Organization, Building or supervisor level. The application was tested and worked well - until the actual evacuation drill.  Limitations on resource usage (implemented on Google's end and undocumented) caused the application to stop processing new entries very early in the exercise.  

As a result, out backup plan was brought into operation 8 minutes into the drill (at 2:38pm).  A simple Google form with a Google docs spreadsheet as the data collection engine was developed. Over 2400 entries were subsequently recorded and reported on using this method. While it was effective in collecting results, we did not have the ability to report out by supervisor, building, and division during the early part of the exercise.  We implemented this as the exercise continued, eventually delivering reports by Division by around 6pm and further refining them by 9pm and then again the next morning.   By this time, most of the interest in the data was at the division-wide level and few supervisors saw the reporting roster.

Lessons Learned

This experience gave us two sorts of lessons, one set about Apps Script and the other about what functionality might be in a future application.

In terms of Apps Script, we need to determine whether applications can be implemented to scale appropriately to hundreds of concurrent users.  We have already had discussions with the Project Manager of Apps Script about this. More broadly, any future production application for this purpose would go through the normal IT development process with direction from the functional owner (protective services).  We will of course continue to favor applications/platforms that are hosted externally on robust infrastructure to support disaster recovery.

In terms of lessons learned about functionality, two things stood out to us.  Many users complained about the difficulty of logging in to the application.  Finding the right balance between ensuring that users of the application are really who they say they are, and making it convenient to check in is an important balancing act, one which a future application needs to address.  This call would be made by PS, but from an IT perspective, our sense is that allowing unauthenticated checking, but requiring a login to view other data is a good balance.   This leads to the other observation: there were many requests for access to the data that didn't follow the model of supervisors and hierarchy.  While this was almost certainly related to the failure of the original application, in a real emergency, there would no doubt be many people with different needs to access the data - as a general rule, hierarchies aren't that useful in emergencies. In addition, quirks of LBNL like matrixed employees further complicate reporting.  A flatter data model (all logged in users can view all data for the lab) might be the right way to go.  

IT would like to apologize for the confusion resulting from the issues with the application.  We made substantial efforts to QA and test the application in advance, but we did not expect the application to trigger Google App Script use limits the way that it did.  We look forward to the opportunity to support a labwide checkin application in the future if this work is supported by PS and the lab community. 

Both Google and Smartsheet have improved their Forms capabilities.

Google Forms: 

Please join MathWorks at complimentary MATLAB training seminars for Lawrence Berkeley researchers, staff and students. The  event features two technical sessions presented by a MathWorks engineer:

10:00 a.m. – 12:00 p.m.: Session 1: Introduction to Data Analysis and Visualization with MATLAB

Learn how to acquire, analyze and visualize data through mathematical, statistical and engineering functions. 

1:00 p.m. – 3:00 p.m.: Session 2: Optimizing and Accelerating Your MATLAB Code

Learn simple ways to optimize your code and create parallel applications to reduce the execution time of computationally intensive MATLAB applications.

Go here  for more information and to register for the event (not required, but helpful for event planning purposes)

Most HPC resources including Lawrencium LR3 and other compute clusters in the Bldg 50B-1275 datacenter will be unavailable during the May 18 weekend due to facilities work to repair a leak in the cooling systems. Job scheduler reservations and reduced access to cluster resources will be put in place from Friday, May 17th 5:00 pm to Monday May 20th Noon; however, we have made accommodations to continue to run jobs for users with critical deadlines. Systems that are still available during this weekend include Lawrencium LR2, JCAP, Riemann, Vulcan, Catamount, and Voltaire. Please email [email protected] if you have any questions or concerns.

Google has announced a unified storage plan in which storage is allocated in one bundle to their Google Gmail, Drive and Google+ services.  We expect to see this roll out to Enterprise customers after May 21.  

5/24/13 - Note:  The rollout is now expected to start May 28)

(credit to Google for the above image)

"As a result of today’s storage change, Gmail inboxes for Google Apps customers are no longer limited to 25 GB - any additional storage you purchase can be shared and used by Gmail. Or alternatively, if you’re only using a few gigabytes of email storage, but have a lot of large documents and files stored in Google Drive, you can now use your storage primarily for Drive."

This means everyone at the lab will have 30GB to use however they wish. Those of you who are close to the current maximum of 25GB in Gmail will now have some additional space (assuming you do not use it in Drive).  

We have around 100 customers at the lab who will benefit from this from a GMail perspective. Many more will benefit from having more Drive Space (many Gmail users are never close to 25GB).  In addition, additional storage can be purchased which effectively  eliminates any  need to periodically delete messages when the max limit has been reached.

For more information - check out the Google Blog article .

Google Docs Performance Review Seminars Scheduled for May 8 and 10

Many divisions are moving to an automated performance review process this year, using a Google-based platform. If your division is participating, you will receive a link to your self assessment form.  The IT Division will offer Google Docs Performance Review Seminars on May 8 and 10 from noon to 1 p.m. in the Building 50 Auditorium. If you need help with completing your self assessment using Google Docs, or want more information about the automated process, be sure to attend. You do not need to register in advance.

Topics

• Goals (why groups opted in to Google Docs as a platform for the Performance Management Process)
• Participants (which Divisions/Departments are going Google)
• Process Overview  (the big picture - what employees and supervisors can expect)
• Google Docs use at the lab (a bit of context - a look at recent stats concerning the use of Google Drive/Docs)
• Google Docs demo (typical tasks involved with updating content in a Google Doc, adding comments, sharing, exporting, etc)

An unscheduled power outage took all systems in the main LBNL datacenter down shortly after midnight on April 29.  Cascading impacts also impacted some services outside the datacenter.  Production business systems and most other services were restored by IT personnel during the night, though some services were not restored until early working hours on April 29.   HPC services will gradually be brought online throughout the day.  The outage caused at least one major hardware failure, which will delay some storage services for HPC users until April 30 when a replacement unit can be delivered (affected customers are being separately notified).

The root cause of the power disruption has not yet been identified.

Update: This appears to be resolved as of ~6:20AM Pacific

Google Apps Services (gmail, calendar, docs, etc) are down this morning due to issues at Google.  There is no ETA, though past experience suggests these outages have an average duration of < 1 hour.  You may be able to read your email on your mobile device.  

LBNL and the UC Berkeley Library are partnering with the Royal Society of Chemistry (RSC) to support free Gold Open Access publishing under the RSC’s Gold for Gold initiative.  This program offers voucher codes that enables four LBNL researchers to publish their paper in RSC journals free of charge without paying the normal article publication fee of between $1500 and $4000.  

Benefits:  Gold Open Access publishing makes electronic versions of papers accessible to readers for free.  By removing paywall barriers, authors may increase the visibility of research findings because works are easier to disseminate, easier to find, and easier to read.  

Eligibility:  Three criteria have to be met:  1) You are affiliated with LBNL and, 2) Your article is new and has been accepted for publication by RSC, but not a previously published article, and 3) You have not previously received a Gold for Gold voucher from this initiative.  

To get your voucher code:  After your article has been accepted for publication by an RSC journal, please complete the form at http://boo.gl/GAUwr to request your Gold for Gold voucher code.  Due to limited numbers, the Libraries will distribute voucher codes on a first-come-first-served basis.  Voucher codes must be used before December 31, 2013.  

 If you have questions, please contact Jeffery Loo, Chemistry Librarian, at [email protected]

Are you running out of space in your office?  Are you moving to a new office?  Is the size of your office being reduced?  Are you responsible for dealing with the records of retiring (or already retired) scientists?  Are you wondering what to do with all the files you are responsible for--which ones need to be kept, which can be archived, and which can be disposed of, all in compliance with Lab and DOE regulations?  This month, as part of Records and Information Management Month (RIMM), you can get the answers to these and other questions by attending a 1.5 hour workshop being put on by the Archives and Records Office, along with the Berkeley Lab Learning Institute (BLI0926).  When?  on Thursday, April 25, from 10:00 AM to 11:30 AM.  Where?  in Building 54, Room 130 (Perseverance Hall).

Go here to register.

An undulator is a periodic structure of magnets through which an electron bunch goes radiating synchrotron light towards the forward direction. It has been widely used at synchrotron light facilities, such as the Advanced Light Source (ALS, US), European Synchrotron Radiation Facility (ESRF, EU), etc. In spectrum calculation, the sampled trajectories of electrons in a bunch are traced through the measured magnetic field given by a table, then the associated radiation is calculated for each trajectory and integrated at a given observation point downstream of the undulator structure. The spectrum calculation of undulator radiation is becoming more and more compute intensive primarily due to the dramatic increase in the sample number of electrons in a bunch.

Today, HPC Services staffer Yong Qin will be presenting his work during a poster session at this week's GPU Technology Conference, GTC 2013, in San Jose, California. Yong's work demonstrates how data parallelism can be applied to spectrum calculation of undulator radiation, which is widely used at synchrotron light facilities across the world.  This poster presents the algorithm design and performance optimization details for NVIDIA Fermi GPUs.  Performance data from multiple optimization efforts and algorithms will be compared.  Advanced topics, such as multiple GPUs, hybrid computing will also be demonstrated on how to further improve the field performance.  An overall more than 400 of parallel speedup is achieved on one Fermi C2050 GPU with 448 cores, with close to linear scalability.

The climate models that scientists use to understand and project climate change are improving constantly, with better representations of the oceans, ice, land surfaces and other factors in the atmosphere. While there is still some degree of uncertainty in all these components, the largest source of uncertainty in today’s climate models are clouds.

Clouds can both cool the planet, by acting as a shield against the sun, and warm the planet, by trapping heat. But why do clouds behave the way they do? And how will a warming planet affect the cloud cover?

Berkeley Lab scientist David Romps

Lawrence Berkeley National Laboratory scientist David Romps has made it his mission to answer these questions. His work involves the development of cloud resolution models on a dedicated 336-core Dell cluster and the Lab's Lawrencium cluster - both managed by the IT Division's High Performances Services Group. More...

Suspicious feeling leads IT’s Susan Green to report targeted attack

Maybe it comes from working for 15 years in Berkeley Lab’s IT Division, but administrative assistant Susan Green likes to keep a close eye on her email, even those messages that end up in her spam folder. On Jan. 25, she saw a message announcing an update to Berkeley Lab Gmail and opened the message. Inside, the text looked legitimate, but the return address caught her attention.

“Although it read like a normal memo, it didn’t seem quite right – it didn’t have an lbl.gov address,” Green said. “I noticed the word ‘lawyer’ in the email address, which wasn’t right.”

Instead of clicking on the link in the message – and since the message mentioned Gmail – Green took it to another IT employee, who was on a conference call. Green showed it to another IT manager, but was worried that she might be making a big deal out of nothing. But then the IT manager asked her to forward the message and they found a sophisticated attack targeting specific Lab employees.

“They took the legitimate Lab logon page, duplicated it exactly, and rehosted it on another server,” said Jay Krous of the Computer Protection Program. “We saw that they took the time to craft it specifically for LBNL folks – it wasn’t like the usual bank scams with misspelled words. In fact, they used the same words as are on our website. That’s when we went into alarm mode.”

The Computer Protection Program team immediately blocked access to the server from the lab, then went to work finding out who received the message. In all, 363 employees were targeted. And what they all have in common is that they buy airline tickets and make travel arrangements.

Apparently the people behind the attack hoped to obtain the employee credentials and purchase tickets on small foreign airlines – airlines that allow purchasers to return unused tickets for cash.

It turns out the targeted phishing attack had been tried in the exact same way against a number of other .edu sites. Another sign of the attack’s sophistication is that the subject line varied and the messages came from two different addresses, making it harder to block.

“People reporting these malicious email are an important part of our protection strategy.  Thanks to Susan following her intuition, we were able to block this early, and no one at the lab logged into the fake site,” Krous said. “It’s important that employees alert us as soon as they see something that doesn’t seem right – the sooner we hear about it, the sooner we can block it.”

Krous said employees should report malicious emails targeting their UC, DOE, or Berkeley Lab affiliations to [email protected]. And, of course, use caution with messages or attachments that are from people you don’t know or that you aren’t expecting.

For more cyber security information, visit the Berkeley Lab Computer Protection Program web site.  For more information and examples of targeted phishing you can go here.

We plan to take our enterprise directory service down at 10pm, Saturday March 9 for a 30 minute maintenance period.  This will impact all authentications to lab systems including cloud based services like Google.  For many, this service is commonly known as "LDAP".  It will also impact access to the new One-time-Password (OTP) Service that is being gradually phased in by IT staff.

One of our goals is to re-implement an approach to Password Expiration - Lab employees who have not changed their password within the past several months will be put on a staggered schedule (with appropriate notifications) - during April and May.  Passwords will be set to expire on workdays when the IT Help Desk can assist if needed (Tuesdays through Thursdays) if notifications have not been acted upon.

This will not impact Mobile phone access to Google mail or calendar.