Background
On Thursday, July 11 at 2:30pm, the Lab conducted a real evacuation of the hill for selected buildings. In parallel with this event, all lab employees were encouraged to check-in to report on their personal status as part of the drill. Some groups used traditional phone trees, text messages or email to report out, while others used a Google based application to record similar information (and some used a combination of both methods). IT offered to develop the Google Application during the planning process for the exercise in the hopes of reducing the burden on Divisions.
The Check-in Program
The application was designed to be a pilot, not the eventual application that the lab might make use of in a real emergency - it was designed to help us learn about how people would use the application in an emergency. Because of the very short time between our offer to develop the application and the drill, we implemented the program in Google Apps Script, a scripting language that is provided as part of the Google Apps Suite which generally provides a reasonably quick platform for simple development projects.
The application we implemented allowed people to to check themselves in, and also provided supervisors with the ability to report on members of their group. In addition, an effort was made to provide feedback on progress by Organization, Building or supervisor level. The application was tested and worked well - until the actual evacuation drill. Limitations on resource usage (implemented on Google's end and undocumented) caused the application to stop processing new entries very early in the exercise.
As a result, out backup plan was brought into operation 8 minutes into the drill (at 2:38pm). A simple Google form with a Google docs spreadsheet as the data collection engine was developed. Over 2400 entries were subsequently recorded and reported on using this method. While it was effective in collecting results, we did not have the ability to report out by supervisor, building, and division during the early part of the exercise. We implemented this as the exercise continued, eventually delivering reports by Division by around 6pm and further refining them by 9pm and then again the next morning. By this time, most of the interest in the data was at the division-wide level and few supervisors saw the reporting roster.
Lessons Learned
This experience gave us two sorts of lessons, one set about Apps Script and the other about what functionality might be in a future application.
In terms of Apps Script, we need to determine whether applications can be implemented to scale appropriately to hundreds of concurrent users. We have already had discussions with the Project Manager of Apps Script about this. More broadly, any future production application for this purpose would go through the normal IT development process with direction from the functional owner (protective services). We will of course continue to favor applications/platforms that are hosted externally on robust infrastructure to support disaster recovery.
In terms of lessons learned about functionality, two things stood out to us. Many users complained about the difficulty of logging in to the application. Finding the right balance between ensuring that users of the application are really who they say they are, and making it convenient to check in is an important balancing act, one which a future application needs to address. This call would be made by PS, but from an IT perspective, our sense is that allowing unauthenticated checking, but requiring a login to view other data is a good balance. This leads to the other observation: there were many requests for access to the data that didn't follow the model of supervisors and hierarchy. While this was almost certainly related to the failure of the original application, in a real emergency, there would no doubt be many people with different needs to access the data - as a general rule, hierarchies aren't that useful in emergencies. In addition, quirks of LBNL like matrixed employees further complicate reporting. A flatter data model (all logged in users can view all data for the lab) might be the right way to go.
IT would like to apologize for the confusion resulting from the issues with the application. We made substantial efforts to QA and test the application in advance, but we did not expect the application to trigger Google App Script use limits the way that it did. We look forward to the opportunity to support a labwide checkin application in the future if this work is supported by PS and the lab community.
