First, we’re in the process of fixing the issue. Fran, the primary ITS Unix admin (apparently there’s now an additional part-timer, and I was just confused) is helping us out; unfortunately circumstances prevented us from addressing the problem earlier.
A brief reconstructed timeline:
- All power to campus is lost sometime between 9:30 and 10:00 this morning, central time. This is Penn Electric’s fault.
- SCCS goes down because our battery backup is old and sucky.
- ITS continues to run on the Beardsley generator and UPSes.
- Some sort of surge on one of the lines into Beardsley shuts down the generator and all of the UPSes at around noon central time. This might be Penn Electric’s fault, this might be Facilities’ fault.
- All of the ITS machines shutdown. Hard.
- All of ITS runs around like so many headless chickens trying to diagnose the problem and resuscitate all of their machines
- Some time between 1:30 and 2:30 this afternoon central time, power to campus comes back.
- ITS’ machines come back.
- SCCS’ machines come back, but Roc can’t restart fully without human intervention. This is our fault.
- Fran, the ITS person who is allowed to look inside the SCCS’ password lockbox, agrees to help out after their fixage is completed.
- Currently, Fran is using printed instructions and over-the-phone coaching by Dan (who is actually still a sys-admin) to bring Roc back up.
Therefore, we expect service to be restored within the hour. Note that the above may be totally inaccurate, since I actually wasn’t there.