Anatomy Of a Saturday On Call

10:30 AM: Blizzard begins.

12:45 PM: Receive call from NOC. Two virtual machines down. Am at a hockey game but a coworker investigates.

3:30 PM: Coworker calls me back, informs me that VMware environments have crashed. Since they just don’t do that I posit that a storage issue is occurring. Coworker is a storage administrator, he investigates.

3:45 PM: It is determined that two RAID groups are offline in one of our EMC CX3-80s. I am driving around in my full-time 4WD Jeep, decide to go to work to spare others from having to do it.

5:00 PM: Storage vendor technician tells me that they are trying to get replacement parts which are on the other side of town. ETA for that is two hours due to the blizzard.

7:00 PM: Storage vendor technician, in conjunction with his support staff, get volumes back online.

7:30 PM: Outage over, I head home, which is a 45 minute operation instead of the normal 15.

8:00 PM: Street lights all shut off while I am driving. Houses did not lose power.

8:01 PM: Call from coworker, saying that local sports arena lost power momentarily.

8:02 PM: Call from NOC, saying that power outage has disrupted the data center chillers. I turn my Jeep around, head back to work.

8:05 PM: Conference call with data center support tech and NOC, determine nothing is wrong with chillers. The NOC was speculating. I turn my Jeep around again, head home.

8:15 PM: Call from data center architect saying UPS didn’t come off of battery. I turn my Jeep around again in exactly the same place as before. Roads are freezing.

8:30 PM: Arrive at work. Storage vendor technician is stuck in the snow plow berm at the parking lot entrance. Several pedestrians and I push him out so I can get in.

8:45 PM: UPS is not on battery, just part of it thinks it is.

9:30 PM: Finally just push reset button on remote alarm panel to clear errors. All is good.

10:30 PM: Arrive home, make myself pizza, fall asleep on couch.