The last couple of weeks have been pretty hectic. My coworkers have been on vacation and generally AWOL, leaving me alone with all our customers. I’m accustomed to being the only guy doing work around the office, but when other emergencies come up it throws a wrench into my normally scheduled insanity. Given the way I work, as a human, it’s hard for me to write creatively when I’m stressed out. So I usually end up drinking more, playing more Wolfenstein: Enemy Territory, and bitching at my boss. Always productive. 🙂
One of the emergencies that ate about 20 hours of my life was the realization of what was causing I/O problems on our VMware ESX Servers. In short, because of the way logical block addressing works in combination with the boot sector on a disk, anything with a PC BIOS will misalign I/O to our EMC disk arrays. The arrays like I/O in sizes that are powers of 2, especially when they’re multiples of 64 (that’s the stripe size of the array). With VMware you get misaligned I/O at the physical server level, and then another layer of misaligned I/O when you virtualize a disk for a VM. Misalignment^2. This translates into really bad performance for VMs, which is what we were seeing.
Finding that cost me about 20 hours of my life. VMware doesn’t mention this problem at all, save a couple of slides in a presentation someone gave at VMworld. EMC mentions it in their best practices documentation, and mentions how to fix it, but the folks who run our storage for us didn’t keep up on the changes to that document. Grrr. That’s a whole other blog entry.
Anyhow, I needed some of my coworkers to help me figure this out. As part of that, you have to tell them what’s going on, right? Give them something to work with — help them help you.
Q: What’s the first thing you do when you find out about a problem we’re having, a problem that needs to be researched and checked out?
A: Blab to every customer you know, telling them just enough of the story to make us all seem like negligent idiots. Don’t tell your coworkers that you told the customers, so that your coworkers get blindsided by the customers.
No, really. Tell everybody just enough to let them jump to conclusions, and just enough that they’ll write nasty email messages to their managers about all this. Don’t wait until we have facts. Or even a single fact. Facts are so last century.
Grrr.
So the other 20 hours of my life were spent controlling rumors. My favorite was a project cancelling a hardware order because when we fix this problem they won’t need the new hardware.
Moral of the story: shut your trap until you have the facts. If you want to tell people before the facts are known ask the person in charge of the operation if it’s okay.
Why ask? Because it’s polite, because the person in charge will probably appreciate not being blindsided, because the person in charge might not realize you have someone with an interest in the problem, and because they’ll probably share more information with you.
And, when you make more work for people, those people aren’t going to tell you anything more than they absolutely have to in the future. I sure learned my lesson. Yes, I am bitter.
