Understanding Blame

On Tuesday I posted about how virtualization teams are the one-stop shop for blame. There was some excellent commentary on it, from people who represent all areas of IT. Two things became clear:

Everybody blames everybody.

App admins blame virtualization admins. Virtualization blames storage and networking. Networking blames virtualization and storage. Storage blames virtualization. Blame isn’t unique to one area of IT, or even to IT among the human race. Nobody likes to think that they’ve messed up, and nobody likes to admit an error, so it’s easier to point the finger at others. Heck, often people don’t even know they’ve made an error, so it’s got to be someone else’s problem.

A subset of comments I received, some privately, is that virtualization admins shouldn’t feel singled out. Most of these comments were from two IT staff types: network engineers and storage admins, and the general gist of them was “welcome to the club, pal.” :)

The newest technology is always the scapegoat.

The newest technology in use is almost always the first one to be blamed, and it’s easy to see why. We work in an environment where the last change is usually the cause of the current problem, and virtualization happens to be the last big change. Hopefully IPv6 deployments hurry up so they can be the scapegoat for a while. (Just kidding!)

But why do we always blame the new thing? There are a couple of reasons, and the biggest is the “fear of the unknown.” Virtualization is very much unknown to all but a few, and the robust monitoring and management tools networking and storage guys have can’t see into virtual environments. When visibility is limited people tend to speculate.

The other thing is that performance tuning in virtual environments is pretty complex. Virtualization takes nice, sequential I/O and combines it so it looks random. It takes manageable, consistent network utilization and makes it unpredictable and dynamic. And with the one-two punch of organic growth of virtual environments in IT, and VMware’s message (explicit and implicit) that virtualization frees sysadmins from having to deal with network and storage admins, virtualization folks don’t think they need those folks and haven’t included them in any design work they might have done. As such, their environment doesn’t draw on the deep experience of all of the areas of IT.

When there are problems in virtual environments the admins go to the storage and network guys, looking for help. The thing is, people don’t like being used as insurance policies. They like being part of the design from the start, doubly so if they’re going to end up fixing it. The storage and network guys take one look at the random I/O and the big mess and don’t want to deal with it, because they had nothing to do with creating the problem. Besides, they don’t want their name attached to a mess that might be getting high-level attention.

And from all this misunderstanding, hard feelings, and blame is born.