The idea of the “good old days” is usually false, especially in IT. With one exception, there hasn’t been a better time to be in IT or working with technology.
The exception is virtualization and blame.
Back in the day it used to be the storage guy’s fault, directly, when the storage was slow. Or the network guy’s problem. Or the app admin, with their inefficient apps. Maybe it was the guy who runs the LDAP servers. Maybe it was the OS vendor, or the hardware vendor shipped us a lemon.
Now, though, it seems that it’s always the virtualization guy’s fault.
For everything.
Virtualization has turned IT into a nanny state. Because virtual environments sit between applications and nearly every piece of infrastructure they just get blanket accusations for every problem. Applications administrators and system administrators that used to do a little diagnosis themselves, that used to show independent thought, now just go straight to their managers and complain that VMs are slow. And it seems that, every single day, I utter the words “how is it slow?”
“What do you mean, ‘how is it slow?’ It just is. Make it faster.”
<sigh>
Sometimes it is actually a problem. As much as I don’t like actual problems, I’ve come to enjoy someone identifying an actual issue that I can pass along to the storage guys, or network guys, or kick right back to the app admins.
No matter who’s fault it actually is, though, I’m the one-stop shop now for blame.
Update: I’ve written a follow-on, “Understanding Blame,” as well as a follow-on to the follow-on, “Preventing Blame.”
Interesting thoughts, however you are making a supposition that is not true in many cases.
The Virtualization Administrator IS the same as the System Administrator. That’s a fact for many companies.
Not every shop can “waste” a guy focused only on the virtualization environment, no matter if it’s vSphere, Hyper-V, Xen or a non-x86 virtualization infrastructure.
Regarding the application admins I fully agree with you, but they always been that way. It’s easier to blame the systems/virt guys than properly configure your damn weblogic/apache/websphere/whatever.
Agreed, I am comingling the idea of sysadmin and virtadmin. Probably because I’m 25% sysadmin right now, 75% virtualization dude, and I blog about my own situation.
Being both makes it worse in a lot of ways, because you’re the target for all complaints, not just virtualization ones. There’s nobody to pass the buck to, like others are doing to you… Not saying you should, but if there’s a problem you now have a lot more stuff to sort through, and no help to do it.
Ugh. Tell me about it. It gets worse when there’s a third party involved, like a vendors big app on your virt infra, and your own folks throw you under the bus TO THE VENDORS. We have a rule in my IT dept: it’s not called a VM. There’s this idea that I’ve struggled against that a VM is somehow less-than when compared to a phy box. Sure a VM might not perform at 100% compared to a phy box, but will anyone really notice? You’re still getting 100% of what that VM has to offer. No to mention the additional benefits: HA, vMotion, DRS, etc.
Hey @sixfootdad, thanks for the comment! I like that “it isn’t a VM” policy. And no, a VM won’t be 100% of a physical box, but A) that’s the point and B) very few things actually need 100% of the physical box.
I often ask my complainers how fast their app needs to go, in measurable, quantifiable numbers. I’ve had two, over the last 5 years, actually give me those numbers, and we worked to get their app the performance it needed. All the rest are getting performance that is good enough, apparently (or, more likely, they’ve just decided to grouse about performance with people who aren’t hardasses about actual facts). 🙂
Outstanding. Now the networking guy can blame the virtualisation guy ?
I’m in heaven. Surely, it’s too good to be true ?
Now? You were already doing it, if I remember Networking Tech Field Day correctly. 🙂
Which reminds me that I have a couple of unfinished networking posts I wanted to ask you about, maybe do a little coauthoring.
And here I am thinking this’d be about {git,svn} blame.
Well-put.
If I may add: Virtualization is the demon of the day. It’s convenient for people who don’t have insights into it to blame virtualization for any manner of immediately unexplainable issues with any given platform.
Yes, we all see these kinds of things from our collective user-bases: “Performance is slow…” or “something is broken… and it’s not us”
But when an engineer starts digging around and doing some actual troubleshooting (you know — troubleshooting — not just making random changes and hoping for the best) the engineer often discovers that there may be many things wrong and that whether a server is physical or virtual has very little to do with overall performance.
Well, except, of course, if somebody forgets to upgrade the VM Tools on a guest because the user-base refuses to allow a downtime to do the 30-second upgrade.
DNS or NTP issues, misconfigured applications or services, excessive monitoring or logging practices, poorly-designed databases or badly-written queries, no effective (or useful, or simply non-existent) load-balancing of work between virtualized servers, file permission problems on storage preventing or slowing access, inappropriate filesystems for type of storage or data volume, and sometimes a fundamental misunderstanding of what virtualization actually does.
In other words: complex systems have complex problems.
For about two years, I, too, have had the same concept as Damian and have been suggesting to (begging) our engineering team that we stop calling them VMs; they’re just servers.