The Lone Sysadmin

Even though PeopleSoft is part of Oracle (and possibly subject to their anti-VMware support policies) our apps guys checked to see if VMware was an option. As it turns out, PeopleSoft solution 200955472 entitled “Does Peoplesoft support VMWare” has the answer: yes.

PeopleSoft certifies our products (PeopleTools and EnterpriseOne Tools) on certain operating systems (including Windows 2000*, Windows Server 2003, Red Hat Enterprise Linux, etc.), not on specific hardware configurations. Therefore, as long as a customer configures VMWare virtual machines with supported operating systems, we will treat them as though they are independent (non-virtual) systems and provide full support. Our support team will attempt to resolve issues using our own environments with the same operating system. We will treat VMWare virtual machines in a similar manner to any other non-virtual hardware system. That is, we will likely configure independent systems with a supported operating system for web/app/database servers and attempt to replicate a problem. In the event that we cannot replicate an issue on separate systems using the same OS, we will look to EMC/VMWare or the OS vendor to address the problem and will work with them to find a resolution.

It’s cool how just a few words can remove a column of hardware in my data center. Thanks PeopleSoft!

So I grabbed a copy of Tripwire’s ConfigCheck for ESX and ran it on one of my test ESX Servers. Sure enough, it found a bunch of defaults that haven’t been changed, and has made recommendations.

Now my question is: is ESX 3.5 an appliance or a host OS? Do I actually want to make the recommended changes? Will it mess up something in the future when a patch from VMware assumes something about my environment that isn’t true because I’ve changed it? Exactly how much do I want to go messing around with things like NTP settings when the recommended way to configure NTP is through VirtualCenter?

I look forward to a time when ESX 3i is on par with ESX 3.5, but in the interim do I change things to gain a little security and run the risk of problems later? Is ESX a Linux distribution or is it an appliance?

A few days ago Team Fusion posted about Apple Mac OS X 10.5 being their 61st supported OS. That’s pretty darn cool. Thing is, though, it doesn’t make a darn bit of difference to me.

Instead, I’d really like to run Mac OS X in ESX Server. I don’t even care if I have to buy Apple Xserve hardware to do it. I’d love to see Mac OS X guests in VirtualCenter, able to use VMotion, snapshots, HA, cloning, and all the enterprise features we already have for Windows, Linux, Netware, and Solaris x86. It would also be very cool to see Mac OS X virtual desktops. Imagine how easy it would be to switch people over then.

As it stands, the column of aging Xserves in my data center is likely to be replaced with Linux VMs when the warranties on the hardware expire. Why? Because hardware isn’t worth it anymore. It doesn’t matter whose logo is on it or how snazzy brushed aluminum is, appearances don’t change the fact that physical hardware in my increasingly warm and full data center is, at most, 5% utilized on average. Customers and their applications need encapsulation and isolation, not hardware. And especially not expensive hardware with short warranties. There are lots of good reasons to run Mac OS X in an enterprise, but without an enterprise virtualization solution it’ll continue losing to Linux, or, heaven forbid, Windows.

In conclusion, I hope the Fusion announcement is a prelude to bigger things, and I look forward to the day when my Mac desktop is just a VM running out of my data center. :-)

Ryan over at vmhero.com is pondering what the numbers behind Virtualization 1.0, 2.0, etc. mean. I wish him luck with that. Personally, I’ve never heard any actual, real-life sysadmins refer to anything by those terms. I don’t have a Virtualization 2.0 environment, I have an environment that lets me get things done. And sometimes I upgrade it. Then I go home for the day.

I do have one question, though. What version of virtualization will get us SkyNet?

Actually, SkyNet uses a lot of physical hardware, so maybe it isn’t virtualized very well. I hope those creepy red eyes are low-voltage.

In a weird twist of fate, I went to lunch before posting this and came back to Don MacAskill posting about SmugMug’s new SkyNet. Hmmm. Hopefully we won’t see those guys getting taken out on the next season of “Sarah Connor Chronicles.”

I noticed Sun’s “Java SE for Business” today. You pay money and you get 15 years of support for each release family, plus some advanced tools for updating desktops. Dealing with old versions of the JDK/JRE now has another option, instead of the two classics: paying staff to upgrade everything, or doing nothing and risking security & support problems.

15 years boggles my mind, though. I often joke that technology years are worse than dog years, as far as obsolescence. 15 years for a technology is 105 years in some other industries. As I think about it, though, this is pretty cool. Especially since technologies like virtualization remove reasons to upgrade.

I have always used hardware replacement cycles to push OS replacement cycles. Red Hat Enterprise Linux has a seven year lifespan, and my hardware lives three to five years, so it’s always meshed up pretty nicely. Get new hardware and put the latest OS on it. If the app folks don’t like or can’t use the latest & greatest we can put the last OS version on it instead. We’ll get a shot at replacing everything again in a few years, so no worries.

Now that virtual machines are killing the hardware replacement cycle I’m left with only my software lifecycles, which really aren’t all that much better than hardware cycles. If those get longer, and I can guarantee an operating environment for 15 years, the amount of staff time and effort it takes to maintain these operating environments will drop rapidly. I’ll be able to upgrade when it makes more business sense for me, like when I’m replacing an application, or I decide it’s too much work to support 7 different versions of Red Hat Enterprise Linux. Not just when a vendor decides they’re done with an OS.

Having more control of my own destiny and more options always makes me happier, and as virtualization takes over I’m glad to see Sun taking a step in the right direction.

Sometimes computers are counterintuitive. One great case continues to be why a virtual machine with two vCPUs runs more slowly than a virtual machine with one vCPU.

Think of virtualization like a movie. A movie is a series of individual frames, but played back the motion looks continuous. It’s the same way with virtual machines. A physical CPU can only run one thing at a time, which means that only one virtual machine can run at a time. So the hypervisor “shares” a CPU by cutting up the CPU time into chunks. Each virtual machine gets a certain chunk to do its thing, and if it gets chunks of CPU often enough it’s like the movie: it seems like the virtual machine has been running continuously, even when it hasn’t. Modern CPUs are fast enough that they can pull this illusion off.

When one virtual machine stops running another virtual machine has an opportunity to run. If you have a virtual machine with one vCPU it needs a chunk of time from a single physical CPU. When a physical CPU has some free time that single vCPU virtual machine will run. No problem.

Similarly, in order for a virtual machine with two vCPUs to run it needs to have chunks of free time on two physical CPUs. When two physical CPUs are both available that virtual machine can run.

The trouble comes when folks mix and match single and dual-vCPU virtual machines in an environment that doesn’t have a lot of CPU resources available. A two-vCPU virtual machine has to wait for two physical processors to free up, but the hypervisor doesn’t like to have idle CPUs, so it runs a single vCPU virtual machine instead. It ends up being a long time before two physical CPUs free up simultaneously, and the two vCPU virtual machine seems really slow as a result. By “seems really slow” I mean it doesn’t perform very well, but none of the performance graphs show any problems at all.

To fix this you need to set the environment up so that two physical CPUs become free more often. First, you could add CPU resources so that the probability of two CPUs being idle at the same time is higher. Unfortunately this usually means buying stuff, which isn’t quick, easy, or even possible sometimes.

Second, you could set all your virtual machines to have one vCPU. That way they’ll run whenever a single physical CPU is free. This is usually a good stopgap until you can add CPU resources.

Last, you can group all your two vCPU machines together where those pesky single vCPU virtual machines won’t bother them. When a two vCPU virtual machine stops running it’ll always free up two physical CPUs. This usually means cutting up a cluster, though, so that will have also have drawbacks.

Virtualization can be awesome, but it can be pretty counterintuitive sometimes, too.

Hmm, just read a post over at virtualization.info about how the latest VMware VirtualCenter build numbers don’t match the web site. I guess I didn’t notice that the web site says build 84782 but my client reports 84767. Yet another detail that didn’t get attended to. It’s always the details that make life miserable.

I was talking to another VMware customer the other day who was telling me that they were waiting for Update 1 to come out before they migrated from VI 3.0. At the time I thought that was prudent, but with reports of Update 1 problems on top of this build issue I’d advise anybody running a stable VI 3.0 environment just to stay there.

I made time to apply VMware VirtualCenter 2.5 Update 1 to my test environment. Worked great but it did require a reboot of the VC host at the end. Older VC clients seem to connect just fine to the newer server software, but I upgraded mine anyhow just out of habit.

Once that was done I used Update Manager to apply the ESX 3.5 Update 1 patches to my test ESX hosts. The process seems to be getting smoother, as it successfully moved through my test cluster and patched everything. Took a while but it was fire & forget.

Overall, looks good so far.