Display of System Health Information – More system health information is displayed in the VI Client for both ESX Server 3.5 and VMware ESX Server 3i.

but only today noticed that my Dell PowerEdge 1950s now have health information listed (and that I lost a drive this morning in one of my test machines… DOH). My PowerEdge 2950s do not, though. Hopefully they’ll make the cut next time.

I like the trend of integrating all the elements of server management back into the VirtualCenter server. Now if I could just have Update Manager update the BIOS, RAID, management controller, and HBA firmware when it updates ESX I’d really be happy. :-)

Update: In the comments Sean suggests disconnecting and reconnecting the ESX hosts, which did the trick for me. Thanks Sean!

ShareThis

1. You can’t fix it without breaking some of your environment, in that you have to set the physical hosts’ time back to get it to work. Then the VMs pick up the time change.
2. You can’t uncheck the “Synchronize guest time with host” option from VirtualCenter while a VM is running, basically condemning you to going to each host to uncheck that option, or letting the time get unsynchronized briefly.
3. [kb,kb2].vmware.com had been mostly unavailable all morning, preventing people from actually getting to see the articles on the problem.
4. In my environment, Windows VMs with Tuesday/Wednesday maintenance windows to pick up Microsoft Patch Tuesday updates had problems where the VMware Tools didn’t complete their post-reboot VMware Tools upgrade (”Check and upgrade Tools before each power on”). Now as we fix the licensing issue those VMs are rebooting themselves outside of their maintenance windows to complete their Tools updates.
5. People who actually have test environments for their Virtual Infrastructure, and actually have a test regimen for new code, have no way to test for problems like this. Setting the clock forward on machines is tenuous at best.
6. Waiting longer to roll out patches like this isn’t a solution, because the time bomb could just as easily be three months from now.
7. Virtual Infrastructure isn’t stable or bug-free enough to wait months to update; each update release like this fixes big problems people are having with their environments.

It all comes down to trust, and there’s a lot of us out here that just got hung out to dry. Doesn’t matter whether Paul Maritz is sorry. We’re sorry, too.

Update: John Troyer reports that the problems with the Knowledge Base are fixed. Thanks guys.

ShareThis

The whole idea of patching sucks. There are always bugs, and you always trade one set of bugs for another when you upgrade. Of course, you use testing to try to figure out if there are more bugs or less, but things like this always show up. I’ve been meaning to write a longer post about patching, especially in the wake of this DNS debacle, but Michael Janke’s post “Patch Now - What Does It Mean?” over at Last In, First Out covers most of what I wanted to say. Especially about security researchers calling for immediate action:

When security researchers/bloggers announce to the world ‘patch now’, are they are implying that the world should ‘patch now without consideration for testing, QA, performance or availability’? Or are they advising an accelerated patch schedule, but in a change managed, tested, QA’d rollout of a patch that considers security and availability? And when they complain about others not patching fast enough, are they assuming that the foot draggers are incompetent? Or are they ignoring the operational realities of making untested changes to critical infrastructure?

Amen. Overall a nice, thoughtful way to present it, and worth the couple minutes to read.

ShareThis

I always get self-conscious when I see an interview posted, but it looks like all three of us were on the same page. It’s really nice to see their takes on where we’re at and where we’re going. Thanks guys, and thanks to Carl Williams for setting it up!

ShareThis

Virtual servers can be even more problematic. If you have all your DNS servers virtualized which cannot be started because of network or shared storage issues, you can run into problems starting other servers and services that rely on DNS. Consider having at least one physical DNS server or having one or two DNS servers running on local storage instead of shared storage.

Running services from VMs might be a great idea, especially if you’re using techniques like anycast DNS. It’s also pretty useful for creating test environments for your infrastructure services. The thing is, VMware uses DNS, so if you don’t have a physical box running DNS and DHCP you have a dependency loop. Likewise if your SAN gear needs DNS to start, or you connect to your iSCSI volumes using DNS names. Having at least one physical machine running DNS and DHCP makes life a lot easier in a crisis.

Moral of the story is twofold: you need to know what your dependencies are and account for them, and VMware has dependencies like everything else.

ShareThis

This should be fun, especially since I’ve never been to Vegas before. I’m not really a gambler, so I’ll have to find other ways of amusing myself. Anybody have any suggestions for things I shouldn’t miss? And what else are people up to during the week?

ShareThis

Tucci chairs VMware’s board, which is stacked with EMC people; of the six members, three are current or former EMC executives and two are EMC directors. (Greene is the sixth.) Greene appears bemused she has stayed on; as her husband points out, “If you look at her résumé, this is far longer than any other job she has ever held.” Her contract, signed in late July, is for only one year. “We have not discussed how long she intends to be there,” Tucci says.

Ignoring the speculation about Greene being fired, it’s relatively common for a founder to step aside or out as the company grows, handing over the reins to someone with more experience at running big companies. Jerry Yang is also a good example of this, as Yahoo! falters and their stock price tumbles while it struggles for better leadership. Given that one of VMware’s biggest competitors is now Microsoft, Paul Maritz might be an excellent choice given his history there and continuing work in cloud computing.

Personally, though, considering my less than stellar opinion of EMC’s products and support, and my enthusiasm for VMware products and technology, I’m hoping that EMC knows what they’re doing here.

ShareThis

Dongles mean that I can’t virtualize your software.

If you’re a software developer and you think that you can protect your software from piracy by using a hardware-based protection scheme please realize this:

If I can’t virtualize your software it’s likely that I’m not going to buy it.

Just ask yourself: will what I’m doing work in a virtual machine? If the answer isn’t a definite ‘yes’ then you’re doing it wrong. Virtualization is here to stay, so invest in your future and figure it out, people.

————————

[0] Like me complaining about something is new around here.

ShareThis

PeopleSoft certifies our products (PeopleTools and EnterpriseOne Tools) on certain operating systems (including Windows 2000*, Windows Server 2003, Red Hat Enterprise Linux, etc.), not on specific hardware configurations. Therefore, as long as a customer configures VMWare virtual machines with supported operating systems, we will treat them as though they are independent (non-virtual) systems and provide full support. Our support team will attempt to resolve issues using our own environments with the same operating system. We will treat VMWare virtual machines in a similar manner to any other non-virtual hardware system. That is, we will likely configure independent systems with a supported operating system for web/app/database servers and attempt to replicate a problem. In the event that we cannot replicate an issue on separate systems using the same OS, we will look to EMC/VMWare or the OS vendor to address the problem and will work with them to find a resolution.

It’s cool how just a few words can remove a column of hardware in my data center. Thanks PeopleSoft!

ShareThis

Now my question is: is ESX 3.5 an appliance or a host OS? Do I actually want to make the recommended changes? Will it mess up something in the future when a patch from VMware assumes something about my environment that isn’t true because I’ve changed it? Exactly how much do I want to go messing around with things like NTP settings when the recommended way to configure NTP is through VirtualCenter?

I look forward to a time when ESX 3i is on par with ESX 3.5, but in the interim do I change things to gain a little security and run the risk of problems later? Is ESX a Linux distribution or is it an appliance?

ShareThis

Instead, I’d really like to run Mac OS X in ESX Server. I don’t even care if I have to buy Apple Xserve hardware to do it. I’d love to see Mac OS X guests in VirtualCenter, able to use VMotion, snapshots, HA, cloning, and all the enterprise features we already have for Windows, Linux, Netware, and Solaris x86. It would also be very cool to see Mac OS X virtual desktops. Imagine how easy it would be to switch people over then.

As it stands, the column of aging Xserves in my data center is likely to be replaced with Linux VMs when the warranties on the hardware expire. Why? Because hardware isn’t worth it anymore. It doesn’t matter whose logo is on it or how snazzy brushed aluminum is, appearances don’t change the fact that physical hardware in my increasingly warm and full data center is, at most, 5% utilized on average. Customers and their applications need encapsulation and isolation, not hardware. And especially not expensive hardware with short warranties. There are lots of good reasons to run Mac OS X in an enterprise, but without an enterprise virtualization solution it’ll continue losing to Linux, or, heaven forbid, Windows.

In conclusion, I hope the Fusion announcement is a prelude to bigger things, and I look forward to the day when my Mac desktop is just a VM running out of my data center. :-)

ShareThis

I do have one question, though. What version of virtualization will get us SkyNet?

Actually, SkyNet uses a lot of physical hardware, so maybe it isn’t virtualized very well. I hope those creepy red eyes are low-voltage.

In a weird twist of fate, I went to lunch before posting this and came back to Don MacAskill posting about SmugMug’s new SkyNet. Hmmm. Hopefully we won’t see those guys getting taken out on the next season of “Sarah Connor Chronicles.”

ShareThis

15 years boggles my mind, though. I often joke that technology years are worse than dog years, as far as obsolescence. 15 years for a technology is 105 years in some other industries. As I think about it, though, this is pretty cool. Especially since technologies like virtualization remove reasons to upgrade.

I have always used hardware replacement cycles to push OS replacement cycles. Red Hat Enterprise Linux has a seven year lifespan, and my hardware lives three to five years, so it’s always meshed up pretty nicely. Get new hardware and put the latest OS on it. If the app folks don’t like or can’t use the latest & greatest we can put the last OS version on it instead. We’ll get a shot at replacing everything again in a few years, so no worries.

Now that virtual machines are killing the hardware replacement cycle I’m left with only my software lifecycles, which really aren’t all that much better than hardware cycles. If those get longer, and I can guarantee an operating environment for 15 years, the amount of staff time and effort it takes to maintain these operating environments will drop rapidly. I’ll be able to upgrade when it makes more business sense for me, like when I’m replacing an application, or I decide it’s too much work to support 7 different versions of Red Hat Enterprise Linux. Not just when a vendor decides they’re done with an OS.

Having more control of my own destiny and more options always makes me happier, and as virtualization takes over I’m glad to see Sun taking a step in the right direction.

ShareThis

Updated: not just for Star Trek anymore. :-)

• Star Trek TNG - Alarm Klaxon.m4r
  
• Star Trek TNG - Photon Torpedo.m4r
• Star Trek TNG - Tricorder.m4r
• Star Trek TNG - Door.m4r
• Star Trek TNG - Incoming Subspace Message.m4r
• Star Wars - Blaster.m4r
• Green Tree Frogs.m4r
• Cattle - Cows Mooing.m4r
• Cattle - Bull Mooing.m4r

If you have an iPhone, enjoy.

ShareThis

Think of virtualization like a movie. A movie is a series of individual frames, but played back the motion looks continuous. It’s the same way with virtual machines. A physical CPU can only run one thing at a time, which means that only one virtual machine can run at a time. So the hypervisor “shares” a CPU by cutting up the CPU time into chunks. Each virtual machine gets a certain chunk to do its thing, and if it gets chunks of CPU often enough it’s like the movie: it seems like the virtual machine has been running continuously, even when it hasn’t. Modern CPUs are fast enough that they can pull this illusion off.

When one virtual machine stops running another virtual machine has an opportunity to run. If you have a virtual machine with one vCPU it needs a chunk of time from a single physical CPU. When a physical CPU has some free time that single vCPU virtual machine will run. No problem.

Similarly, in order for a virtual machine with two vCPUs to run it needs to have chunks of free time on two physical CPUs. When two physical CPUs are both available that virtual machine can run.

The trouble comes when folks mix and match single and dual-vCPU virtual machines in an environment that doesn’t have a lot of CPU resources available. A two-vCPU virtual machine has to wait for two physical processors to free up, but the hypervisor doesn’t like to have idle CPUs, so it runs a single vCPU virtual machine instead. It ends up being a long time before two physical CPUs free up simultaneously, and the two vCPU virtual machine seems really slow as a result. By “seems really slow” I mean it doesn’t perform very well, but none of the performance graphs show any problems at all.

To fix this you need to set the environment up so that two physical CPUs become free more often. First, you could add CPU resources so that the probability of two CPUs being idle at the same time is higher. Unfortunately this usually means buying stuff, which isn’t quick, easy, or even possible sometimes.

Second, you could set all your virtual machines to have one vCPU. That way they’ll run whenever a single physical CPU is free. This is usually a good stopgap until you can add CPU resources.

Last, you can group all your two vCPU machines together where those pesky single vCPU virtual machines won’t bother them. When a two vCPU virtual machine stops running it’ll always free up two physical CPUs. This usually means cutting up a cluster, though, so that will have also have drawbacks.

Virtualization can be awesome, but it can be pretty counterintuitive sometimes, too.

ShareThis