The Lone Sysadmin

Three days until VMworld starts! Heck yeah! I’m looking forward to blogging from the conference. I am also working with TechTarget to judge the Best of VMworld 2008 awards. That should be great, there’s a lot of new stuff being announced.

I’m in Tonopah, NV right now. If you look at the map the big gray spot south of us is Area 51. We traveled down the Extraterrestrial Highway (NV Highway 375) last night, but all we saw was a closed Little Ale’Inn, some deer, and a bunch of hares. No aliens. :-(

Today’s agenda: back to Vegas via Death Valley, and the Pioneer Saloon in Jean, NV. Photos to come shortly.

I had seen this in the release notes for Virtual Infrastructure 3.5 Update 2:

Display of System Health Information – More system health information is displayed in the VI Client for both ESX Server 3.5 and VMware ESX Server 3i.

but only today noticed that my Dell PowerEdge 1950s now have health information listed (and that I lost a drive this morning in one of my test machines… DOH). My PowerEdge 2950s do not, though. Hopefully they’ll make the cut next time.

I like the trend of integrating all the elements of server management back into the VirtualCenter server. Now if I could just have Update Manager update the BIOS, RAID, management controller, and HBA firmware when it updates ESX I’d really be happy. :-)

Update: In the comments Sean suggests disconnecting and reconnecting the ESX hosts, which did the trick for me. Thanks Sean!

Why is this VMware time bomb issue such a big deal?

1. You can’t fix it without breaking some of your environment, in that you have to set the physical hosts’ time back to get it to work. Then the VMs pick up the time change.
2. You can’t uncheck the “Synchronize guest time with host” option from VirtualCenter while a VM is running, basically condemning you to going to each host to uncheck that option, or letting the time get unsynchronized briefly.
3. [kb,kb2].vmware.com had been mostly unavailable all morning, preventing people from actually getting to see the articles on the problem.
4. In my environment, Windows VMs with Tuesday/Wednesday maintenance windows to pick up Microsoft Patch Tuesday updates had problems where the VMware Tools didn’t complete their post-reboot VMware Tools upgrade (”Check and upgrade Tools before each power on”). Now as we fix the licensing issue those VMs are rebooting themselves outside of their maintenance windows to complete their Tools updates.
5. People who actually have test environments for their Virtual Infrastructure, and actually have a test regimen for new code, have no way to test for problems like this. Setting the clock forward on machines is tenuous at best.
6. Waiting longer to roll out patches like this isn’t a solution, because the time bomb could just as easily be three months from now.
7. Virtual Infrastructure isn’t stable or bug-free enough to wait months to update; each update release like this fixes big problems people are having with their environments.

It all comes down to trust, and there’s a lot of us out here that just got hung out to dry. Doesn’t matter whether Paul Maritz is sorry. We’re sorry, too.

Update: John Troyer reports that the problems with the Knowledge Base are fixed. Thanks guys.

Let’s just say that if you’re running VMware Virtual Infrastructure 3.5 Update 2 you probably can’t power your VMs on anymore. DOH. Unfortunately, that’s me. I updated everything on Sunday after testing for two weeks, and I can’t even imagine how I’d test for this.

The whole idea of patching sucks. There are always bugs, and you always trade one set of bugs for another when you upgrade. Of course, you use testing to try to figure out if there are more bugs or less, but things like this always show up. I’ve been meaning to write a longer post about patching, especially in the wake of this DNS debacle, but Michael Janke’s post “Patch Now - What Does It Mean?” over at Last In, First Out covers most of what I wanted to say. Especially about security researchers calling for immediate action:

When security researchers/bloggers announce to the world ‘patch now’, are they are implying that the world should ‘patch now without consideration for testing, QA, performance or availability’? Or are they advising an accelerated patch schedule, but in a change managed, tested, QA’d rollout of a patch that considers security and availability? And when they complain about others not patching fast enough, are they assuming that the foot draggers are incompetent? Or are they ignoring the operational realities of making untested changes to critical infrastructure?

Amen. Overall a nice, thoughtful way to present it, and worth the couple minutes to read.

Scott Lowe, Martin MacLeod, and I were interviewed by odinJobs about virtualization technology, namely how you get started with the technology.

I always get self-conscious when I see an interview posted, but it looks like all three of us were on the same page. It’s really nice to see their takes on where we’re at and where we’re going. Thanks guys, and thanks to Carl Williams for setting it up!

I’ve commented before on how I generally object to plans to virtualize absolutely everything in a data center. It looks like Eric Siebert agrees with me in his article on recovering servers after power failures:

Virtual servers can be even more problematic. If you have all your DNS servers virtualized which cannot be started because of network or shared storage issues, you can run into problems starting other servers and services that rely on DNS. Consider having at least one physical DNS server or having one or two DNS servers running on local storage instead of shared storage.

Running services from VMs might be a great idea, especially if you’re using techniques like anycast DNS. It’s also pretty useful for creating test environments for your infrastructure services. The thing is, VMware uses DNS, so if you don’t have a physical box running DNS and DHCP you have a dependency loop. Likewise if your SAN gear needs DNS to start, or you connect to your iSCSI volumes using DNS names. Having at least one physical machine running DNS and DHCP makes life a lot easier in a crisis.

Moral of the story is twofold: you need to know what your dependencies are and account for them, and VMware has dependencies like everything else.

As of 10:50 AM CDT I have a flight to Las Vegas for VMworld 2008. This means I’m completely registered and ready to go, waiting only for the course reservations to open up and the date to arrive.

This should be fun, especially since I’ve never been to Vegas before. I’m not really a gambler, so I’ll have to find other ways of amusing myself. Anybody have any suggestions for things I shouldn’t miss? And what else are people up to during the week?

I’ve been reading some of the coverage of the Diane Greene/Paul Maritz swap today, as well as the new revenue projections. I first caught Om Malik’s take, read Bernd Harzog’s comments on Paul Maritz, and then went from Adam Lashinsky’s blog post to his October 2007 Fortune article on Greene. Something there caught my eye (in bold):

Tucci chairs VMware’s board, which is stacked with EMC people; of the six members, three are current or former EMC executives and two are EMC directors. (Greene is the sixth.) Greene appears bemused she has stayed on; as her husband points out, “If you look at her résumé, this is far longer than any other job she has ever held.” Her contract, signed in late July, is for only one year. “We have not discussed how long she intends to be there,” Tucci says.

Ignoring the speculation about Greene being fired, it’s relatively common for a founder to step aside or out as the company grows, handing over the reins to someone with more experience at running big companies. Jerry Yang is also a good example of this, as Yahoo! falters and their stock price tumbles while it struggles for better leadership. Given that one of VMware’s biggest competitors is now Microsoft, Paul Maritz might be an excellent choice given his history there and continuing work in cloud computing.

Personally, though, considering my less than stellar opinion of EMC’s products and support, and my enthusiasm for VMware products and technology, I’m hoping that EMC knows what they’re doing here.