The Lone Sysadmin

“There are no foreign lands. It is the traveler only who is foreign.” - Robert Louis Stevenson

Bamberg, Regensburg, Salzburg, Berchtesgaden, and finally Munich were wonderful. I’m very grateful to the hundreds of folks that helped all six of us out, translated my terrible, broken German and hand waving, and generally made everything enjoyable.

In 12 days I took 2718 photos with my Nikon D80 and my little Panasonic FX-01. I drank 43.5 liters of excellent beer in numerous beer gardens, usually a Helles or Pils, a Maß at a time. I ate like a king, probably several pigs worth of pork, a bunch of roasted chickens a half at a time, roasted fish, pretzels, and those amazing huge radishes, salted of course. I know that sauerkraut should be sweet, good beer can be very smoky, and onions are awesome when stuffed with meat and baked. And I walked it all off, 5 to 10 miles each day. We didn’t drive in the cities once we got there, but we did bicycle in Munich, which was great. The Autobahn is very nice. Germans are serious about driving and it’s refreshing. I got to just over 222 kph in a Mercedes S-350 and felt comfortable the whole time. We also had a Mercedes CLS, which was definitely built for speed. Its tires were rated to 240 kph. Navigation systems are a must in strange territories, but even so, getting lost can be awesome. I probably learned as much German from the navigation system as anywhere else. :-)

“Our battered suitcases were piled on the sidewalk again; we had longer ways to go. But no matter, the road is life.” - Jack Kerouac

It’s going to be even slower than usual around here for the next couple of weeks.[0] I’m out of here for Munich and surrounding cities! I look forward to adventuring in the land of my ancestors, while drinking their beer.

I thought about scheduling some posts for when I’m gone, but nothing I want to post is completely free of controversy. :-) Since I like the discussion that’s been happening here and want to be part of it I’m just going to let it be a little quiet for a while.

————–

[0] It’s been slow recently because, as much as I love vacation, I hate going on vacation. I get incredibly busy ramping up to leave. Ugh.

• Dell introduces ten new laptops - The INQUIRER
  "On is a flash-based barebones Linux install that gets you up and running, on the Net, and with much of what you need to run the computer in instantly. " That's pretty cool, have your whole OS in flash.

• Dell Latitude Laptops - E6400
  Holy crap, Dell has a high capacity battery and is advertising 19 hours of life on their new Latitude E6400s. Might have to find a way to get a new laptop a few years before the normal replacement cycle.

I had seen this in the release notes for Virtual Infrastructure 3.5 Update 2:

Display of System Health Information – More system health information is displayed in the VI Client for both ESX Server 3.5 and VMware ESX Server 3i.

but only today noticed that my Dell PowerEdge 1950s now have health information listed (and that I lost a drive this morning in one of my test machines… DOH). My PowerEdge 2950s do not, though. Hopefully they’ll make the cut next time.

I like the trend of integrating all the elements of server management back into the VirtualCenter server. Now if I could just have Update Manager update the BIOS, RAID, management controller, and HBA firmware when it updates ESX I’d really be happy. :-)

Update: In the comments Sean suggests disconnecting and reconnecting the ESX hosts, which did the trick for me. Thanks Sean!

Why is this VMware time bomb issue such a big deal?

1. You can’t fix it without breaking some of your environment, in that you have to set the physical hosts’ time back to get it to work. Then the VMs pick up the time change.
2. You can’t uncheck the “Synchronize guest time with host” option from VirtualCenter while a VM is running, basically condemning you to going to each host to uncheck that option, or letting the time get unsynchronized briefly.
3. [kb,kb2].vmware.com had been mostly unavailable all morning, preventing people from actually getting to see the articles on the problem.
4. In my environment, Windows VMs with Tuesday/Wednesday maintenance windows to pick up Microsoft Patch Tuesday updates had problems where the VMware Tools didn’t complete their post-reboot VMware Tools upgrade (”Check and upgrade Tools before each power on”). Now as we fix the licensing issue those VMs are rebooting themselves outside of their maintenance windows to complete their Tools updates.
5. People who actually have test environments for their Virtual Infrastructure, and actually have a test regimen for new code, have no way to test for problems like this. Setting the clock forward on machines is tenuous at best.
6. Waiting longer to roll out patches like this isn’t a solution, because the time bomb could just as easily be three months from now.
7. Virtual Infrastructure isn’t stable or bug-free enough to wait months to update; each update release like this fixes big problems people are having with their environments.

It all comes down to trust, and there’s a lot of us out here that just got hung out to dry. Doesn’t matter whether Paul Maritz is sorry. We’re sorry, too.

Update: John Troyer reports that the problems with the Knowledge Base are fixed. Thanks guys.

• Last In - First Out: Patch Now - What Does it Mean?
  "A failed system or application has known, documented consequences. It is not a game of probability or chance. An unpatched security vulnerability is a game of chance where in most cases the odds against you are not known. " Amen.

Let’s just say that if you’re running VMware Virtual Infrastructure 3.5 Update 2 you probably can’t power your VMs on anymore. DOH. Unfortunately, that’s me. I updated everything on Sunday after testing for two weeks, and I can’t even imagine how I’d test for this.

The whole idea of patching sucks. There are always bugs, and you always trade one set of bugs for another when you upgrade. Of course, you use testing to try to figure out if there are more bugs or less, but things like this always show up. I’ve been meaning to write a longer post about patching, especially in the wake of this DNS debacle, but Michael Janke’s post “Patch Now - What Does It Mean?” over at Last In, First Out covers most of what I wanted to say. Especially about security researchers calling for immediate action:

When security researchers/bloggers announce to the world ‘patch now’, are they are implying that the world should ‘patch now without consideration for testing, QA, performance or availability’? Or are they advising an accelerated patch schedule, but in a change managed, tested, QA’d rollout of a patch that considers security and availability? And when they complain about others not patching fast enough, are they assuming that the foot draggers are incompetent? Or are they ignoring the operational realities of making untested changes to critical infrastructure?

Amen. Overall a nice, thoughtful way to present it, and worth the couple minutes to read.