I thought about scheduling some posts for when I’m gone, but nothing I want to post is completely free of controversy. :-) Since I like the discussion that’s been happening here and want to be part of it I’m just going to let it be a little quiet for a while.

————–

[0] It’s been slow recently because, as much as I love vacation, I hate going on vacation. I get incredibly busy ramping up to leave. Ugh.

ShareThis

Display of System Health Information – More system health information is displayed in the VI Client for both ESX Server 3.5 and VMware ESX Server 3i.

but only today noticed that my Dell PowerEdge 1950s now have health information listed (and that I lost a drive this morning in one of my test machines… DOH). My PowerEdge 2950s do not, though. Hopefully they’ll make the cut next time.

I like the trend of integrating all the elements of server management back into the VirtualCenter server. Now if I could just have Update Manager update the BIOS, RAID, management controller, and HBA firmware when it updates ESX I’d really be happy. :-)

Update: In the comments Sean suggests disconnecting and reconnecting the ESX hosts, which did the trick for me. Thanks Sean!

ShareThis

1. You can’t fix it without breaking some of your environment, in that you have to set the physical hosts’ time back to get it to work. Then the VMs pick up the time change.
2. You can’t uncheck the “Synchronize guest time with host” option from VirtualCenter while a VM is running, basically condemning you to going to each host to uncheck that option, or letting the time get unsynchronized briefly.
3. [kb,kb2].vmware.com had been mostly unavailable all morning, preventing people from actually getting to see the articles on the problem.
4. In my environment, Windows VMs with Tuesday/Wednesday maintenance windows to pick up Microsoft Patch Tuesday updates had problems where the VMware Tools didn’t complete their post-reboot VMware Tools upgrade (”Check and upgrade Tools before each power on”). Now as we fix the licensing issue those VMs are rebooting themselves outside of their maintenance windows to complete their Tools updates.
5. People who actually have test environments for their Virtual Infrastructure, and actually have a test regimen for new code, have no way to test for problems like this. Setting the clock forward on machines is tenuous at best.
6. Waiting longer to roll out patches like this isn’t a solution, because the time bomb could just as easily be three months from now.
7. Virtual Infrastructure isn’t stable or bug-free enough to wait months to update; each update release like this fixes big problems people are having with their environments.

It all comes down to trust, and there’s a lot of us out here that just got hung out to dry. Doesn’t matter whether Paul Maritz is sorry. We’re sorry, too.

Update: John Troyer reports that the problems with the Knowledge Base are fixed. Thanks guys.

ShareThis

The whole idea of patching sucks. There are always bugs, and you always trade one set of bugs for another when you upgrade. Of course, you use testing to try to figure out if there are more bugs or less, but things like this always show up. I’ve been meaning to write a longer post about patching, especially in the wake of this DNS debacle, but Michael Janke’s post “Patch Now - What Does It Mean?” over at Last In, First Out covers most of what I wanted to say. Especially about security researchers calling for immediate action:

When security researchers/bloggers announce to the world ‘patch now’, are they are implying that the world should ‘patch now without consideration for testing, QA, performance or availability’? Or are they advising an accelerated patch schedule, but in a change managed, tested, QA’d rollout of a patch that considers security and availability? And when they complain about others not patching fast enough, are they assuming that the foot draggers are incompetent? Or are they ignoring the operational realities of making untested changes to critical infrastructure?

Amen. Overall a nice, thoughtful way to present it, and worth the couple minutes to read.

ShareThis

The late Jim Gray had a fantastic interview in ACM Queue back in 2003 where he talked about disk access times vs. capacity vs. Moore’s Law, and especially how he was mailing computers and disks to people. His price comparisons are a little dated now, but the rest is a good use of ten minutes, if you ask me. (and you didn’t, I know). :-)

ShareThis

I always get self-conscious when I see an interview posted, but it looks like all three of us were on the same page. It’s really nice to see their takes on where we’re at and where we’re going. Thanks guys, and thanks to Carl Williams for setting it up!

ShareThis

Enjoy.

ShareThis

…but by 134 years into the future?

I didn’t actually get to see what it was set to, as the machine fixed itself via NTP shortly afterwards. I suspect the last mount time is some sort of unsigned integer that overflowed.

Of course, on the subsequent reboot it needed to check the filesystems yet again, putting the last checked time back to normal.

ShareThis

Virtual servers can be even more problematic. If you have all your DNS servers virtualized which cannot be started because of network or shared storage issues, you can run into problems starting other servers and services that rely on DNS. Consider having at least one physical DNS server or having one or two DNS servers running on local storage instead of shared storage.

Running services from VMs might be a great idea, especially if you’re using techniques like anycast DNS. It’s also pretty useful for creating test environments for your infrastructure services. The thing is, VMware uses DNS, so if you don’t have a physical box running DNS and DHCP you have a dependency loop. Likewise if your SAN gear needs DNS to start, or you connect to your iSCSI volumes using DNS names. Having at least one physical machine running DNS and DHCP makes life a lot easier in a crisis.

Moral of the story is twofold: you need to know what your dependencies are and account for them, and VMware has dependencies like everything else.

ShareThis

“Those servers should come up cleanly after a reboot.”
“That storage array upgrade should not cause an outage.”
“The customer should be fine with this.”

Right.

If you can’t say “going to” then you need to do more work.

Update: if you think I’m wrong don’t take it personally, join the comments where the beatdown is already happening. Please, no nails in the 2×4s, though. :-)

ShareThis

This should be fun, especially since I’ve never been to Vegas before. I’m not really a gambler, so I’ll have to find other ways of amusing myself. Anybody have any suggestions for things I shouldn’t miss? And what else are people up to during the week?

ShareThis

Lesson here: autofocus doesn’t work very well on fires.

ShareThis

1. The only optional field in /etc/passwd, its name is an acronym for what?

2. A double-sided, double-density 5.25″ floppy disk holds how much data?

3. What is the name of the computing “law” which states that the value of a telecommunications network is proportional to the square of the number of users of the system?

4. The acronym “SCSI” is pronounced “scuzzy” but it was originally intended to be pronounced in what way?

5. How many pins did the print head of the venerable Epson FX-80 have?

ShareThis

Unlike the network carriers and their commuter surrogates, which operate all manner of regional jets, turboprops, and narrow-body and wide-body aircraft, Southwest flies just one plane type, the Boeing 737 series. That saves Southwest millions in maintenance costs—spare-parts inventories, mechanic training and other nuts-and-bolts airline issues. It also gives the airline unique flexibility to move its 527 aircraft throughout the route network without costly disruptions and reconfigurations.

Standardization of server hardware pays off for organizations, too. In the somewhat distant past each member of my team was responsible for specifying and ordering servers for their projects. There was some documentation on how to do it, and some standards, but they were followed pretty loosely. As a result we ended up with servers sporting different disk sizes, single power supplies, different warranties, with or without hardware RAID controllers, and with wildly variable CPU speeds. Each machine is a one-off, and when you’re talking about 500 machines it starts being a complete nightmare. Sure, a project might save $200 by ordering a slower CPU, but that $200 disappears quickly in staff time for documentation and other work that has to be done one server at a time. Nothing is easily interchangeable between servers, and servers are not interchangeable with each other. Ick.

Now my team has two standard server configurations that we order when we need physical hardware. We buy all of our x86 gear from Dell, so we have standard configurations for PowerEdge 1950 and 2950s, where only the number of CPUs and amount of RAM is variable. Everything else is standardized, with two power supplies, hardware-RAID 1 or RAID 5 146 GB 15K disks, 3.0 GHz Intel 5450 CPUs, five year warranty, etc. Dell has even given us a custom page on their web site that allows us to order machines in these standard configurations. An admin selects the type of machine, number of CPUs, amount of RAM, and submits the order. All the standard choices are checked & uneditable.

Despite initial reservations about a one-size-fits-all hardware policy, in practice we rarely need to go outside the standard configurations. Like Southwest, when you’re flying all the same servers you can stock common spare parts. You can even stock spare servers and skip the costly 24×7 hardware replacement warranties, since servers are more interchangeable. With a standard warranty we know exactly what type of service we’re getting and when it ends, and by getting the full five-year warranty up front we don’t have to spend time later renewing warranties.

Standardization like this seems like an obvious idea, but many organizations are like the frog in the soup pot: you don’t notice your demise until it’s too late. When you go from 50 servers to 500 in just a few years it’s easy to focus solely on keeping up with demand. But eventually you need to think about saving time and money. Standardizing wherever possible is a great way to save both.

ShareThis

Now I get to see if the activation process works. Wish me luck.

Update: No brick, it activated like a champ.

ShareThis