Over the weekend my storage guys conducted an outage on several of our EMC CLARiiON disk arrays, to patch them, change their IP addresses, and fix some issues with our SAN. Just like any other system these disk arrays need maintenance, too. In the past we’ve had serious issues with updates to these arrays, so we treat each one as completely disruptive and shut down the hosts attached to them.
While a complete shutdown is annoying, most system administrators don’t mind because it gives them an opportunity to patch and do maintenance. Us VMware guys are no different. While I might be able to use VMotion to work on my environment at any time, this isn’t the same for the guest OSes. My team saw this as a great opportunity to rebalance the workloads between our VMware clusters and upgrade VMware Tools, enabling the new “Check and Upgrade Tools before each power-on” option.
In theory this was going to be easy, but we ran into problems with some of our VMs. Prior to the VI 3.5 automatic Tools upgrade option we had to rely on the individual system administrators to keep the Tools up to date. Our Linux and Windows teams do automated patch management, but never integrated the Tools into that system (partially my fault for not pressing them to do so). The result was an environment with many different Tool versions, as old as those delivered with ESX 3.0.0.
The VMs with the newer Tools upgraded seamlessly. The VMs with old tools (all Windows VMs for us) got hung up with display acceleration settings dialogs. There were less than 20 VMs like that so it wasn’t a huge deal, just annoying to have to personally attend to each. I hope our new strategy keeps us from having to do that ever again.
In short: Virtualization technologies are maturing rapidly, and as a result there is a lot of work going into the VMware Tools to make them better. As with most software the farther behind you fall the harder it gets to catch up. To save trouble in the future make sure you do whatever it takes to keep your VMware Tools updated.
