vSphere 5 Update 1 and vCenter Agent Upgrades

We applied vCenter 5 Update 1 on Sunday in my environment, which is earlier than we often do because there’s some bugs we needed fixed (namely the issues with Fault Domain Manager/HA and SSL certificate replacement). We’ve been running Update 1 in our test environment since its release, and it looked solid. But as any experienced IT person will tell you, production isn’t test. Ever.

After the upgrade none of the hosts were connected to vCenter, and there was a task for each cluster entitled “Upgrade vCenter agents on cluster hosts” that did not proceed, did not terminate, and was uncancellable. The hosts themselves had a warning message on the summary tab indicating that a manual upgrade of the agent was necessary. I could manually reconnect the hosts, and it would do its thing with the agents, but if vCenter was restarted the tasks would reappear, with new start times, and everything would be disconnected again.

When I upgraded vSphere from 4.1 U2 to 5.0 I was given a choice to manually upgrade the agents, or do it automatically. At the time the manual upgrade seemed a better choice, since I was having upgrade issues. That choice seems to have flipped the vCenter Server Setting AgentUpgrade.autoUpgradeAgents to false:

Setting that back to true, reconnecting the hosts, and restarting vCenter fixed the problem.

I can’t take credit for this fix myself, as it was my VMware BCS guy Jonathan Proskey that thought of it, working off notes from an already open case/bug filed with VMware Engineering. But it’ll be a while until a KB article appears, and it’s extremely new, so with his permission I thought I’d post it for anybody who encounters the same issue. At least this way Google will return something for the phrase “Upgrade vCenter agents on cluster hosts.” :) If anybody needs it, it’s SR 12157263203.