Notes on Upgrading the VMware vCenter Server Appliance 5.1 to 5.5

I’ve done a few upgrades of the VMware vCenter Server Appliance (vCSA) 5.1 now, to the GA release of 5.5 (build 1312297). Here are my observations:

You need a second IP temporarily for the upgrade. The way it works is that you deploy a new vCSA, then the two of them talk to each other to do the upgrade. When they’re done copying stuff around the process will shut the old one off and reboot the new one so it’s fully functional. While the need for a second IP is fairly obvious, I managed to overlook it.

Don’t specify a hostname for the new vCSA in the OVF/OVA deployment wizard if you don’t want to change the name of the appliance. If it’s blank on the new one it’ll get copied over from the old one. If it isn’t blank it’ll warn you about this, at which point you’ll need to shut it all down and fix it in the vApp properties.

It’s probably worth mentioning that the vApp properties are only visible when you’re attached to vCenter. If you are using the Windows vSphere Client to connect straight to an ESXi host to manage the upgrade you won’t be able to clear the hostname you inadvertently defined. Not that I learned this the hard way or anything. 🙂

You start the upgrade from the new vCSA wizard, the one you see when you first log into the VAMI (port 5480) on the freshly-deployed vCSA. Accept the EULA, pick the upgrade option.

The first time I added the upgrade key on the 5.1 source/old side and clicked the button it just sat there. It might be a Chrome problem, but after a minute I ended up clicking on another tab, then going back to the Upgrade tab, then doing it again and it worked right. It has happened every time I have done an upgrade.

You can cancel the upgrade on both sides, at least until it starts copying, at which point you better have a snapshot or something. I took a cloned copy and a snapshot of both the old and new vCSAs so I could easily roll back and restart. If you do cancel on the source side you will have to restart the vCenter Server from the vCenter Server tab.

If you’re using self-signed certificates it may complain about your hostname not being in the Subject Alternate Name of the certificate. If you check the box to regenerate the certificates it’ll fix that. I’ve had mixed results here. When I regenerated the certs they have the right hostname in them, which makes it easier to get them signed later because you can use the keys and everything they already generate. My SSO continued working after the upgrade, too, but all the other tools (Update Manager, Replication, etc.) needed manual intervention to accept the new cert (and Replication just flipped out completely, I need to investigate that more). When I didn’t regenerate the certificate my SSO & Active Directory authentication didn’t work after the upgrade.

If you’re using custom, actually-signed SSL certificates there’s guidance on that in the KB article on upgrading the vCSA, which you should read anyhow.

I’ve actually never had SSO work 100% right after an upgrade. Even when it’s been “successful” it’s taken Active Directory domains out of the search order (and SSO 2.0 doesn’t appear to even have a search order anymore). Make sure you know what your root password is, and practice logging in as root@localos.

The upgrade takes an unnerving amount of time. That KB article says you can tail the upgrade log but even that sat there for a while. You might have some luck by watching the performance graphs for CPU activity, too. My successful runs take between 10 and 25 minutes, for small environments using embedded databases. If you’re upgrading a production system you might want to announce/schedule at least two hours of downtime. That way if there’s a problem you can restart the process, too.

Make sure you double-check the time zones, NTP settings, etc. in the VAMI. That sort of thing doesn’t appear to get copied over.

Updates — November 13, 2013:

Remove all non-standard users from SSO before the upgrade. If you added users to the 5.1 Single Sign-On system directly those users will be copied to the 5.5 vCSA as members of SYSTEM-DOMAIN. Unfortunately they will then become trapped, undeleteable & unchangeable, as VMware didn’t think to make the SYSTEM-DOMAIN an editable domain. You can see them, and you can still log in, but you cannot remove them or change their passwords. Your only recourse is to remove the permissions for that user from vCenter, which still means they can log in, but won’t have access to anything. There are bugs open on this, but like all bugs filed with VMware don’t expect a resolution soon.

Test Active Directory connectivity before the upgrade. vSphere 5.5 changes SSO, generally for the better, but it also means that things that worked under 5.1 may not anymore. I have had some issues with AD configurations that worked previously and no longer do. It also appears that vSphere 5.5 cares more about certain AD attributes than in previous versions. For example, a user that was incorrectly set in AD as part of the wrong domain, [email protected] rather than [email protected], was able to log in but then got Inventory Service permission errors. Under 5.1 it worked correctly. Another oddity I noted was if SSO’s search path cannot see the Domain Users group you cannot log in, and see an error about Group SID in the web client (and the SID it shows is that of Domain Users). Doesn’t seem to matter if you are or aren’t actually using Domain Users in your permission model. It’s easy enough to deploy a vCSA to test AD with, make sure you do it ahead of the upgrade to avoid issues.

Good luck!