Over the last few months I’ve been cataloging the complaints I’ve heard about the deprecation of VMware ESX, in favor of ESXi. I’ve been running 100% ESXi since shortly after the vSphere 4.1 release. In the words of Samuel L. Jackson as Jules in Pulp Fiction, “well, allow me to retort!”
“I have software installed on the Console OS, and I need to keep doing that.”
ESX wasn’t really a Linux box, it was an appliance. Sure, in a lot of ways it looked like a Linux box, but it was missing a lot of useful packages, with no maintainable way to add them. Yes, you could copy the RPMs from Red Hat Enterprise Linux, but then you’d have software installed on the machine you’d have to patch manually. You also run the risk of conflicts, support issues, and general chaos. You were better off from the start if you defined it as ESX, not Linux, and considered the console OS a hypervisor delivery method. Nothing more.
The appliance-like nature of ESXi is way more pronounced than ESX was. I applaud them for reducing the surface area of their products, and also reducing the amount of time and effort needed to maintain software packages that are irrelevant to the hypervisor. When it comes down to it, by embracing the appliance-like nature of the product you open yourself to all the cool new possibilities, like disklessness or replacing traditional drives with SD cards, autoconfigured PXE-booted stateless hosts, central configuration, patching by just rebooting, etc.
“I run third-party software which requires the Console OS.”
VMware has shipped APIs for interacting with ESX & ESXi for many years, and certainly at least since the first release of ESXi in late 2007. If you’ve kept up with your service & support on those products you are probably entitled to a version of whatever you’re running that supports the APIs. Of course, then you have to upgrade…
“Everybody talking about how great ESXi is really means that ESXi 4.1 is great. We’re still at ESX 3.x and ESXi 3.5 isn’t as full-featured.”
Correct. ESXi 4.1 is the first release where there’s feature parity with old-school ESX, and to be honest it was the first ESXi I really, truly, was willing to run.
Personally, if I was still on 3.x I’d be looking at getting to 4.1 pretty soon, because you’re going to start falling off vendor support matrices. When VMware does another major release ESX 3.0 will be four major versions behind, and your VMs will probably be multiple virtual hardware revisions behind, too. Given my personal experience, it’s usually a lot easier to get to the current release from the last release, if only because support staff are usually more familiar with it, and it’s been tested more by developers.
“ESXi is great, blah blah blah, but our environment is really stable, and if it ain’t broke don’t fix it.”
Hey, it’s up to you. My opinion is that all software is broken at all times, having bugs and security holes and missing features. It’s just a question of whether you notice or care. As one of my local VMware guys, Rob, put it the other day, “you don’t have to upgrade; ESX will be supported for many years.”
As far as stability goes, VMware is one of those companies whose software releases work well with the “wait for the first service pack” approach. By an “Update 1” release all the major issues are usually found and worked out.
“ESXi doesn’t support traditional logging, like ESX did.”
KB 1016621 outlines how to configure remote syslog logging for all versions of ESXi. You’ll need to set up a remote syslog host first, though. Besides, good security practice is to keep logs somewhere other than the host they were generated on, so if someone breaks in you still have the logs even if the local copies were tampered with.
“I need the esxcfg-* commands to configure hosts.”
“I need to configure jumbo frames and that can’t be done from the GUI.”
“I need to have console access to maintain SSL certificates, etc.”
Three words: vSphere Management Assistant (vMA for short).
It’s a CentOS-based virtual appliance that IS a Linux box, and can be treated like one. It’s got the vSphere CLI installed which includes all the commands you’d normally want to run from the console. You just have to specify a server, username, and password now, or configure vi-fastpass.
If you don’t like the idea of a virtual appliance just build your own and install the vSphere CLI yourself. This is also a good way to solve the remote syslog host issue. You can also turn on “Remote Tech Support Mode (SSH)” as well (see below).
“I want to be able to SSH to the hosts.”
“I have headless machines in remote locations and SSH is a must.”
You can SSH into ESXi, and if you’re using 4.1 or newer it isn’t a hack or workaround. You can enable “Remote Tech Support Mode (SSH)” from the Security Profile section of the configuration tab. It will put a warning message in the Summary tab for that host, though, so you may not want to leave it on all the time. Goes without saying that it also is another vector for attack.
Frankly, while I used to SSH to my ESX hosts all the time I don’t miss it at all now with ESXi. I do everything I need to do through the vMA or the CLI.
An upgrade like this is also a good time to start using any iLO/BMC/DRAC out-of-band management features of your servers, both local and remote. Not only is it great for remote maintenance, you can also enable Distributed Power Management and the Standby/Wake functionality.
“I do chargeback with the ‘du’ command on the datastores themselves.”
Use Remote Tech Support Mode on one host, then. Or script it in PowerCLI. Or use the API. Or dump that data from vCenter using the export functions. Lots of options. Switching to an API-based approach has the advantage of exposing the provisioned/not-shared/used storage numbers more easily, if that’s of interest.
“All this ‘do everything with the CLI’ talk just means that passwords are stored in scripts now. We won’t do that.”
Actually, VMware has been working to improve this. The vi-fastpass functionality on the vMA lets you authenticate to ESXi hosts with either a stored & obfuscated set of credentials or, with 4.1 and later, your Active Directory credentials. William Lam has a good write up on it over at virtuallyGhetto.
Keep in mind that vi-fastpass isn’t part of the CLI, it’s a vMA-specific feature.
“I don’t have time to learn all this new stuff.”
Nobody does. There are good business cases to be made for doing it, though, like saving more money, not getting hacked, better DR & HA options, staying supported, etc.
Make a case, build a plan, schedule some time, get it done.
Nice, succinct write-up, Bob. Thanks!
Thank you very much for this informative article Bob. I hope people will continue to see that there is no reason not to move over to ESXi
Very well said Bob. There is no reason for people to NOT move to ESXi and VMware has actually made it very easy. The datastores/VMs don’t need any change and you can always run ESXi off a USB thumb drive and test things out first.
One thing that bugs me about ESXi is that the vmkernel port only supports 1 Gateway IP address.
Obviously if you keep all your vmotion traffic within the same subnet its not so much of an issue. However if you want to route vmotion traffic between subnets then you are flat of luck if your management network also requires a gateway.
The big thing that has annoyed me about our move to ESXi has been that we’ve lost a simple way to monitor everything about our HP servers – SNMP polling, which is how we monitor pretty much everything else in our (very mixed) environment. i.e. we don’t really want to retool with a WBEM/CIM based monitoring system.
spoken like a true sysadmin
Thanks for all the good comments, folks!
@Dan – I guess I didn’t realize that about ESXi, I’m going to have to check that out. I’ve got my vMotion and management interfaces on the same IP range so I hadn’t noticed before.
@Tim – I seem to recall being able to do SNMP polling on ESXi, just needing to configure it via the vicfg-snmp CLI command. I’ll look that up.
@JimB: 🙂 The corollary is that all OSes suck.
Keep in mind that routing vMotion traffic isn’t supported currently!
Routing vMotion traffic is not a supported configuration. So having a single gateway isn’t really a problem at all 😉 .
Doesn’t mean you can’t do it or it doesn’t work. In fact on regular ESX the Vmotion Interface still has a Gateway interface option.
There’s an interesting article about it over at yellow-bricks: http://www.yellow-bricks.com/2010/08/19/layer-2-adjacency-for-vmotion-vmkernel/
Anyway it was brought up that on ESXi I can simply use esxcfg-route which is even supported by the vMA/vCLI.
I’m guessing its not officially supported by VMware due to their Best Practice documents in which a layer2 network obviously has no hops to affect performance of VMotion traffic.
Excellent compilation – I’ve fallen behind on my blog reading but just ended up here thanks to Duncan’s recent post on ESXi.
As an aside, last week at Partner Exchange, after my session someone asked me about ESXi – he was still under the impression that it was a less-capable “free” alternative to ESX. I guess that is an unintended consequence of VMware’s decision to make a free stand-alone ESXi edition.
BTW, those TSM alerts go away after a reboot, but it’s not obvious since ESXi is so infrequently rebooted.
http://www.vcritical.com/2011/02/get-rid-of-those-esxi-tech-support-mode-warnings/
Although this is such an old post, it amazes me that many still feel the same way about ESX. I know of one who avoided ESXi completely and moved to Hyper-V. What a shame.