vSphere 6.7 Will Not Run In My Lab: A Parable

“Hey Bob, I tried installing vSphere 6.7 on my lab servers and it doesn’t work right. You tried using it yet? Been beating my head against a wall here.” “Yeah, I really like it. A lot. Like, resisting the urge to be irresponsible and upgrade everything. What are your lab servers?” I knew what he was going to say before he said it. “Dell PowerEdge R610s.” I was actually surprised it was that new, and rack-mountable. “Yeah, you’re out of luck. CPUs before the E3/E5/E7 family didn’t have VT-x extensions in them to make virtualization easy so VMware had to do this thing called binary translation. vSphere 6.5 was the last release that they supported that on because, frankly, it’s slow …

Read More

Intel X710 NICs Are Crap

(I’m grumpy this week and I’m giving myself permission to return to my blogging roots and complain about stuff. Deal with it.) In the not so distant past we were growing a VMware cluster and ordered 17 new blade servers with X710 NICs. Bad idea. X710 NICs suck, as it turns out. Those NICs do all sorts of offloads, and the onboard processor intercepts things like CDP and LLDP packets so that the OS cannot see or participate. That’s a real problem for ESXi hosts where you want to listen for and broadcast meaningful neighbor advertisements. Under Linux you can echo a bunch of crap into the right spot in /dev and shut that off but no such luck on …

Read More

esxupdate Error Code 99

So I’ve got a VMware ESXi 6.0 host that’s been causing me pain lately. It had some storage issues, and now it won’t let VMware Update Manager scan it, throwing the error: The host returns esxupdate error code:99. An unhandled exception was encountered. Check the Update Manager log files and esxupdate log files for more details. A little Google action later and it’s clear there isn’t a lot of documentation, recent or otherwise, about this out there. People suggest rebuilding Update Manager, or copying files from other hosts to repair them. The VMware KB has documentation of the particular error but only in context of the Cisco Nexus 1000V, and only for ESXi 5.0 and 5.1. Here’s another thought, if you’re …

Read More

Latest ESXi Turns Off Transparent Page Sharing, So Watch Your RAM

Transparent Page Sharing is a technology from VMware that essentially deduplicates memory. Instead of having 100 copies of the same memory segment it keeps just one, and returns the savings to the user in the form of additional free capacity. In a move that further encourages people to never patch their systems VMware has set the new default for Transparent Page Sharing to “off.” They did this in the latest Updates to ESXi (ESXi 5.5 Update 2d, for example). More specifically, in order to use it by default you now need to configure your virtual machines to have a “salt,” and only VMs with identical salts will share pages. To specify a salt you need to manually edit a virtual …

Read More

How to Replace an SD Card in a Dell PowerEdge Server

We use the Dell Internal Dual SD module (IDSDM) for our VMware ESXi hosts. It works great, and saves us a bunch of money per server in that we don’t need RAID controllers, spinning disks, etc. Ours are populated with two 2 GB SD cards from the factory, and set to Mirror Mode in the BIOS. The other day we received an alarm: Failure detected on Internal Dual SD Module SD2 We’d never seen a failure like this so we had no idea how to fix it, and the Internet was only slightly helpful (hence the point of this writeup). Here’s what we did to replace it. Note: I’m certified to work on Dell servers, and have been messing with …

Read More

Thoughts on the VMware Code Leak

VMware has confirmed that there was a theft of the ESX source code around April 8th, 2012. I have some non-linear thoughts on this whole thing. First, the code is from 2003 & 2004, and for ESX. ESX was the big, bloated hypervisor that shipped with whole Linux installs, and the purported 300 MB of source code sounds like they might have code for a lot of the Linux utilities that shipped with. So what? The newer version is ESXi which forgoes the Linux install in favor of being very small. That said, I’m going to assume they have the source code for the base hypervisor itself. I’m also going to assume that some of the hypervisor code from then …

Read More

Change the Default PSP in VMware vSphere 5

One thing I do to my VMware ESXi hosts is set the default Path Selection Policy (PSP) for certain Storage Array Type Plugins (SATPs) to do the right thing. This eliminates my need to reconfigure each datastore’s multipath settings on each host, and helps guarantee that a new LUN added by someone other than me will function correctly from the start. Consider it part of my “make it easy to do the right thing” sysadmin mantra. This has been covered by others at various points for older vSphere versions, but the vMA & esxcli changed some with version 5, so here are the commands I use. I have two different arrays, one that is active/active without a specific SATP (so …

Read More

8 Things I Really Like About VMware Workstation 8

I’ve been using VMware Workstation 8 for a few days now. I really like it. A lot of posts have been made hitting some of the major highlights, but there are hundreds of small improvements that add up to be a big deal for me. Here are eight. 1. You can connect to a vSphere setup, and upload your VMs into the server environment. I’ve already used it to set up a new VM template locally, then upload it to my production environment. As I get more into vCloud Director I anticipate using this heavily to set up templates for new (or old) OSes. It’ll also be a great solution for dealing with virtual appliances that make assumptions, like DHCP. …

Read More