The Lone Sysadmin

Rod Gabriel’s tweet clued me in that my latest work for TechTarget, VMware vSwitch vs. Cisco Nexus 1000V, is up, wherein the inimitable David Davis and I take opposing stances on VMware virtual networking. In true sysadmin form we’re both right and both wrong, of course, because it always depends on the situation, but I like the article.

Popularity: 1% [?]

My favorite question from manager types is:

“How many more VMs can we run before we have to expand?”

I can never answer this without someone sticking it to me later. I always do end up answering it, and my answer is always wrong because it’s based on averages and the very little I’m told about future projects, upcoming P2Vs, server replacements, etc. We aren’t going to get 25 more 1.28 vCPU/2.398 GB of RAM VMs, though. It’s like having 1.75 kids — it just doesn’t work that way. I could try to tell them that we have 108 GB of RAM available, but that isn’t what they want, either. They want a concrete number they can multiply by our chargeback rates and put in the budget.

It’s hard to explain the problem with all of this, though, and I’ve been searching for a good analogy to make people realize why I’m so cagy about an answer. My awesome financial analyst, Michelle Fritze, just came up with it:

“How many boxes fit in your office?”

I can’t wait to ask my CIO that.

Popularity: 1% [?]

WebEx and Microsoft Windows 7 don’t seem to get along 100% quite yet. If you are using WebEx on Windows 7 it’ll disable Aero during your session. However, if your session is over and you don’t get Aero back here’s how to fix it without rebooting:

1. Make sure you’ve closed/exited all WebEx components.

2. Right click on Windows Menu->All Programs -> Accessories -> Command Prompt and choose “Run as administrator.” You will need to accept a User Access Control warning about this.

3. Issue the commands:

net stop uxsms

net start uxsms

That should fix it.

Alternately (and potentially easier): you could restart the “Desktop Window Manager Session Manager” service via the Services administrative tool.

Popularity: 1% [?]

It’s been a while in the making, but I finally started consolidating all my Linux VM tuning notes into a single document for all to read: Linux Virtual Machine Tuning Guide.

Please take a look at it, and if there are corrections or additions to be made let me know. I know there is a lot to be done with network stack tuning, which will be added to a future revision when I get my notes sorted out.

Popularity: 2% [?]

“I’m filling out a survey. Can you tell me if we have a cloud?”

“Yes, we do,” I reply.

“We do?”

“Absolutely.”

“Really?”

“Yeah, I’d know — I built it.”

“You built it? No, I think the survey wants to know if we have a real cloud.” Well, thanks a lot.

“We do have a real cloud, and it’s the same one I’m talking about. In fact, we have two clouds, in two different locations. They’d probably be best described as ‘private clouds.’”

“Well, there isn’t an option for private clouds, so I’ll just say no.”

ARGGGH. Ten minutes pass…

“Hey, I’ve got another question. Do we use virtualization software?”

Popularity: 1% [?]

From Wikipedia:

In quantum mechanics, the Heisenberg uncertainty principle states that certain pairs of physical properties, like position and momentum, cannot both be known to arbitrary precision. That is, the more precisely one property is known, the less precisely the other can be known… The measurement of position necessarily disturbs a particle’s momentum, and vice versa.

Stated a little more simply, the sheer act of measuring a particle disturbs it, such that you can only get approximate measurements.

This is also true of computing systems and monitoring. The act of watching a system consumes resources on that system, which in turn skews the numbers you get from the monitoring system. The more data you collect, the more intensive the data collection is, the more resources it consumes. The effect is quite observable on virtual machines. I’ve got some virtual machines where customers are running their own performance monitoring tools, and those tools make what would otherwise be an idle VM into something consuming quite a bit of CPU. Multiply that by the number of VMs involved, and even a 100 MHz CPU increase makes a huge difference, en masse. Especially if those tools all choose to report on the same schedule (every minute, every 5 minutes, etc. from the top of the hour). Your performance monitoring tool might actually be causing performance problems.

Running performance monitoring tools directly on virtual machines might be a bad idea anyhow. Not only do you waste resources by doing so, you also may get incomplete results because the VM itself doesn’t know the whole story. This is especially true if resource limits are in effect in your virtualization environment[0]. What the VM thinks is 100% of a vCPU might only be 25% of an actual CPU because of resource contention. Out-of-band tools like esxtop, or vCenter’s performance charts, can tell a more factual story[1]. Besides, if you really need the guest OS point of view you can always log in and use Resource Monitor or top/iostat/vmstat to find out what the virtual machine thinks. Just make sure you’re not doing all that extra work to collect the wrong data. :-)

—————————-

[0] The VMware Descheduled Time Accounting service, which comes with the VMware Tools, can help Windows VMs by correctly accounting for time spent waiting for ESX to run the VM again. Newer & upcoming Linux distributions also can account for that with their tickless clock kernel features. But it’s usually more efficient to gather the data from the hypervisor itself.

[1] Remember, though, that esxtop and vCenter’s performance charts use resources somewhere, too (usually the ESX console OS, and/or vCenter Server & SQL Server).

Popularity: 1% [?]

A GoDaddy representative left a comment on the post about ipsCA, saying:

GoDaddy.com is happy to help ipsCA customers that have found themselves in a jam. For a limited time, our Standard SSLs are $12.99 with code sslqyh1w. Call 480-505-8877 or order online at http://bit.ly/91M3NV

I’m not usually the kind of person to parrot an ad, especially one left on my site, but it’s actually a decent deal if you want a new, real SSL cert. Admittedly it’s not for their advanced certificates, but if you have a couple of ipsCA certs to replace it might work out just fine. Personally, I’ve been quite happy with GoDaddy as a domain registrar.

Popularity: 2% [?]

Q: I’m trying to evacuate a machine that is suffering a hardware failure, and one of the VMs thinks it is installing the VMware Tools, so it won’t VMotion. The error is “The virtual machine is installing VMware Tools and cannot initiate a migration operation.” I also cannot right-click the VM in vCenter and choose “End VMware Tools Install.” Is there a way for me to end the VMware Tools installation from the ESX command line?

A: Yes, it’s a two-step process once you’ve logged into the console OS of the ESX server where the VM is executing.

First, you need the ID of the VM (all on one line if it wraps):

/usr/bin/vmware-cmd /vmfs/volumes/datastore-name/vm-folder/vmx-file.vmx getid

Then you can do a:

/usr/bin/vmware-vim-cmd vmsvc/tools.cancelinstall idnumber

Have fun!

Popularity: 2% [?]