Update to VMware vCenter Server Appliance & NTP Issues

Earlier today I posted “VMware vCenter Server Appliance 5.5.0 Has An Insecure NTP Server.” One of the reasons I like VMware is that they’re responsive to customer issues. This situation is no different. I just spoke with a few guys involved in VMware security, and this is what I’ve learned. 1. There has been mitigation information available internally to VMware Support/GSS since shortly after the vulnerability was published. If you call VMware Support your best bet is to reference the CVE number, CVE-2013-5211. I have not called VMware Support to confirm this, or to verify that they’re able to properly resolve the issue if you don’t reference the CVE number. In the future I’ll make sure to reference the CVE number if …

Read More

VMware vCenter Server Appliance 5.5.0 Has An Insecure NTP Server

Update: I have updated this article to reflect some new information provided by VMware. I have also published new notes and discussion as a separate blog post. On January 10, 2014 a vulnerability in ntpd, the Network Time Protocol daemon, was made public (US CERT VU#348126): UDP protocols such as NTP can be abused to amplify denial-of-service attack traffic. Servers running the network time protocol (NTP) based on implementations of ntpd prior to version 4.2.7p26 that use the default unrestricted query configuration are susceptible to a reflected denial-of-service (DRDoS) attack. Other proprietary NTP implementations may also be affected. I have encountered several vCenter Server Appliances, version 5.5.0 build 1476327 and older, that were exposed to the general Internet, and have …

Read More

Zero Out Free Space

This is post #10 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” When we talked about the rationale behind storing logs centrally one big reason was thin-provisioned virtual disks. Those disks grow over time because filesystems on a virtual machine currently have no way to tell the underlying storage that they’re done using certain blocks on disk. There is a way to make these VMs thin again, and I wrote about it as step 9 in my guide to preparing Linux Template VMs. In short, we run a script on the VM that writes zeroes to most of the free space on the VM: #!/bin/sh # Determine …

Read More

Disk Partition Alignment Is Still Important

This is post #9 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” I have written about this almost yearly (beginning all the way back in 2006), but even now I routinely run across something, like a virtual appliance, that has poor partition alignment. What’s the big deal? In short, misaligned I/O is killing your disk performance. Blame Logical Block Addressing, or LBA. Back in the day, a BIOS interacted with drives by knowing the exact geometry of the drive, namely how many cylinders, heads, and sectors were on a disk (CHS). Unfortunately that limited the size of the drives that could be used, and ignored some basic …

Read More

Store Logs Centrally

This is post #7 in my December 2013 series about Linux Virtual Machine Performance Tuning. For more, please see the tag “Linux VM Performance Tuning.” It is universally true that over time a thin-provisioned virtual machine will slowly expand on disk. This happens because of the way files are deleted from disks. When files are written to disk the storage subsystem writes those blocks out, and the thinly-provisioned disk file grows a bit. But when a filesystem deletes a file all it does is update its own internal storage map to “forget” about the file. Nothing ever removes the data from the disk itself (which is why undelete utilities work, and why there are stringent procedures for securely erasing a …

Read More

Updates to My vCSA 5.1 to 5.5 Notes

I just updated my notes on upgrading the VMware vCenter Server Appliance from 5.1 to 5.5. I added a couple of things that have become issues: Remove all non-standard users from SSO before the upgrade. If you added users to the 5.1 Single Sign-On system directly those users will be copied to the 5.5 vCSA as members of SYSTEM-DOMAIN. Unfortunately they will then become trapped, undeleteable & unchangeable, as VMware didn’t think to make the SYSTEM-DOMAIN an editable domain. You can see them, and you can still log in, but you cannot remove them or change their passwords. Your only recourse is to remove the permissions for that user from vCenter, which still means they can log in, but won’t have …

Read More

Notes on Upgrading the VMware vCenter Server Appliance 5.1 to 5.5

I’ve done a few upgrades of the VMware vCenter Server Appliance (vCSA) 5.1 now, to the GA release of 5.5 (build 1312297). Here are my observations: You need a second IP temporarily for the upgrade. The way it works is that you deploy a new vCSA, then the two of them talk to each other to do the upgrade. When they’re done copying stuff around the process will shut the old one off and reboot the new one so it’s fully functional. While the need for a second IP is fairly obvious, I managed to overlook it. Don’t specify a hostname for the new vCSA in the OVF/OVA deployment wizard if you don’t want to change the name of the …

Read More

VMware vCenter Server Appliance & NTP

If you’re trying to configure NTP on the VMware vCenter Server Appliance (vCSA) 5.1 builds 799730, 880472, or 947940 according to the official documentation you might be seeing what I’m seeing: vcenter:~ # yast2 ntp-client add server=0.us.pool.ntp.org Error: Cannot update the dynamic configuration policy. vcenter:~ # yast2 ntp-client enable Error: Cannot update the dynamic configuration policy. This appears to be a SuSE bug. Seems serious but it isn’t, the commands actually do complete correctly. If you want to check the work just use the command: cat /etc/ntp.conf to check for lines starting with “server” near the bottom. /sbin/chkconfig ntp on will enable the service at boot, and /etc/rc.d/ntp start will start it immediately if it isn’t started. /usr/sbin/ntpq -p will …

Read More

VMware vSphere Web Client & Internet Explorer 10

Being adventuresome and/or an idiot, I upgraded Internet Explorer to version 10. I can report two things: 1. I like it as a speedy web browser, and the vSphere Web Client performance feels vastly improved over IE 9. That’s actually been one of my complaints about the web client, that it’s pokey. 2. The remote console plugins don’t work. I have tried fidgeting with a bunch of the security controls and reinstalling the Console Helper, but it continues to report “Remote Console plugin is not properly installed.” For now, IE 10 joins the ranks of Apple Mac OS X users with no console access. I’ll post more information as I mess with it… I’m not really a Windows guy so …

Read More

VMware & Virsto

Howard Marks has a great piece on VMware buying Virsto over at Network Computing (link is below): Some of my fellow analysts have lumped Virsto into the flash acceleration category along with caching solutions like Proximal Data, Sandisk’s Flashsoft and Intel’s CAS. While Virsto can use flash to accelerate some storage I/O, it’s not primarily a flash acceleration product. In fact, Virsto is a log-based, clustered file system that uses a dedicated log device, which can be a shared SSD, to accelerate virtual machine I/O. I saw Virsto for the first time at VMworld 2012, and it looked interesting as something that tries to turn a lot of the random I/O from a virtualization environment back into sequential I/O that arrays can better handle, while adding a …

Read More