Update: I have updated this article to reflect some new information provided by VMware. I have also published new notes and discussion as a separate blog post.
On January 10, 2014 a vulnerability in ntpd, the Network Time Protocol daemon, was made public (US CERT VU#348126):
UDP protocols such as NTP can be abused to amplify denial-of-service attack traffic. Servers running the network time protocol (NTP) based on implementations of ntpd prior to version 4.2.7p26 that use the default unrestricted query configuration are susceptible to a reflected denial-of-service (DRDoS) attack. Other proprietary NTP implementations may also be affected.
I have encountered several vCenter Server Appliances, version 5.5.0 build 1476327 and older, that were exposed to the general Internet, and have been found to have this vulnerability. In these cases they were participating in DDoS attacks.
Yesterday I looked to the VMware KB to see if there were any security updates for these vCSAs, or mitigation approaches. Despite the vulnerability being over a month old there is no mention of it from VMware, nor is there a fix of any sort. The vulnerability probably extends to older versions of VMware ESX, too, if you are using NTP on them (as per best practices).
If you are running a vCenter Server Appliance I strongly suggest that you open a case with VMware Support regarding this problem. They have internal KB information about mitigating this. Ask them to search for CVE-2013-5211.
If you want to mitigate this problem on your own there are two ways to do it. First, VMware actually has public KB information in 1006427. It’s just buried (search that KB for CVE-2013-5211). Follow my steps below to edit the file and add their information.
If you want to mitigate the problem in a completely unsupported manner, but the one recommended by SANS and other organizations, you can SSH into the vCSA as root, and add “disable monitor” to /etc/ntp.conf. You can do this with the following steps:
- vi /etc/ntp.conf
- Move the cursor using the arrow keys to just below the entry called “driftfile /var/lib/ntp/drift/ntp.drift”
- Type an ‘i’ to put vi into insert mode. Don’t type the single quotes I use here, just the letter i.
- Type “disable monitor” and hit Enter.
- Type ‘ESC’ to get vi out of insert mode.
- Type ‘:wq’ to get vi to write the file and quit.
- service ntp restart
As with all problems like this we should ask why this happened in the first place. My questions to VMware are:
- Why is there an open NTP server running on the vCSA at all? I understand that when I configure NTP it will start an NTP daemon to keep the time in sync, but my expectation is that it would be completely firewalled, given that it isn’t intended to be a real NTP server.
- Why is the NTP configuration (/etc/ntp.conf) on the vCSA not secured?
- Why are there no firewall controls for any virtual appliances available through the web interface at port 5480? I should be able to control what services are open or not, and to what IPv4 and IPv6 addresses. This should be basic functionality in the Appliance Studio, so that all appliances built with it get these basic security features.
- Why has it been a month since the publication of a security vulnerability that affects VMware products, and days since DDoS attacks started, yet there has been no action taken and no patches released?
These are basic & obvious security measures, and part of the common security practice called “defense in depth.” VMware, you have failed in a few different ways. Fix it.
Constructive comments are welcome. Please don’t tell me that the fix for this is to use an external firewall. In the short term that may be the case, but real security on all the appliances delivered as part of a software package should be the goal, as well as proper security response from our vendors.