Update: I have updated this article to reflect some new information provided by VMware. I have also published new notes and discussion as a separate blog post.
On January 10, 2014 a vulnerability in ntpd, the Network Time Protocol daemon, was made public (US CERT VU#348126):
UDP protocols such as NTP can be abused to amplify denial-of-service attack traffic. Servers running the network time protocol (NTP) based on implementations of ntpd prior to version 4.2.7p26 that use the default unrestricted query configuration are susceptible to a reflected denial-of-service (DRDoS) attack. Other proprietary NTP implementations may also be affected.
I have encountered several vCenter Server Appliances, version 5.5.0 build 1476327 and older, that were exposed to the general Internet, and have been found to have this vulnerability. In these cases they were participating in DDoS attacks.
Yesterday I looked to the VMware KB to see if there were any security updates for these vCSAs, or mitigation approaches. Despite the vulnerability being over a month old there is no mention of it from VMware, nor is there a fix of any sort. The vulnerability probably extends to older versions of VMware ESX, too, if you are using NTP on them (as per best practices).
If you are running a vCenter Server Appliance I strongly suggest that you open a case with VMware Support regarding this problem. They have internal KB information about mitigating this. Ask them to search for CVE-2013-5211.
If you want to mitigate this problem on your own there are two ways to do it. First, VMware actually has public KB information in 1006427. It’s just buried (search that KB for CVE-2013-5211). Follow my steps below to edit the file and add their information.
If you want to mitigate the problem in a completely unsupported manner, but the one recommended by SANS and other organizations, you can SSH into the vCSA as root, and add “disable monitor” to /etc/ntp.conf. You can do this with the following steps:
- vi /etc/ntp.conf
- Move the cursor using the arrow keys to just below the entry called “driftfile /var/lib/ntp/drift/ntp.drift”
- Type an ‘i’ to put vi into insert mode. Don’t type the single quotes I use here, just the letter i.
- Type “disable monitor” and hit Enter.
- Type ‘ESC’ to get vi out of insert mode.
- Type ‘:wq’ to get vi to write the file and quit.
- service ntp restart
As with all problems like this we should ask why this happened in the first place. My questions to VMware are:
- Why is there an open NTP server running on the vCSA at all? I understand that when I configure NTP it will start an NTP daemon to keep the time in sync, but my expectation is that it would be completely firewalled, given that it isn’t intended to be a real NTP server.
- Why is the NTP configuration (/etc/ntp.conf) on the vCSA not secured?
- Why are there no firewall controls for any virtual appliances available through the web interface at port 5480? I should be able to control what services are open or not, and to what IPv4 and IPv6 addresses. This should be basic functionality in the Appliance Studio, so that all appliances built with it get these basic security features.
- Why has it been a month since the publication of a security vulnerability that affects VMware products, and days since DDoS attacks started, yet there has been no action taken and no patches released?
These are basic & obvious security measures, and part of the common security practice called “defense in depth.” VMware, you have failed in a few different ways. Fix it.
Constructive comments are welcome. Please don’t tell me that the fix for this is to use an external firewall. In the short term that may be the case, but real security on all the appliances delivered as part of a software package should be the goal, as well as proper security response from our vendors.
This is actually worse than your initial article —
VCSA 5.0 -> Not vulnerable
VCSA 5.1 -> Not vulnerable
VCSA 5.5 -> Vulnerable.
The way I was told to check for monlist DDoS was
ntpdc -n -c monlist
http://www.symantec.com/connect/blogs/hackers-spend-christmas-break-launching-large-scale-ntp-reflection-attacks
Also of note on this DDoS is that Dell EqualLogic management interface is vulnerable and there is no easy way to mitigate (since they layer over the OS). I know of EQL management that was a part of Monday’s DDoS but the vCenter on the same network was not. Now I know why (5.0 not vuln but 5.5 is).
Thanks for the info.
The appliance is also running outdated version of OpenSSL.
You probably know some of these answers already and are just being rhetorical but I just can’t resist commenting.
There is one dominant ntpd implementation which is from another time and it doesn’t distinguish between client and server. This makes a certain amount of sense because the majority of the logic is going to be in the client part, synchronizing over the network and maintaining the local clock, responding to external queries is the easy part.
There have been other ntp implementations like http://cr.yp.to/clockspeed.html Clockspeed which was privilege separated from the start (but may not be redistributable) or OpenNTPd from the OpenBSD project. Security for the local machine is more important for NTPd than other daemons because local timekeeping is often a critical security function of everything the machine does, although in this case the local security of the machine is not affected by this issue.
The configuration recommendation for setting up ntpd securely is usually “noquery nomodify” which I believe was followed here, the disabling of other administrative functions, which don’t affect the security of the local machine, was something that wasn’t well known, or in the default config as distributed by anyone, so they weren’t alone here.
The security issue which was being violated is a general one, which certainly exists in other daemons, which is that for a UDP based service it shouldn’t be possible to send a response substantially larger than the unauthenticated request which generated it. The problem is that these kinds of vulnerabilities don’t affect the vendors or customers who are amplifying the DDoS traffic with the same level of pain that the targets of the DDoS are affected.
An automated packet filter which pulled in the contents of ntpd.conf, resolv.conf, etc. and added them to the firewall would be a good idea but would take some small development effort. Redhat does this with their lokkit based default packet filter, it could be pretty simple and still get the job done. The packet filter would need to be rebuilt every time the config is changed, hopefully this doesn’t cause any second order reliability issues.
I’m not sure about securing ntpd.conf from the local shell user since almost by definition local shell accounts are admins and are authorized to edit this file. Network-facing management daemons also can’t be sandboxed away from important local config files because their job is to maintain them.
I agree though that these kind of appliances, especially Linux-based ones, have all of tools necessary to make them hard enough to put on the wild open internet like any “cloud” service. The days where you _needed_ a firewall appliance because your IP stack, OS and applications were impossibly bad are long over (not that the firewall vendors have stopped taking your money) pretty much every widely available OS comes with a hardened IP stack, built-in packet filter just as good as any firewall (and actually the core engine of some of them) and applications are much more security aware than they used to be, most “cloud” apps are directly exposed to the wild internet anyway as their very reason for existence.
Hi,
if you do the DoD hardeining inside the appliance like the VMware hardening recommendation with the script:
/etc/dodscript.sh in vcac
you will have the same problem.
Strange !
Best regards
Just got an email from my datacenter that my vCenter server IP was used as part of a DDoS attack, using the NTP server.
I followed your instructions, to add the “disable monitor” line in, but I’m also seeing a few places recommend (including in the VMWare KB) to add these lines in:
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1
I added in those lines as well.
Is there any reason not to add those lines in?
And are there any other steps that need to be taken at this point, especially with the NTP server on ESXi itself?
Thanks!
Hi,
disable monitor ist not really solving the problem: from the ntp documentation:
Use noquery in your default restrictions to block all status queries.
Use disable monitor to disable the ntpdc -c monlist command while still allowing other status queries.
try this:
restrict 127.0.0.1
restrict ::1
restrict -6 ::1
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
but at the end the best way will be to hide your esx server from the internet: isolate your vmkernel in a separate network
check with ntpdc -n -c monlist IP
you should have:
172.20.10.3: timed out, nothing received
***Request timed out
see also:
http://vpierre.it/wp/?p=200
Thank you for this detailed straight forward approach to solving this issue. I agree why is this the default distribution and why would they not start by adding this simple entry into the ntp settings from the start. I like IsaacL was notified by my data center this attack was using my server but no matter how much you search VMware KB it is all a bunch of old information that might nor might not apply. Thous settings he is talking about are not for VCenter but for older versions of VSphere. not even sure if they apply. If anyone finds out if any additional settings have any value please let us know. I would like to know if they will help make our server more secure.