I see a lot of misinformation floating around about the OpenSSL Heartbleed bug. In case you’ve been living under a rock, OpenSSL versions 1.0.1 through 1.0.1f are vulnerable to a condition where a particular feature will leak the contents of memory. This is bad, because memory often contains things like the private half of public-key cryptographic exchanges (which should always stay private), protected information, parts of your email, instant messenger conversations, and other information such as logins and passwords for things like web applications.
This problem is bad, but freaking out about it, and talking out of our duffs about it, adds to the problem.
You can test if you’re vulnerable with http://filippo.io/Heartbleed/ — just specify a host and a port, or with http://s3.jspenguin.org/ssltest.py from the command line with Python.
1. Not all versions of OpenSSL are vulnerable. Only fairly recent ones, and given the way enterprises patch you might be just fine. Verify the problem before you start scheduling remediations.
2. Heartbleed doesn’t leak all system memory. It only leaks information from the affected process, like a web server running with a flawed version of OpenSSL. A modern operating system prevents one process from accessing another’s memory space. The big problem is for things like IMAP servers and web applications that process authentication data, where that authentication information will be present in the memory space of the web server. That’s why this is bad, but it doesn’t automatically mean that things like your SSH-based logins to a host are compromised, nor just anything stored on a vulnerable server.
Of course, it’s always a good idea to change your passwords on a regular basis.
3. People are focusing on web servers being vulnerable, but many services can be, including your email servers (imapd, sendmail, etc.), databases (MySQL), snmpd, etc. and some of these services have sensitive authentication information, too. There’s lots of email that I wouldn’t want others to gain access to, like password reset tokens, what my wife calls me, etc.
4. A good way, under Linux, to see what’s running and using the crypto libraries is the lsof command:
$ sudo lsof | egrep "libssl|libcrypto" | cut -f 1 -d " " | sort | uniq cupsd dovecot dsmc httpd imap-logi java mysqld named nmbd ntpd sendmail smbd snmpd snmptrapd spamd squid ssh sshd sudo tuned vsftpd
This does not list things that aren’t running that depend on the OpenSSL libraries. For that you might try mashing up find with ldd, mixing in -perm and -type a bit.
5. Just because you patched doesn’t mean that the applications using those libraries are safe. Applications load a copy of the library into memory when they start, so you replacing the files on disk means almost nothing unless you restart the applications, too. In my item #3 all of those processes have a copy of libcrypto or libssl, and all would need to restart to load the fixed version.
Furthermore, some OSes, like AIX, maintain a shared library cache, so it’s not even enough to replace it on disk. In AIX’s case you need to run /usr/sbin/slibclean as well to purge the flawed library from the cache and reread it from disk.
In most cases so far I have chosen to reboot the OSes rather than try to find and restart everything. Nuke it from orbit, it’s the only way to be sure.
6. Patching the system libraries is one thing, but many applications deliver libraries as part of their installations. You should probably use a command like find to search for them:
$ sudo find / -name libssl*; sudo find / -name libcrypto* /opt/tivoli/tsm/client/ba/bin/libssl.so.0.9.8 /opt/tivoli/tsm/client/api/bin64/libssl.so.0.9.8 /home/plankers/pfs/openssl-1.0.1e/libssl.a /home/plankers/pfs/openssl-1.0.1e/libssl.pc /usr/lib/libssl.so.10 /usr/lib/libssl.so.1.0.1e /usr/lib64/libssl.so.10 /usr/lib64/libssl3.so /usr/lib64/libssl.so /usr/lib64/pkgconfig/libssl.pc /usr/lib64/libssl.so.1.0.1e /opt/tivoli/tsm/client/ba/bin/libcrypto.so.0.9.8 /opt/tivoli/tsm/client/api/bin64/libcrypto.so.0.9.8 /home/plankers/pfs/openssl-1.0.1e/libcrypto.a /home/plankers/pfs/openssl-1.0.1e/libcrypto.pc /usr/lib/libcrypto.so.1.0.1e /usr/lib/libcrypto.so.10 /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libcrypto.so.10 /usr/lib64/libcrypto.so /usr/lib64/pkgconfig/libcrypto.pc
In this example you can see that the Tivoli Storage Manager client has its own copy of OpenSSL, version 0.9.8, which isn’t vulnerable. I’ve got a vulnerable copy of OpenSSL 1.0.1e in my home directory from when I was messing around with Perfect Forward Secrecy. The rest looks like OpenSSL 1.0.1e but I know that it’s a patched copy from Red Hat. I will delete the vulnerable copy so there is no chance something could link against it.
7. If you were running a vulnerable web, email, or other server application you should change your SSL keys, because the whole point is that nobody but you should know your private keys. If someone knows your private keys they’ll be able to decrypt your traffic, NSA-style, or conduct a man-in-the-middle attack where they insert themselves between your server and a client and pretend to be you. Man-in-the-middle is difficult to achieve, but remember that this vulnerability has been around for about two years (April 19, 2012) so we don’t know who else knew about it. The good assumption is that some bad guys did. So change your keys. Remember that lots of things have SSL keys, mail servers, web servers, Jabber servers, etc.
8. While you’re messing with all your SSL certs, step up your SSL security in general. A great testing tool I use is the Qualys SSL Labs Server Test, and they link to best practices from the results page.
Good luck.
I would point out a few additional things
– SSH is not affected as it doesn’t expose TLS Heartbeat
– The problem is not only can an attacker root around in the memory space of your TLS using daemon but that if they get the private key and they have previously recorded TLS sessions which used that key, unless you have Perfect Forward Secrecy, they can decrypt these saved sessions as well. The risk is dependent on how likely someone is to have tapped the path between you and your server, something local where you control all the equipment is less likely (but not impossible), over the open internet it is pretty much guaranteed.
– The big fun is after all the patching of highly maintained systems which you have full control over is finding all of the appliances and embedded systems where you need to engage a vendor who may not understand or push out updates quickly. There will probably be a long tail of vulnerable systems after this weeks emergency patching is over.
True — VMware is a good example of that, where we are currently awaiting a patch to ESXi 5.5.
As for decrypting sessions, yeah, but again you’d have to have sniffed that traffic first. Frankly, most people are using such bad SSL practices anyhow that the decrypting of traffic is often easy even without a crutch like Heartbleed.
Load-balancers terminating SSL certificates that used OpenSSL may be vulnerable although the web servers behind them may not be. For example, Microsoft IIS servers are not vulnerable, but SSL acceleration on a load-balancer to handle SSL traffic with clients and only sends plain-text http to the web server could still be an issue that compromises private cert data.