I’m a little behind on my reading, but I wanted to address Major Hayden’s blog posts about disabling Security-Enhanced Linux, or SELinux, which brings mandatory access control to Linux. Mandatory access control is a completely different permission model for UNIX-based hosts, and Mr. Hayden feels it is underutilized:
After many discussions with fellow Linux users, I’ve come to realize that most seem to disable SELinux rather than understand why it’s denying access. In an effort to turn the tide, I’ve created a new site as a public service to SELinux cowards everywhere: stopdisablingselinux.com.
It’s pretty rare for me to argue against a security technology but in my eyes SELinux isn’t a solution to very many problems. I know how SELinux works, what it does, how to configure it and troubleshoot it, and as a result I disable it everywhere. Here’s why:
1. While SELinux ships as part of most Linux distributions, few distribution maintainers have it set up correctly for all the services that ship with their OS. Perhaps web servers work fine if you use the defaults, but deviate from those, or install another service like a mail server or tftpd and there’s a real good chance it’ll malfunction. And by “malfunction” I mean “fail silently” which just leads to a bunch of time spent troubleshooting. That’s time out of my life I’m never getting back again. On top of that, patching those distributions sometimes resets the SELinux configurations, which both annoying and a symptom of Linux vendors not caring enough to do a good job. You can deal with patching problems with more testing, but that’s also more time and effort down the drain.
2. Absolutely no third-party vendors support SELinux[0]. Which is unfortunate, as that’s one of the most compelling use cases since those vendors often have gaping, unpatched security holes in their products. As a result it’s rare that you’ll have an entire environment with SELinux turned on, which means you’ve just added yet another variable to account for in your data center (has SELinux enabled, has SELinux in permissive mode, or does not have SELinux enabled), three times as many system configurations to account for, and you should do three times more testing, too.
3. SELinux protects the interface between applications and the OS. It doesn’t inherently protect the application, beyond keeping the host itself secure. That’s important, but less so now that we have clouds and virtualization. Back when we bought one expensive server, ran one OS image on that server, and ran a ton of different apps alongside each other in that single OS image it was very important to protect one application from another. Now that we have lots of smaller OS images running in isolation inside virtual machine monitors the focus is on application security. If an intruder breaks through the application and manages to do something to the OS you redeploy the application on a fresh VM. Only this time it’s with the latest OS & application patches applied, too.
For busy IT shops and busy sysadmins return on investment is always a factor. In my eyes SELinux is a tool which will never save you enough time to justify having implemented it in the first place. Very low ROI. When it is up to me I’d rather see scarce IT time and money spent on more productive security initiatives:
- Implementing regular OS patching. So few places do this well or in a timely fashion and it treats the root cause of many security problems directly. SELinux treats symptoms, not causes.
- Implementing better service design, so that patching and system outages can take place without service outages, and that services might be scalable to handle additional workload from users or denial-of-service attacks.
- Implementing configuration management tools, like Chef or Puppet. If time is money these systems print money. They open the door to better testing, rapid & repeatable deployments, elastic performance scaling, and good system documentation. I’ve also found they also lead to better, tighter security controls. For example, instead of adding large IP ranges to firewall rules “because it’s easier” you add just the specific IPs you need. With these tools it’s not a big deal to add another IP later if you need to. It’s also possible to do configuration audits with these tools, thereby finding discrepancies, potential backdoors, permission errors, etc. Heck, proper implementation of these tools should make enabling SELinux easier, but I still would take a critical look at the overall ROI of that decision versus other moves you can make.
- Implementing better application security, whether it’s code reviews, automated code testing, IDS/IPS, patching third-party apps, etc. The application is where your data lives and is often the weakest link in the security chain. Nowadays the OS is expendable, and rather than protect the OS under the application we should focus instead on hardening the application itself. You focus on intruders coming into your house via the doors and windows, right? Not through the middle of the basement floor.
In an ideal world we’d all have time to set this sort of thing up, and it would work perfectly. This isn’t an ideal world, and as such compromises are made. Don’t let someone shame you into wasting your time by calling you a SELinux coward. You’re in good company[1].
—-
[0] I have not checked all of them. It’s possible one or two do, but I would be surprised.
[1] At the very least you have company. 🙂
I’m pretty much in agreement with you, and the point about third party support is spot on – we run various Oracle systems, all of which recommend disabling SELinux which made justifying the choice easier with my security team. I would say put SELinux in ‘permissive’ mode rather than disable it altogether. For those that don’t know this simply logs what actions SELinux would take without actually interfering with the running of the system. This means (should you get time) you can review the logs and see what impact enabling it would have. You’ll have to manage the logs but at least it’s a start.
Problem is that nobody gets the time…
I’m going to have to say I strongly disagree with you and am in the same boat as Major Hayden, although I don’t think shaming people is an effective way to evangelize. I’ve used SELinux on all the systems I’ve set up for many years on RHEL5 and RHEL6 with many different services and have not found it to be a significant impediment or time sink so the ROI has been quite good.
1) You are making a couple of points here, that SELinux isn’t setup correctly by your distribution, that it doesn’t cover all the software, that permission failures are not reported and that patches change the SELinux config in an unspecified way.
a) On RHEL5/6 I have not had this experience, everything that ships with the distribution just works out of the box and does not have permissions related failures
b) As far as coverage, the targeted policy covers a substantial subset of the daemons and software that comes with the system but not everything and not locally built and installed software but I don’t see this as a problem. Anything which isn’t covered by the targeted policy runs as unconfined type and has the base POSIX permissions so there is nothing gained by disabling SELinux across the board, and you lose the policy for the things which do have it.
c) Failures are reported to applications with the standard EPERM/permission denied just like any other permission and failures are logged in machine readable format to the audit.log. As is pointed out the best practice is to enable the setroubleshooter which will pull out permission failures from the audit.log and write out more human readable warnings to syslog, and includes commands and help to research and fix any permission changes you wish.
d) I can’t comment on this too much without further detail but there are ways to shoot yourself in the foot by not making changes persistant, the same can be said for basic POSIX permissions as well which can be reset by updates. Using semanage to add new regexes to restorecon can do a lot to make sure your file changes persist even as files are moved around and replaced.
2) As I said before, any software which doesn’t have an SELinux policy or types associated with it gets the unconfined type which runs with standard POSIX permissions only so you aren’t ahead by disabling permissions for the rest of the system as well. It’s not a magic bullet and you can’t fault it for not protecting things it dosen’t cover.
3) It’s true, large multi-user, multi-tenant systems have been abstracted to the hypervisor/guest level rather than the kernel/process level as the world returns to the era of the Mainframe called Cloud. That doesn’t mean that protecting your machine from getting popped, by adding sanity checks and permissions to remote-facing daemons isn’t a good thing and won’t save you time. You can’t always guarantee that a patch is out and deployed before an exploit is discovered, there are many zero-days where the patch is only made after the fact, but system permissions can and do prevent exploits from working which keeps you from having corrupted data or having to rebuild systems unnecessarily. It isn’t magic and doesn’t protect against everything but it’s such an easy, low overhead thing, why not take advantage of it?
All those other things you mention can and should be done too but there is no reason to take SELinux out of your toolkit. When was the last time you used it and what problems did you run into? I think it’s time for a second look 😎
Perhaps. Red Hat is where I’ve had the most experience so your experiences to the contrary are interesting, and a second look is probably warranted. I still stand by my assertion that SELinux is not what you should be doing if you haven’t got patching & configuration management under control first.
I have to concur with Mark above. I keep SELinux enabled on all of my systems. And I am looking at installing some systems with Multi-Category Security (MCS) which is a security model that comes with SELinux for a client with particular needs in a “cloud” situation. Web applications reaching out into the OS in unauthorized ways isn’t unheard of and in a “cloud” setup you need to protect customers from each other. This is where SELinux and MCS come in.
As for your points above:
1. Everything in RHEL5/6 just seems to work with SELinux enabled. But you do have to get into the habit of checking the logs. When things go weird one of the first things I do is check /var/log/audit/audit.log. I also have setroubleshoot enabled by puppet on all of my machines.
2. I’m not sure what sort of SELinux support you would want from third party vendors. They shouldn’t be expected to troubleshoot your policy. Run the third party apps in unconfined domains or write your own policy. It’s a lot like writing firewall rules and they don’t expect you to completely open up your firewall.
3. SELinux is particularly useful for a web server because when it gets exploited (I once had a web app exploited and SELinux stopped the proliferation of badness) and the attacker starts doing things that the httpd_t domain isn’t allowed to do it causes a flurry of log activity via auditd. This was promptly noticed and the badness stopped. PLUS, because we had reasonable assurances that the attacker didn’t get root or modify anything else cleanup was a snap and didn’t require re-install of the entire box which saved a ton of time.
HIDS (host based intrusion detection) is often asked for in my business. In the Windows world that generally means a resource sucking anti-virus/malware package like Sophos or McCaffee etc. Between SELinux, auditd, AIDE FIM, puppet, and logstash I actually have a pretty complete HIDS for a Linux server which makes my clients happy.
Oh, one more thing:
http://arstechnica.com/security/2013/05/attack-hitting-apache-sites-goes-mainstream-hacks-nginx-lighttpd-too/
I pretty much guarantee you that unless this is being done by somehow logging in with valid admin credentials via ssh, SELinux would put a stop to this also with the default config. And if you run SELinux in strict mode it might prevent this even if you did ssh in with valid credentials! (because you would have to be able to change your role)
The ssh crc attack of 2001? SELinux would have prevented it.
The recent nginx remote code execution vulnerabilities such as CVE-2012-2089? SELinux mitigates those too!
The just announced Exim/Dovecot remote exploit? http://www.exploit-db.com/exploits/25297/ SELinux has you covered!
There are many more.
I think you have grossly underestimated the utility of having SELinux enabled.
From that Ars Technica article: “Researchers still don’t know how servers are being infected with Cdorked. Because compromised machines are running a variety of administration controls, cPanel and competing software aren’t obvious suspects. Cdorked doesn’t have the ability to spread by itself and doesn’t exploit a vulnerability in any other specific piece of software, either.”
I think this, alone, is what’s changing my mind here. All your other examples were fairly high-profile, so there will be a patch, but I’m apparently assuming that all vulnerabilities in use by attackers are publicly known ones. Totally not the case, as with Cdorked, where the attackers are probably using new, unpublished vulnerabilities they’ve found but not disclosed.
Still, though, it’s just a layer in a good defense-in-depth strategy, and I think time is better spent on more fundamental aspects of IT operations, like patching & config management, before attempting to run SELinux.
You can add another patchwork item, SELinux often works because of “security by obscurity” because basic exploits do not target it out-of-the-box, but could easily circumvent it had they wished to.
https://bugzilla.redhat.com/show_bug.cgi?id=962792#c21
We provide HPC environment to local researchers and we disable SELinux because it has, by far, been more trouble than it’s worth. I would say that much of that is because of the silent failure modes, poor support for anything slightly uncommon ( NFS home directories on non-SELinux server, LDAP authentication, … ), and tendency for even the stock targets to spew errors into the audit.log often making diagnosis difficult.
The other major point is that the few exploits I have seen recently have not needed root, but have taken advantage of user-level access to serve up malware and ads. These are things that a reasonable SELinux policy would not have prevented. Virtualization was successful at limiting the scope of these attacks and were trivial to cleanup.
I will put it back on my long term “to-do” list to re-evalute it in the next major release, but it will take a significant improvement in the utility, debugging, and failure modes for me to make a recommendation to change the default from ‘disabled’.