How to Configure Remote syslogd on Red Hat/CentOS 5

It’s fairly easy to configure syslogd on one host to accept syslog messages from other hosts. This is useful in normal system administration, to keep logs off the original system in case of a security breach that might compromise them, as well as for systems like VMware ESXi which don’t store logs locally due to their architecture.

This assumes a basic level of familiarity with Linux, particularly the Red Hat types of distributions (Red Hat Enterprise Linux, Fedora, CentOS, Scientific Linux, Oracle Enterprise Linux, etc.). The concepts are likely the same for other distributions. I used Red Hat Enterprise Linux 5 and I’ll assume you are able to handle the provisioning of a similar host for your purposes. I also assume you’re able to use an editor of your choice (nano, vi, emacs) and have sudo privileges on the host.

If there is an error here please let me know via the comments. I know I need to devote more time to the new options in RHEL 6, that will come as a separate post.

The pretty way:

1. Use Splunk. If you will have less than 500 MB of log data per day it’s free, provides a very nice web-based interface for searching and whatnot, and works with all sorts of log formats. Plus they have good instructions on getting it all installed, for the parts they haven’t automated. I call this the “pretty way” but it’s fairly easy, too.

I have received nothing from Splunk for this promotion, I just like their product, and once upon a time I knew some people that worked there. If you tell them I sent you they’ll probably charge you more. 🙂

The easy way:

1. Alter your iptables/ip6tables rulesets (/etc/sysconfig/{iptables,ip6tables}) to permit traffic on UDP port 514:

-A RH-Firewall-1-INPUT –s 192.168.10.0/25 -m udp -p udp --dport 514 -j ACCEPT

-A RH-Firewall-1-INPUT –s fc00::/7 -m udp -p udp --dport 514 -j ACCEPT

Change the “-s” parameter to accurately reflect the range of IP addresses you want to accept syslog messages from. The stock syslogd has no authentication so this is basically your only line of defense against random machines sending messages to you.

Reload your iptables/ip6tables rules with:

/usr/bin/sudo /sbin/service iptables restart
/usr/bin/sudo /sbin/service ip6tables restart

2. Edit /etc/sysconfig/syslog. Add “-r” to the SYSLOGD_OPTIONS line:

SYSLOGD_OPTIONS="-m 0 -r"

Restart syslogd with:

/usr/bin/sudo /sbin/service syslog restart

(note that the service is ‘syslog’ and not ‘syslogd’)

3. Verify that syslogd is listening on port 514 using netstat:

$ sudo netstat –anp | grep 514
udp        0      0 0.0.0.0:514      0.0.0.0:*       5332/syslogd

4. Change another host to use the syslogd host. On another Linux box the format is in /etc/syslog.conf is something like:

*.info;cron.!=info;mail.none;local0.notice          @logs.company.com

where logs.company.com is the machine you just set up to listen to syslog messages.

You’ll need to restart that host’s syslog to make the change. If you “tail –f /var/log/messages” on the log host you should be able to use /usr/bin/logger on the client host to make messages appear.

Client:

$ logger hey

Syslog server:

$ sudo tail -f /var/log/messages
…
Jan 13 15:33:37 clienthost plankers: hey

Keep in mind that syslog will sort messages into the categories it already has defined in /etc/syslog.conf. So if you send mail log data (mail.*) they’ll end up in /var/log/maillog by default.

5. You may wish to change your log rotation schedule to prevent large files. You can do this in /etc/logrotate.conf.

6. Searching, etc. can be done with standard UNIX tools like grep, tail, less, etc. in /var/log.

Advanced topics:

If you like IP addresses instead of DNS names you can use the “-x” option in /etc/sysconfig/syslog.

I suggest creating a service DNS name for your syslog host, like “logs.company.com.” That way you can move it later without having to update all your hosts, and it looks more professional. It’s also easier to remember, especially if you go the Splunk route with the web interface.

If you want more advanced filtering, or database support, look into syslog-ng or rsyslog. rsyslog has the advantage of shipping by default with Red Hat Enterprise Linux 6, too, so that should make it easy to create remote syslog services that can do more (filtering by hosts, etc.). If you want to retrofit a Red Hat Enterprise Linux 5 host with another syslog daemon I suggest creating a service IP to go along with the service name. There have been (and are reported to be) various bugs present in the stock syslog daemons wherein they bind to 514/udp regardless of whether you told them to or not. Using a service IP appears to alleviate that in some cases.