Filtering by Host and Message in syslog-ng

I had a requirement to filter logs by source IP and by part of the messages, sending them off to another file. Proof of concept was done as follows:

– Single source IP for logs
– Match two source subnets in message text

Configuration in /etc/syslog-ng.conf is:

source s_net {
       tcp(max-connections(10));
       udp();
};

destination d_test { file ("/var/log/test.log"); };

filter f_test { (netmask(178.79.153.178/32)) and
                    (message("Client IP: 10\.1\.1\.*") or
                     message("Client IP: 10\.10\.10\.*")) 
};

log { source(s_sys); filter(f_test); destination(d_test); };

This separates out logs containing the specific Client IP addresses and doesn’t seem to hit CPU too hard. Netmask was used rather than host to avoid any issues with hostnames being resolved and then not matching the filter.

Syslog-ng and log rotation

I decided that I would change the way that system logs are archived via a custom (ie: non logrotate) script, however, I managed to forget that using unix “mv” means that the inode of the new filename is the same as the old filemane. rm the new moved temporary file and you’re in trouble.

When rotating logs without logrotate it’s probably best to use a command that will read until end of file properly (such as gzip) then truncate (eg cp /dev/null targetfile) the original. This way, loss is minimal and inode doesn’t change. I thought I’d be smart, move the log and then gzip it, hoping that syslog-ng would be smart enough to create a new file and hence new inode. Unfortuanately, this meant that syslog-ng kept logging to the moved file, then threw a fit and stopped logging altogether once the mv’d temporary file was gzipped off elsewhere and then deleted.

Summary:

– Use an archiving solution that properly reads through to EOF before you truncate the original file.
– Use a postrotate action with logrotate to send a HUP signal to the process (eg: syslog-ng) after archiving/moving each file. copytruncate may be a useful option with logrotate here too.

I’ve seen a few posts on forums that blame syslog-ng, but the fact is that this is the way that UNIX behaves.

Using syslog-ng to split logs by facility and/or hostname

An example (not exhaustive) syslog-ng file to filter out logs on local6 facility to specific files using the $HOST macro. If you’d configured all your firewalls to log at local6 (facility 22) then this might be of use. Also takes in other remote syslogs from our network devices and puts them into a single file (and excludes local6 as we already have those elsewhere).

BEWARE of some vendors using local facilities for their syslogs. Juniper and F5 tend to do this a lot. You may want to override syslog facilities on the devices themselves.

Permissions should be changed to your preference.

The line we’re interested in is:

destination dst_fwlog { file("/netlogs/messages.firewall.$HOST" perm (0644)); };

Example config below:

options { long_hostnames(off);
        sync(0);
        owner(root);
        group(root);
        use_dns(persist_only);
        dns_cache_hosts(/etc/hosts);
};

source src_local {
        internal();
        unix-stream("/dev/log");
        file("/proc/kmsg");
};

source src_remote{
        tcp(max-connections(50));
        udp();
};

destination dst_messages { file("/var/log/messages" perm(0644)); };
destination dst_authpriv { file("/var/log/secure" perm(0600)); };
destination dst_maillog { file("/var/log/maillog" perm(0600)); };
destination dst_cron { file("/var/log/cron" perm(0600)); };
destination dst_alltty { usertty("*"); };

destination dst_netlog { file("/netlogs/messages" perm(0644)); };
destination dst_fwlog { file("/netlogs/messages.firewall.$HOST" perm (0644)); };

filter f_syslog { not facility(authpriv, mail, cron, local6); };
filter f_authpriv { facility(auth, authpriv); };
filter f_mail { facility(mail); };
filter f_cron { facility(cron); };
filter f_local5 { facility(local5); };
filter f_local6 { facility(local6); };
filter f_info_to_notice { level(info .. notice); };
filter f_emerg { level(emerg); };

# Local Syslog
log { source(src_local); filter(f_syslog); destination(dst_messages); };

# Local Authpriv
log { source(src_local); filter(f_authpriv); destination(dst_authpriv); };

# Local Mail
log { source(src_local); filter(f_mail); destination(dst_maillog); };

# Local Cron
log { source(src_local); filter(f_cron); destination(dst_cron); };

# Local Emerg
log { source(src_local); filter(f_emerg); destination(dst_alltty); };

# Remote Syslog Network Devices
log { source(src_remote); filter(f_syslog); destination(dst_netlog); };

# Remote syslog Firewalls (local6 override on all devices)
log { source(src_remote); filter(f_local6); destination(dst_fwlog); };