Filtering by Host and Message in syslog-ng

I had a requirement to filter logs by source IP and by part of the messages, sending them off to another file. Proof of concept was done as follows:

– Single source IP for logs
– Match two source subnets in message text

Configuration in /etc/syslog-ng.conf is:

source s_net {
       tcp(max-connections(10));
       udp();
};

destination d_test { file ("/var/log/test.log"); };

filter f_test { (netmask(178.79.153.178/32)) and
                    (message("Client IP: 10\.1\.1\.*") or
                     message("Client IP: 10\.10\.10\.*")) 
};

log { source(s_sys); filter(f_test); destination(d_test); };

This separates out logs containing the specific Client IP addresses and doesn’t seem to hit CPU too hard. Netmask was used rather than host to avoid any issues with hostnames being resolved and then not matching the filter.

Using syslog-ng to split logs by facility and/or hostname

An example (not exhaustive) syslog-ng file to filter out logs on local6 facility to specific files using the $HOST macro. If you’d configured all your firewalls to log at local6 (facility 22) then this might be of use. Also takes in other remote syslogs from our network devices and puts them into a single file (and excludes local6 as we already have those elsewhere).

BEWARE of some vendors using local facilities for their syslogs. Juniper and F5 tend to do this a lot. You may want to override syslog facilities on the devices themselves.

Permissions should be changed to your preference.

The line we’re interested in is:

destination dst_fwlog { file("/netlogs/messages.firewall.$HOST" perm (0644)); };

Example config below:

options { long_hostnames(off);
        sync(0);
        owner(root);
        group(root);
        use_dns(persist_only);
        dns_cache_hosts(/etc/hosts);
};

source src_local {
        internal();
        unix-stream("/dev/log");
        file("/proc/kmsg");
};

source src_remote{
        tcp(max-connections(50));
        udp();
};

destination dst_messages { file("/var/log/messages" perm(0644)); };
destination dst_authpriv { file("/var/log/secure" perm(0600)); };
destination dst_maillog { file("/var/log/maillog" perm(0600)); };
destination dst_cron { file("/var/log/cron" perm(0600)); };
destination dst_alltty { usertty("*"); };

destination dst_netlog { file("/netlogs/messages" perm(0644)); };
destination dst_fwlog { file("/netlogs/messages.firewall.$HOST" perm (0644)); };

filter f_syslog { not facility(authpriv, mail, cron, local6); };
filter f_authpriv { facility(auth, authpriv); };
filter f_mail { facility(mail); };
filter f_cron { facility(cron); };
filter f_local5 { facility(local5); };
filter f_local6 { facility(local6); };
filter f_info_to_notice { level(info .. notice); };
filter f_emerg { level(emerg); };

# Local Syslog
log { source(src_local); filter(f_syslog); destination(dst_messages); };

# Local Authpriv
log { source(src_local); filter(f_authpriv); destination(dst_authpriv); };

# Local Mail
log { source(src_local); filter(f_mail); destination(dst_maillog); };

# Local Cron
log { source(src_local); filter(f_cron); destination(dst_cron); };

# Local Emerg
log { source(src_local); filter(f_emerg); destination(dst_alltty); };

# Remote Syslog Network Devices
log { source(src_remote); filter(f_syslog); destination(dst_netlog); };

# Remote syslog Firewalls (local6 override on all devices)
log { source(src_remote); filter(f_local6); destination(dst_fwlog); };