Tag Archives: splunk

Drilldown on a Single Value Field in Splunk

  By Sol | September 20, 2013 - 21:33 | Splunk
Leave a comment

By default you can’t drill down on a single value field visualisation in a splunk view if you are using a rangemap to change colours.

eg: rangemap field=count low=0-0 default=elevated

This can be circumvented with the following addition to the XML in the dashboard (thankfully this works in simplified XML):

      <option name="linkFields">result</option>
      <option name="linkSearch">
        search index=main c_msg_severity=0   
      </option>
      <option name="linkView">flashtimeline</option>

Splunk Cisco ASA App – Getting it working!

  By Sol | September 18, 2013 - 11:42 | ASA, Splunk
Leave a comment

There are some apps on splunkbase for Cisco Firewalls (in particular a Cisco Security Suite and Cisco ASA App) – these work well but there are a few gotchas that stop this app from working.

Prerequisites: Install the latest Sideview Utils from http://sideviewapps.com/apps/sideview-utils and install the Google Maps app from splunkbase.

1) Ensure that you have a “firewall” index created and searchable by the appropriate roles. Be careful if the firewall index is owned by another app; if you remove that app then the index will disappear and you’ll wonder why this one is no longer working!

2) Ensure that the source is being tagged for the “firewall” index (if using a forwarder, you need to set index = firewall in the monitor statement)

3) Copy the etc/apps/Splunk_CiscoFirewalls/default/transforms.conf and props.conf files into the etc/apps/Splunk_CiscoFirewalls/local directory, and edit the local version of transforms.conf so that the the asa sourcetype is correctly set. This must depend on software version but one is commented out here. You may need to swap these around: certainly on 8.2 the log format is ASA- and not ASA–

[force_sourcetype_for_cisco_asa]
DEST_KEY = MetaData:Sourcetype
REGEX = %ASA-\d+-\d+
#REGEX = %ASA--\d+-\d+
FORMAT = sourcetype::cisco_asa

If you really need to cater for both eventualities, then you could use:

REGEX = %ASA--?\d+-\d+

4) I also came across an issue where the sourcetypes were being correctly set, but the host field was incorrectly being detected as the machine running my light forwarder. I got around this by editing the etc/apps/Splunk_CiscoFirewalls/local/props.conf file and changing the first TRANSFORMS line, adding syslog-host as the final entry:

#[source::...cisco]
TRANSFORMS-force-sourcetype_for_cisco_devices = force_sourcetype_for_cisco_pix, force_sourcetype_for_cisco_asa, force_sourcetype_for_cisco_fwsm, force_sourcetype_for_cisco_acs, force_sourcetype_for_cisco_ios, force_sourcetype_for_cisco_catchall, syslog-host

5) This app also has a cisco “catch-all” sourcetype formatter which may cause problems with other apps (eg: they might expect sourcetype=syslog or cisco_syslog). You may want to comment this out because it’s not exhaustive and will result in some of your cisco logs being split sourcetype:

#[force_sourcetype_for_cisco_catchall]
#DEST_KEY = MetaData:Sourcetype
#REGEX = :\s\%((SNMP|CDP|FAN|LINE|LINEPROTO|RTD|SYS|C\d+_[^-]+)-\d+-\S+)
#FORMAT = sourcetype::cisco

Selectively monitor files in a directory with Splunk Forwarder

  By Sol | September 16, 2013 - 11:22 | Splunk
Leave a comment

Scenario: Lots of log files all in the same directory on a remote host, we don’t want to monitor all of them and we don’t want to specify a long list of files to monitor in our forwarding configuration.

Solution: Use a blacklist entry. The below example monitors all files in the /logs directory, sets a sourcetype of fw_log and ignores any filenames ending with LONDONA or AMSTERDAMA

Edit file: /home/splunk/opt/splunkforwarder/etc/system/local/inputs.conf

[monitor:///logs/]
disabled = false
blacklist = (\.LONDONA$|\.AMSTERDAMA$)
sourcetype = fw_log
index = firewall

Or to use a wildcard and monitor certain files..

[monitor://logs/firewallmsgs.*]
disabled = false
sourcetype = asa_log
blacklist = (\.LONDONA$|\.AMSTERDAMA$)
index = firewall

Similarly, we can create a whitelist instead:

[monitor:///logs/messages.fw.*]
whitelist = (\.CDCA$|\.CDCB$)
disabled = false
index = firewall

(we could use add monitor /logs/ -index main -sourcetype fw_log but as we’re blacklisting, we may as well edit manually)

Note: forwarder was added with

./splunk add forward-server remotehostname:9997

Check forwarding with:

splunk list forward-server
Splunk username: admin
Password:
Active forwards:
        remotehostname:9997
Configured but inactive forwards:
        None

Then configuring a receiver on port 9997 on the indexer.

How to do static lookups with Splunk and a CSV file

  By Sol | September 11, 2013 - 22:37 | Splunk
Leave a comment

The documentation for the lookup function on Splunk’s website proved to be immensely confusing and did me no favours at all. A few hours of trial and error later, I figured it out. This example will use a csv file that translates IP Protocol to Names intended for use with an SRX app I’m working on.

Procedure

Create a csv file with headers. Bear in mind that the key you want to look up against (the first field) MUST be an existing field name in Splunk. At the time I had a field extraction called srx_protocol_id so the first field is labelled as such. Be careful of calling your lookups and primary keys by the same name, otherwise it gets confusing.

ip-protocol-numbers.csv (truncated)

srx_protocol_id,protocol_name
0,HOPOPT
1,ICMP
2,IGMP
3,GGP
4,IPv4
5,ST
6,TCP
7,CBT
8,EGP
9,IGP
10,BBN-RCC-MON
11,NVP-II
12,PUP
13,ARGUS
14,EMCON
15,XNET
16,CHAOS
17,UDP

First of all, select App > Your App from the top menu.

Go to Manager > Lookups > Lookup Table Files > New

Select the relevant App, select the file with [ Choose File ] and give the same filename, eg: ip-protocol-numbers.csv

Once done, your path may be an admin path, so edit the permissions of the file you uploaded to be “This app only” and select Read for Everyone and write for admin and power. The path should change to

home/splunk/opt/splunk/etc/apps/[APPNAME]/lookups/ip-protocol-numbers.csv

Create a protocol definition.

Destination App: Your App
Name: ip-protocols
Type: File based
Lookup file: ip-protocol-numbers.csv
[X] Advanced options

Minimum matches: 1
Maximum matches: 1
Default matches: NA

The above settings are chosen because we only expect 1 name per protocol ID, and if it doesn’t match, we want NA to be returned. If the Lookup file you want isn’t showing up and your permissions are OK, you may need to restart splunk.

Searching
Now I can look up the protocol name in the following search:

RT_FLOW_SESSION | dedup 1 host,srx_sess_id keepempty=true | top limit=0 srx_protocol_id  | lookup ip-protocols srx_protocol_id | fields - srx_protocol_id percent | rename protocol_name AS "Protocol"

By default, a lookup will return all fields that match the primary key. If you have several fields you can modify your search with the following to restrict the fields you want output.

lookup [lookup-name] [search field name] OUTPUT [field name in csv file]

Splunk Field Extractions for Juniper SRX

  By Sol | September 11, 2013 - 09:49 | Juniper, Security, Splunk
Leave a comment

The first two were found elsewhere on the web but I noticed there was no deny event extraction so made my own.

To make your SRX send syslogs, the following example can be modified. You might find it easier to use local facilities to split out your logs by type using syslog-ng. Be sure to monitor performance as you enable logging – lots of logging on a extremely busy firewall may generate a fair bit of extra CPU overhead.

SRX Config

    syslog {
         host 192.168.1.100 {
            any any;
            match RT_FLOW_SESSION;
            facility-override local5;
            source-address 10.0.1.254;
        }
    }

Set your desired policies to log… eg:

edit security policies from-zone trust to-zone untrust
set policy web-traffic-outbound then log session-init session-close
set policy default-drop-trust-untrust then log session-init session-close

Splunk Extractions

Create events

RT_FLOW_SESSION_CREATE:\ssession\screated\s(?P<srx_src_ip>\d+\.\d+\.\d+\.\d+)\/(?P<srx_src_port>\d+)\D+(?P<srx_dst_ip>\d+\.\d+\.\d+\.\d+)\/(?P<srx_dst_port>\d+)\s(?P<srx_svc_name>\S+)\s(?P<srx_nat_src_ip>\d+\.\d+\.\d+\.\d+)\/(?P<srx_nat_src_port>\d+)\D+(?P<srx_nat_dst_ip>\d+\.\d+\.\d+\.\d+)\/(?P<srx_nat_dst_port>\d+)\s(?P<srx_src_nat_rule_name>\S+)\s(?P<srx_dst_nat_rule_name>\S+)\s(?P<srx_protocol_id>\d+)\s(?P<srx_policy_name>\S+)\s(?P<srx_src_zone>\S+)\s(?P<srx_dst_zone>\S+)\s(?P<srx_sess_id>\d+)

Close events

RT_FLOW_SESSION_CLOSE:\ssession\sclosed\s(?P<srx_closed_reason>[^:]+)\D+(?P<srx_src_ip>\d+\.\d+\.\d+\.\d+)\/(?P<srx_src_port>\d+)\D+(?P<srx_dst_ip>\d+\.\d+\.\d+\.\d+)\/(?P<srx_dst_port>\d+)\s(?P<srx_svc_name>\S+)\s(?P<srx_nat_src_ip>\d+\.\d+\.\d+\.\d+)\/(?P<srx_nat_src_port>\d+)\D+(?P<srx_nat_dst_ip>\d+\.\d+\.\d+\.\d+)\/(?P<srx_nat_dst_port>\d+)\s(?P<srx_src_nat_rule_name>\S+)\s(?P<srx_dst_nat_rule_name>\S+)\s(?P<srx_protocol_id>\d+)\s(?P<srx_policy_name>\S+)\s(?P<srx_src_zone>\S+)\s(?P<srx_dst_zone>\S+)\s(?P<srx_sess_id>\d+)\s(?P<srx_pkts_from_client>\d+)\((?P<srx_bytes_from_client>\d+)\)\s(?P<srx_pkts_from_server>\d+)\((?P<srx_bytes_from_server>\d+)\)\s(?P<srx_sess_elapsed_time>\d+)

Deny Events

RT_FLOW_SESSION_DENY:\ssession\sdenied\s(?P<srx_src_ip>\d+\.\d+\.\d+\.\d+)\/(?P<srx_src_port>\d+)\D+(?P<srx_dst_ip>\d+\.\d+\.\d+\.\d+)\/(?P<srx_dst_port>\d+)\s(?P<srx_svc_name>\S+)\s(?P<srx_protocol_id>\d+)\((?P<srx_icmp_type>\d+)\)\s(?P<srx_policy_name>\S+)\s(?P<srx_src_zone>\S+)\s(?P<srx_dst_zone>\S+)

Policy Action Field (common)

RT_FLOW:\s\S+:\ssession\s(?P<srx_policy_action>\S+)

Next personal project: Write a decent Splunk app for SRX!