Policy tracing from CLI on firewalls.

This post is just a quick reference on how to see whether or not some traffic will pass the rulebase or not.

Tracing policy matching on Juniper SRX (for actual live sessions use ‘show security-flow session’)

show security match-policies from-zone trust to-zone untrust source-port 1024 destination-port 40961 protocol tcp source-ip destination-ip

Tracing policy matching on Cisco ASA (for live connections, use ‘show conn’)

packet-tracer input inside tcp 1024 443 [detailed]

Tracing policy matching on Palo Alto

test security-policy-match application twitter-posting source-user acme\mcanha destination destination-port 80 source protocol 6 

NB: protocol 6 = tcp, 17= UDP, 1 = ICMP
see https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml

Cisco ASA denying traffic between interfaces despite correct policy

After a bit of head-scratching and frustration, I found an issue on a Cisco ASA (v9.1) with the configuration that has caught me out twice previously. I felt like a bit of an idiot for not clicking sooner.

Basically the policy was allowing “icmp any any” both sides along with the connection traffic from the specific external to internal interfaces, however the logs were showing denials:

Inbound TCP connection denied from A.B.C.D/53112 to W.X.Y.Z/443 flags SYN  on interface Ext-2
Deny inbound icmp src Ext-2:A.B.C.D dst Ext-1:W.X.Y.Z (type 8, code 0)

The issue here was that both interfaces were configured with the same security-level. This is a hangover from the old PIX days where this kind of communication was denied by default. The policy doesn’t seem to matter.

The fix in this case was that the security-level on the “dirty” interface was lowered to a more appropriate value. To allow ACLs to control communications between same-level interfaces, the following command can be used:

same-security-traffic permit inter-interface

ASA stuck on “Booting system, please wait…” after power cycle.

A Cisco ASA5550 was stuck on “Booting system, please wait…” after a power cycle (physically turning off and on, not just a reload). It was impossible to break into ROMMON from here.

After taking the cover off and doing some experimentation, the issue was found to be a faulty DIMM slot. We removed all DIMMs and replaced them one by one. Upon placing a DIMM in slot 1, the firewall failed to boot again. We swapped DIMMs 1 and 2, still no joy.

Removing the DIMM from slot 1 again meant that the firewall came back up.

Solution: If an RMA is going to take time, bring the firewall back up with less memory. Otherwise, swap the thing out straight away as you don’t know what else the power cycle has fried!

You can make your life easier doing the replacement by moving the old ASA’s compact flash card to the external (disk1) slot of the new one. You can then get the right OS on the replacement quickly at least.

NEWASA# copy disk1:/[IOS-image-name.bin] disk0:/[IOS-image-name.bin]
NEWASA# boot system disk0:/[IOS-image-name.bin]
NEWASA# wr mem

Note: you can copy the running-config on the old ASA to a visible file on CF (eg: copy running-config disk0:/myconfig.cfg), but copying that from disk1 to running-config on the replacement tends to not work very well if you have TACACS config in place. Good old copy and paste from a backup is the way forward.

Splunk Cisco ASA App – Getting it working!

There are some apps on splunkbase for Cisco Firewalls (in particular a Cisco Security Suite and Cisco ASA App) – these work well but there are a few gotchas that stop this app from working.

Prerequisites: Install the latest Sideview Utils from http://sideviewapps.com/apps/sideview-utils and install the Google Maps app from splunkbase.

1) Ensure that you have a “firewall” index created and searchable by the appropriate roles. Be careful if the firewall index is owned by another app; if you remove that app then the index will disappear and you’ll wonder why this one is no longer working!

2) Ensure that the source is being tagged for the “firewall” index (if using a forwarder, you need to set index = firewall in the monitor statement)

3) Copy the etc/apps/Splunk_CiscoFirewalls/default/transforms.conf and props.conf files into the etc/apps/Splunk_CiscoFirewalls/local directory, and edit the local version of transforms.conf so that the the asa sourcetype is correctly set. This must depend on software version but one is commented out here. You may need to swap these around: certainly on 8.2 the log format is ASA- and not ASA–

DEST_KEY = MetaData:Sourcetype
REGEX = %ASA-\d+-\d+
#REGEX = %ASA--\d+-\d+
FORMAT = sourcetype::cisco_asa

If you really need to cater for both eventualities, then you could use:

REGEX = %ASA--?\d+-\d+

4) I also came across an issue where the sourcetypes were being correctly set, but the host field was incorrectly being detected as the machine running my light forwarder. I got around this by editing the etc/apps/Splunk_CiscoFirewalls/local/props.conf file and changing the first TRANSFORMS line, adding syslog-host as the final entry:

TRANSFORMS-force-sourcetype_for_cisco_devices = force_sourcetype_for_cisco_pix, force_sourcetype_for_cisco_asa, force_sourcetype_for_cisco_fwsm, force_sourcetype_for_cisco_acs, force_sourcetype_for_cisco_ios, force_sourcetype_for_cisco_catchall, syslog-host

5) This app also has a cisco “catch-all” sourcetype formatter which may cause problems with other apps (eg: they might expect sourcetype=syslog or cisco_syslog). You may want to comment this out because it’s not exhaustive and will result in some of your cisco logs being split sourcetype:

#DEST_KEY = MetaData:Sourcetype
#REGEX = :\s\%((SNMP|CDP|FAN|LINE|LINEPROTO|RTD|SYS|C\d+_[^-]+)-\d+-\S+)
#FORMAT = sourcetype::cisco

Resolve MAC addresses to Port, IP and DNS Name

Resolving MAC address to port, IP and DNS or name service name (or more simply for some, resolve mac to name) is a challenge that every network engineer has come across at some point in their career. It’s easily solved with a bit of thought and logic. Unfortunately the past few products I’ve dealt in the past with for this purpose have either been abandoned or aren’t as multi-vendor as I’d like, so it seems that the only solution is to write your own… bash and expect is sufficient.

If you’re thinking about doing this (and it’s a great learning exercise), you need to get around the following:

– Determining which interfaces are trunks on the switches so you can strip those MAC entries out (CDP works quite well)
– Converting ARP and MAC info into a “clean” format (eg: CatOS and IOS output is a different format)
– Detecting the fields across various pieces of hardware as display output isn’t always consistent for the same commands
– Inconsistent logins/passwords
– Correlating the IP/MAC/Interface information together. This can be done with the UNIX join command and some awk/sed
– What you do with MACs that don’t resolve to an IP address (I include a flag to print these if required)
– Whether the machine you run DNS queries on will be able to resolve the IPs to PTR records
– If using expect, stripping out stray characters (eg \r) that will mess up your greps and other string searches
– Add plenty of debugging so you can quickly tell why something isn’t working properly

I used expect to go and grab the ARP, CDP and MAC information seeing as you can’t get all the required information from SNMP on many devices these days. In my case, this results in the following type of output:

Switch       Interface       VLAN  MAC             IP               DNSName
nycsw12      Fa3/10          100   0060.b0aa.0000    NO_DNS
nycsw12      Fa2/16          99    1060.4b61.0001     nyc-pc573.company.corp.
nycsw12      Fa2/37          101   1060.4b64.0002    nyc-pc555.company.corp.
nycsw12      Fa2/42          101   1060.4b68.0003   nyc-pc572.company.corp.
nycsw12      Fa2/45          98    1060.4b6a.0004     nyc-pc588.company.corp.
nycsw12      Fa2/32          98    1060.4b6a.0005    nyc-pc601.company.corp.
nycsw12      Fa3/3           100   2c41.389e.d19f    nyc-pc480.company.corp.
nycsw13      Fa2/4           100   5c26.0a01.0ac4    nyc-pc246.company.corp.
nycsw13      Fa2/6           100   6c3b.e531.2ddf    nyc-pc745.company.corp.

Of course, you can always just use Excel to do a VLOOKUP of your mac-address table output against a sorted table containing all your arp entries, but that’s a bit less automatic.