ASA stuck on “Booting system, please wait…” after power cycle.

A Cisco ASA5550 was stuck on “Booting system, please wait…” after a power cycle (physically turning off and on, not just a reload). It was impossible to break into ROMMON from here.

After taking the cover off and doing some experimentation, the issue was found to be a faulty DIMM slot. We removed all DIMMs and replaced them one by one. Upon placing a DIMM in slot 1, the firewall failed to boot again. We swapped DIMMs 1 and 2, still no joy.

Removing the DIMM from slot 1 again meant that the firewall came back up.

Solution: If an RMA is going to take time, bring the firewall back up with less memory. Otherwise, swap the thing out straight away as you don’t know what else the power cycle has fried!

You can make your life easier doing the replacement by moving the old ASA’s compact flash card to the external (disk1) slot of the new one. You can then get the right OS on the replacement quickly at least.

NEWASA# copy disk1:/[IOS-image-name.bin] disk0:/[IOS-image-name.bin]
NEWASA# boot system disk0:/[IOS-image-name.bin]
NEWASA# wr mem

Note: you can copy the running-config on the old ASA to a visible file on CF (eg: copy running-config disk0:/myconfig.cfg), but copying that from disk1 to running-config on the replacement tends to not work very well if you have TACACS config in place. Good old copy and paste from a backup is the way forward.

Splunk Cisco ASA App – Getting it working!

There are some apps on splunkbase for Cisco Firewalls (in particular a Cisco Security Suite and Cisco ASA App) – these work well but there are a few gotchas that stop this app from working.

Prerequisites: Install the latest Sideview Utils from http://sideviewapps.com/apps/sideview-utils and install the Google Maps app from splunkbase.

1) Ensure that you have a “firewall” index created and searchable by the appropriate roles. Be careful if the firewall index is owned by another app; if you remove that app then the index will disappear and you’ll wonder why this one is no longer working!

2) Ensure that the source is being tagged for the “firewall” index (if using a forwarder, you need to set index = firewall in the monitor statement)

3) Copy the etc/apps/Splunk_CiscoFirewalls/default/transforms.conf and props.conf files into the etc/apps/Splunk_CiscoFirewalls/local directory, and edit the local version of transforms.conf so that the the asa sourcetype is correctly set. This must depend on software version but one is commented out here. You may need to swap these around: certainly on 8.2 the log format is ASA- and not ASA–

[force_sourcetype_for_cisco_asa]
DEST_KEY = MetaData:Sourcetype
REGEX = %ASA-\d+-\d+
#REGEX = %ASA--\d+-\d+
FORMAT = sourcetype::cisco_asa

If you really need to cater for both eventualities, then you could use:

REGEX = %ASA--?\d+-\d+

4) I also came across an issue where the sourcetypes were being correctly set, but the host field was incorrectly being detected as the machine running my light forwarder. I got around this by editing the etc/apps/Splunk_CiscoFirewalls/local/props.conf file and changing the first TRANSFORMS line, adding syslog-host as the final entry:

#[source::...cisco]
TRANSFORMS-force-sourcetype_for_cisco_devices = force_sourcetype_for_cisco_pix, force_sourcetype_for_cisco_asa, force_sourcetype_for_cisco_fwsm, force_sourcetype_for_cisco_acs, force_sourcetype_for_cisco_ios, force_sourcetype_for_cisco_catchall, syslog-host

5) This app also has a cisco “catch-all” sourcetype formatter which may cause problems with other apps (eg: they might expect sourcetype=syslog or cisco_syslog). You may want to comment this out because it’s not exhaustive and will result in some of your cisco logs being split sourcetype:

#[force_sourcetype_for_cisco_catchall]
#DEST_KEY = MetaData:Sourcetype
#REGEX = :\s\%((SNMP|CDP|FAN|LINE|LINEPROTO|RTD|SYS|C\d+_[^-]+)-\d+-\S+)
#FORMAT = sourcetype::cisco