Adj resolve request: Failed to resolve… [ Cisco 3750X ]

After a migration to a pair of 3750Xs I was getting a bunch of disconcerting ARP errors in the logs. After a bit of digging, this appears to be a known bug. Error message content examples are shown below:

Nov  4 12:41:00: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 172.17.22.57 Vlan21
Nov  4 12:41:18: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 172.17.24.53 Vlan28

This was fixed/worked around with:

no ip cef optimize neighbor resolution

GRE over IPSEC between Juniper and Cisco Router

This caused headaches as it needed slightly different configuration to normal. ip mtu not being set here was the cause of things sort-of-but-not-quite-working.

Normally with Cisco to Cisco over IPSEC we’d add “ip tcp adjust mss-1392” to the Tunnel interfaces either side.

This is the config that worked in the end.

GRE Juniper router side
=======================

interfaces {
    lo0 {
        unit 0 {
            family inet {
                address 192.168.255.1/32;
            }
        }
    }

    gr-1/1/10 {
        unit 2 {
            clear-dont-fragment-bit;
            description "-= Gre Tunnel to Remote Office =-";
            tunnel {
                source 192.168.255.1;
                destination 192.168.255.2;
            }
            family inet {
                mtu 1400;
                address 10.0.0.2/30;
            }
        }
    }
}

routing-options
    static {
        route 192.168.255.2/32 next-hop [IPSEC FW Addr];
    }
}


---------


GRE Cisco Side
==============

interface Loopback1
 description * Loopback for GRE Tunnel source/endpoint *
 ip address 192.168.255.2 255.255.255.255


interface Tunnel2
 description * GRE Tunnel to Juniper GR-1/1/10.2 *
 ip address 10.0.0.1 255.255.255.252
 ip mtu 1400
 load-interval 30
 tunnel source Loopback1
 tunnel destination 192.168.255.1
 hold-queue 2000 in
 hold-queue 2000 out

! Route GRE endpoint via IPSEC FW
ip route 192.168.255.1 255.255.255.255 [IPSEC FW Addr]

Reference config for normal Cisco – Cisco.

interface Tunnel2
 description * GRE Tunnel to Remote site int tunnel2 *
 bandwidth 2000
 ip address 10.0.0.1 255.255.255.252
 ip tcp adjust-mss 1392
 load-interval 30
 tunnel source Loopback1
 tunnel destination 192.168.16.13
 hold-queue 2000 in
 hold-queue 2000 out

NNMi 9.24 Custom Poller Bus Adapter Errors

I was getting the following alerts regarding the custom poller on NNMi which was annoying users with the constant on/off status alerts.

“The Performance SPI Custom Poller Bus Adapter has status Critical because the average input queue duration…..”
“The CustomPoller Export Bus Adapter has status Minor because file space limit (2,000 MB) has been reached and older export data files are being removed to make room for new files.”

After some digging, it seems that the older files weren’t being deleted so this issue had actually crept up over time.

Apparently this was fixed by patch 5 but a workaround until then is to delete older files manually as follows. I strongly suggest running the find command without the “| xargs rm” part first to verify that you are indeed only finding regular files within the correct directory.

# cd /var/opt/OV/shared/nnm/databases/custompoller/export/final
# find . -mtime +365 -type f | xargs rm

Check the usage is under the size limit:

du -hs .
649M    .

Print specified line number in file with sed

How to use sed to print a specific line in a file, and also how to use a variable within the command to print a specific line from that file.

Handy for generating random passphrases from a dictionary file, for example… you’d just create a loop and pick 4 random lines from a dictionary file.

[sol@testbox scripts]$ sed '52q;d' /usr/share/dict/words
AAA
[sol@testbox scripts]$ LINENUM=52
[sol@testbox scripts]$ sed ''"$LINENUM"'q;d' /usr/share/dict/words
AAA

OR more simply

[sol@testbox scripts]$ sed $LINENUM'q;d' /usr/share/dict/words
AAA

Script to get cisco memory usage

Quick and dirty script to get the memory pool usage from a Cisco IOS device. Uses join to put everything together so needs to write temporary files to the current directory. Perhaps change the output of these to /tmp/filename if desired.

Tested on a variety of devices but may not cover everything and doesn’t work on Nexus as there is no response from these OID requests.

#!/bin/sh
# Get memory usage from Cisco IOS devices
# sol@subnetzero.org
# Vars
NODE=$1
STRING=READSTRING
if [ ! -w . ]; then
   echo "Can't write temp files to current dir."
   exit 1
fi
#MEMFREE
snmpwalk -v2c -c $STRING $NODE .1.3.6.1.4.1.9.9.48.1.1.1.6 |\
 sed 's/^.*\.//g'  |\
 awk '{print $1,$4}' > memfree.tmp
if [ ! -s memfree.tmp ]; then
   echo "Invalid host or response."
   exit 1
fi

#MEMUSED
snmpwalk -v2c -c $STRING $NODE .1.3.6.1.4.1.9.9.48.1.1.1.5 |\
 sed 's/^.*\.//g'  |\
 awk '{print $1,$4}' > memused.tmp

#MEMTYPE
snmpwalk -v2c -c $STRING $NODE .1.3.6.1.4.1.9.9.48.1.1.1.2 |\
 sed 's/^.*\.//g' |\
 awk '{print $1,$4}' > memtype.tmp

# Join together
join -t" " -j1 1 -j2 1 -o 1.1 1.2 2.2 memfree.tmp memused.tmp > memcombined.tmp
join -t" " -j1 1 -j2 1 -o 1.1 1.2 2.2 2.3 memtype.tmp memcombined.tmp > memtotals.tmp

# Print output
awk '{printf ("%-4.4s%-12.12s%3.1f%%\n",$1,$2,$4/($3+$4)*100)}' < memtotals.tmp

#Cleanup
rm mem*.tmp
[sol@testserver1 scripts]$ ./getmem testrtr01
1   "Processor" 23.7%
2   "I/O"       2.1%