Adj resolve request: Failed to resolve… [ Cisco 3750X ]

After a migration to a pair of 3750Xs I was getting a bunch of disconcerting ARP errors in the logs. After a bit of digging, this appears to be a known bug. Error message content examples are shown below:

Nov  4 12:41:00: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 172.17.22.57 Vlan21
Nov  4 12:41:18: %ADJ-3-RESOLVE_REQ: Adj resolve request: Failed to resolve 172.17.24.53 Vlan28

This was fixed/worked around with:

no ip cef optimize neighbor resolution

Resolve MAC addresses to Port, IP and DNS Name

Resolving MAC address to port, IP and DNS or name service name (or more simply for some, resolve mac to name) is a challenge that every network engineer has come across at some point in their career. It’s easily solved with a bit of thought and logic. Unfortunately the past few products I’ve dealt in the past with for this purpose have either been abandoned or aren’t as multi-vendor as I’d like, so it seems that the only solution is to write your own… bash and expect is sufficient.

If you’re thinking about doing this (and it’s a great learning exercise), you need to get around the following:

– Determining which interfaces are trunks on the switches so you can strip those MAC entries out (CDP works quite well)
– Converting ARP and MAC info into a “clean” format (eg: CatOS and IOS output is a different format)
– Detecting the fields across various pieces of hardware as display output isn’t always consistent for the same commands
– Inconsistent logins/passwords
– Correlating the IP/MAC/Interface information together. This can be done with the UNIX join command and some awk/sed
– What you do with MACs that don’t resolve to an IP address (I include a flag to print these if required)
– Whether the machine you run DNS queries on will be able to resolve the IPs to PTR records
– If using expect, stripping out stray characters (eg \r) that will mess up your greps and other string searches
– Add plenty of debugging so you can quickly tell why something isn’t working properly

I used expect to go and grab the ARP, CDP and MAC information seeing as you can’t get all the required information from SNMP on many devices these days. In my case, this results in the following type of output:

Switch       Interface       VLAN  MAC             IP               DNSName
nycsw12      Fa3/10          100   0060.b0aa.0000  192.168.10.30    NO_DNS
nycsw12      Fa2/16          99    1060.4b61.0001  192.168.9.72     nyc-pc573.company.corp.
nycsw12      Fa2/37          101   1060.4b64.0002  192.168.11.78    nyc-pc555.company.corp.
nycsw12      Fa2/42          101   1060.4b68.0003  192.168.11.115   nyc-pc572.company.corp.
nycsw12      Fa2/45          98    1060.4b6a.0004  192.168.8.99     nyc-pc588.company.corp.
nycsw12      Fa2/32          98    1060.4b6a.0005  192.168.8.121    nyc-pc601.company.corp.
nycsw12      Fa3/3           100   2c41.389e.d19f  192.168.10.99    nyc-pc480.company.corp.
nycsw13      Fa2/4           100   5c26.0a01.0ac4  192.168.10.67    nyc-pc246.company.corp.
nycsw13      Fa2/6           100   6c3b.e531.2ddf  192.168.10.85    nyc-pc745.company.corp.

Of course, you can always just use Excel to do a VLOOKUP of your mac-address table output against a sorted table containing all your arp entries, but that’s a bit less automatic.