Spoofing SNMP Traps for testing

Somehow I missed the fact that JunOS allows you to spoof SNMP traps. I discovered this recently and must say it’s very handy, especially when testing new NNMi or other NMS incident configurations. It helpfully populates the varbinds for you with preset values (although you can specify them if desired).

This is done as follows from the JunOS command line:

user@SRX> request snmp spoof-trap ospfNbrStateChange variable-bindings "ospfNbrState = 8"

You can look up varbind names in the appropriate MIB.

A bit easier than the traditional equivalent in shell:

/usr/bin/snmptrap -v 2c -c mycommunity nms.mydomain.com:162 '' .1.3.6.1.2.1.14.16.2.0.2 \
.1.3.6.1.2.1.14.1.1 a 0.0.0.0 \
.1.3.6.1.2.1.14.10.1.1 a "1.2.3.4" \
.1.3.6.1.2.1.14.10.1.2 i "0" \
.1.3.6.1.2.1.14.10.1.3 a "2.3.4.5" \
.1.3.6.1.2.1.14.10.1.6 i "8"

HP NNMi9 False Redundancy Group alerts/Cisco Nexus Duplicate Discoveries

It seems that HP are getting around to fixing the issue of duplicate Cisco Nexus nodes being discovered (it’s not in patch 5) but until then, it’s possible to work around this. Duplicate discoveries play havoc with RRG alerting which isn’t funny when someone’s woken up in the middle of the night for it.

To stop NNMi discovering duplicate nodes once you have the devices in your topology, do the following:

1) Create an Auto-Discovery rule with the lowest ordering in the list (eg: No-AutoDiscover-Rule)
2) Uncheck all options on the left hand pane and uncheck Enable Ping Sweep
3) Add the IP addresses of all the mgmt0 interfaces in the management VRF (Type: Include in rule)

High CPU on Nexus 5K and no SNMP response – snmpd

Strange issue with SNMP not responding today on a nexus 5K. Tried removing and re-adding the SNMP config, removing the acl altogether that we use to control access and still no joy.

Upon checking CPU usage, it seemed quite high. show proc cpu sort output showed that snmpd was quite busy:

PID    Runtime(ms)  Invoked   uSecs  1Sec    Process
-----  -----------  --------  -----  ------  -----------
 4559           59  991518226      0   44.5%  snmpd
 4605          179        87   2065    9.0%  netstack
 1178         2091  1733135010      0    1.0%  kirqd
    1          157  25653477      0    0.0%  init
    2          837   3474116      0    0.0%  migration/0
    3          600  3970856252      0    0.0%  ksoftirqd/0

I was sure I’d dealt with this before and it seems that I was hitting a bug.

Official word is that: There is a memory leak in one of the processes called libcmd that is used by SNMP. Workaround is entering the hidden command:

no snmp-server load-mib dot1dbridgesnmp

The best solution, however, would be to perform a software upgrade to 5.0(3)N2(2) or later where this is fixed.