Nexus FEX Bouncing

I came across an odd problem where a FEX was bouncing and was asked to look at it. The logs were a flood of interfaces going up and down and FEX status messages, however buried in amongst the logs and quite easy to miss was the following, less frequent syslog message:

%SATCTRL-FEX132-2-SATCTRL_FEX_MISCONFIG: FEX-132 is being configured as 131 on different switch

Pretty obvious clue there. Configuration was correct for the uplinks on both 5Ks:

interface Ethernet1/13
  switchport mode fex-fabric
  fex associate 131
  channel-group 131

interface Ethernet1/14
  switchport mode fex-fabric
  fex associate 132
  channel-group 132

Checking the serial numbers of the attached FEXes confirmed the problem:


First 5K

FEX: 131 Description: FEX213 - CAB 28   state: Offline
  FEX version: 7.1(3)N1(1) [Switch version: 7.1(3)N1(1)]
  FEX Interim version: 7.1(3)N1(1)
  Switch Interim version: 7.1(3)N1(1)
  Extender Serial: FOC00011122

FEX: 132 Description: FEX214 - CAB 28   state: Online
  FEX version: 7.1(3)N1(1) [Switch version: 7.1(3)N1(1)]
  FEX Interim version: 7.1(3)N1(1)
  Switch Interim version: 7.1(3)N1(1)
  Extender Serial: FOC12345678

Second 5K


FEX: 131 Description: FEX213 - CAB 28   state: Registered
  FEX version: 7.1(3)N1(1) [Switch version: 7.1(3)N1(1)]
  FEX Interim version: 7.1(3)N1(1)
  Switch Interim version: 7.1(3)N1(1)

FEX: 132 Description: FEX214 - CAB 28   state: Online
  FEX version: 7.1(3)N1(1) [Switch version: 7.1(3)N1(1)]
  FEX Interim version: 7.1(3)N1(1)
  Switch Interim version: 7.1(3)N1(1)
  Extender Serial: FOC00011122

As we can see above, the same FEX is associated with FEX131 on the first 5K and FEX132 on the second 5K. The solution was to verify which serial number was which FEX in the cabinets and to swap the cables for the two ports around on the incorrectly patched 5K. Looks like someone had been doing some patching and put things back in the wrong way around! O_o