Juniper MX480 PSU Gripes

This has happened three times in a row to me. It seems that Juniper MX480 PSUs are not very graceful when they blow. I now think it’s unwise to ask anyone to try “re-seating the PSU and power cable” once a PSU has failed because this has a tendency to make the PDU trip, knocking out everything on that power strip. Ouch! Better to just RMA the thing.

Not so much of a problem when all kit in the rack has dual PSUs, but it’s worth bearing in mind in case there is any single PSU kit that isn’t on a static transfer switch in there! Lucky the environment I deal with is doesn’t involve the latter. :)

Faulty PSU shows as:

PEM 0:
  State:     Present
  AC input:  Out of range (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 0 W (zone 0, 0 A at 0 V, 0% of capacity)

Effective Interface Monitoring with iSPI for Metrics

While iSPI for Metrics is useful for graphing, arguably the more critical use for it is in generating alerts when things are not working as expected.

I found today that Juniper does not include “FCS Lan Errors” in the ifInErrors counters which made finding an issue take a bit longer than usual. This is what I’m talking about:

Interface: ge-1/1/0, Enabled, Link is Up
Encapsulation: Ethernet, Speed: 1000mbps
Traffic statistics:                                           Current delta
  Input bytes:            42120054608900 (6704304 bps)           [16457880]
  Output bytes:           33903229061248 (11640616 bps)          [38429868]
  Input packets:             60739200100 (3676 pps)                 [82119]
  Output packets:            51621757514 (5316 pps)                [125843]
Error statistics:
  Input errors:                        0                                [0]
  Input drops:                         0                                [0]
  Input framing errors:          1435325                            [12231]
  Policed discards:                    0                                [0]
  L3 incompletes:                      0                                [0]

For effective monitoring of critical WAN interfaces, you can create an interface group using for example, a filter by interface name (or ifAlias – description if you have a naming standard), which then is set up as a monitoring policy with the thresholds you want. Alternatively you can use a Node group based policy bearing in mind you’ll be alerted for all interfaces on those nodes in the group.

Example:
thresh

You want to enable Interface Fault Polling and Interface Performance Polling to make this work, and may also want to consider lowering the default 5 minute poll period for critical infrastructure.

I tend to set the trigger count at 1, and the rearm count higher (at least 2) so that the incident doesn’t just disappear before someone’s had a chance to have a look at it when the error condition clears.

As a side note, you’ll also want to set generic thresholds… Eg: For “Routers” at least the following:

router-thresh

Cisco SVI equivalent config on Juniper MX Series

Scenario: How to configure a vlan with routed SVI on Juniper MX series with VRRP, trunked to a second MX and with a single (eg: server) access port in that vlan, with the network advertised by OSPF (set as passive). VSTP configuration is used here (equivalent to rapid-pvst in cisco land).

Set up the Vlan

bridge-domains {
   TestVlan {
       vlan-id 100;                                                       
        routing-interface irb.100;
    }
}

Configure the access, trunk to other switch and irb (SVI) interfaces. VRRP is configured as an equivalent to Cisco HSRP setup.

interfaces {
   ge-4/0/5 {
        description "Test Vlan 100 Server";
        unit 0 {
            family bridge {
                interface-mode access;
                vlan-id 100;
            }
        }
    }

    xe-5/2/0 {
        description "To MX2 Switch xe-5/2/0 Layer2 Link";
        mtu 2000;
        unit 0 {
            family bridge {
                interface-mode trunk;
                vlan-id-list [ 100 ];
            }
        }
    }

    irb {
        unit 100 {
        description "Test Vlan 100 SVI";
            family inet {
                mtu 1500;
                address 192.168.1.2/24 {
                    vrrp-group 100 {
                        virtual-address 192.168.1.1;
                        priority 120;
                        authentication-type simple;
                        authentication-key test100;
                        preempt {
                            hold-time 10;
                        }
                        accept-data;
                    }
                }
            }
        }
    }
}

NB: If you want to do interface tracking on the vrrp group, an example is given below. This is applied within the vrrp group stanza. Note that the priority-cost is the equivalent of the cisco decrement command, so 21 here would reduce 120 to 99, causing the secondary (cost 100) to become the master.

Tip: To load track config in easily..

me@mx1# edit
me@mx1# edit interfaces irb unit 100 family inet address 192.168.1.2/24 vrrp-group 100
me@mx1# load merge terminal relative

Interface tracking config:

track {
    interface xe-5/3/0 {
        priority-cost 21;
    }
    priority-hold-time 10;
}

NB: If you want to do route tracking on the vrrp group, an example is given below. This is applied within the vrrp group stanza. Note that the priority-cost is the equivalent of the cisco decrement command, so 21 here would reduce 120 to 99, causing the secondary (cost 100) to become the master.

Route tracking config:

track {
    route 10.0.0.0/24 routing-instance default priority-cost 21
}

Set as OSPF passive so we advertise but won’t form adjacencies over the irb interface.

protocols {
    ospf {
        area 0.0.0.0 {
            interface irb.100 {
                passive;
            }
        }
    }
}

Set up spanning tree – we are using VSTP (equivalent of Rapid-PVST – maximum of 256 vlans supported). We block any edge ports that we receive BPDUs on, and set this switch as the root bridge (priority 0). “edge” is the equivalent of “spanning-tree portfast” here. “no-root-port” does what it says on the tin.

protocols{
    vstp {
        bpdu-block-on-edge;
        vlan 100 {
            bridge-priority 0;
            interface ge-4/0/5 {
                edge;
                no-root-port;
            }
            interface xe-5/2/0;
        }
    }
}

Equivalent configuration needs to be done on the second MX with the following changes:

irb interface: address 192.168.1.3/24
irb interface vrrp: priority lower than 120 (eg: 100)
vstp vlan config: bridge-priority 4096

Gotcha: Bear in mind that if you are trunking to Cisco kit, then the default MTU on the Juniper won’t allow things to work. Any trunks to Cisco kit can be configured as follows or you can enable flexible-vlan-tagging:

Example 1:

ge-4/1/9 {
    description " To Cisco SW01 Gi0/1 Trunk ";
    mtu 1522;
    unit 0 {
        family bridge {
            interface-mode trunk;
            vlan-id-list [ 100 101 102 103 ];
        }
    }
}

Example 2:

ge-4/1/9 {
    description " To Cisco SW01 Gi0/1 Trunk ";
    flexible-vlan-tagging;
    unit 0 {
        family bridge {
            interface-mode trunk;
            vlan-id-list [ 100 101 102 103 ];
        }
    }
}

You can see here that flexible-vlan-tagging changes the MTU for you:

Physical interface: ge-4/1/9, Enabled, Physical link is Up
  Interface index: 187, SNMP ifIndex: 545
  Description:  To Cisco SW01 Gi0/1 Trunk 
  Link-level type: Ethernet, MTU: 1522, Speed: 1000mbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled,

Gotcha: The MTU on the trunk between switches also needs to be raised to allow for Vlan tagging if it isn’t already. It’s quite likely a large MTU may be desirable between the two anyway (support for Jumbo frames etc).

Useful commands for monitoring/troubleshooting

show interfaces <interface name>
show bridge-domain
show bridge mac-table
show spanning-tree bridge
show spanning-tree interface
show spanning-tree statistics interface
show vrrp brief
show ospf interface
show ospf neighbor

Re-enable a BPDU blocked interface:

clear error bpdu interface <interface name>

Cisco GLC-T Copper SFP Compatibility

Just a “gotcha”…. Before deciding to re-use GLC-T SFPs elsewhere in the network, be aware that the documentation states the following:

Speeds Supported by the 1000BASE-T(GLC-T) SFP Module on the Cisco Catalyst 3750 Series Switch
The 1000BASE-T SFP can support 10/100/1000 speeds only on the Cisco Catalyst 2970, 3560, and 3750 Series Switches.

Putting some these in other vendor’s (Juniper MX) kit and trying to use them caused a fair bit of confusion as we had all manner of problems getting them to work at 100 Mbit mode.

When set to auto where the other end was auto and only 100Mbit capable, sometimes the link light came up making us think it’d work fine but that wasn’t the case. We ended up ordering Juniper SFPs in the end and all the headaches went away.

Juniper BGP – “unsupported optional parameter” message

This seems to happen mainly with legacy kit such as old Nortel routers and the like. The peering won’t come up at all despite everything looking OK config wise.

When doing a bit more digging, the following message can be seen in the Juniper logs:

bgp_recv_open: peer x.x.x.x (External AS 12345): received NOTIFICATION code 2
 (Open Message Error) subcode 4 (unsupported optional parameter)

Solution (Example)

set protocols bgp group VENDORA neighbor 192.168.250.2 disable-4byte-as

This sets the peering to 2 byte AS number support only and allows the peering to come up.