BGP Peer Summary Reporting

After a previous BGP scheduled report script had become a bit unwieldy by way of the massive amounts of output, I decided to rewrite it to give a summary of each router and its peer statuses.

The script’s function is to poll each BGP router in a list, grab peer state and admin state for each peering, figure out how many peers are OK, abnormal or shutdown, then give a summary we can work with to generate a table on.

Note that for the MIB used, only IPv4 works properly. IPv6 support is on the “to do” list.

Variables that need setting are BASE_DIR, MAILRECIPIENT, COMMUNITY. Tweak others if desired.

Input file format is:
host1.mydomain.com,description text,{optional SNMP string}

For Juniper routers, we only see peerings for the main system. We can target logical systems with [logical-system-name]/[routing-instance]@community – for example: LSYSNAME/default@MYCOMMUNITY – this could be placed as a custom SNMP string in the input file.

Any lines starting with — will pull out the following text and add that as a divider bar.

I like to run this as a cron job twice a day, eg:

0 7,17 * * * /home/netscripts/peersumary.sh | /usr/sbin/sendmail -t

Basically the end result is a HTML email like this:

Of course, the email part could be stripped out and the HTML output could be dumped into a folder somewhere and served as a web page. I don’t see much merit in that given that your NMS should be doing your polling – this is just meant as a morning and afternoon nudge to deal with any outstanding issues.

Bear in mind each poll runs in sequence so longer lists may take a while to complete. I’ve not felt the need to parallel-ise this yet.

#!/bin/bash
#
# Script: peersummary.sh
# Function: Poll BGP peers and create HTML summary report
# Author: Mark M (sol@subnetzero.org)
# Verion: 1.2 18/05/2019
#
# Input file format: resolvable-hostname,text-description,optional-snmp-string
#
# lines beginning with --- will add dividers, eg: ---USA_Peers 
# Don't use spaces in divider titles - use underscore instead.
#
# Change BASE_DIR to the directory this script resides in. This is to
# avoid any issues when running from cron given that we have to
# generate some temporary files.
#

bgpquery() {

PEERSTATUSES=`/usr/bin/snmpwalk -v 2c -On -c "$2" $1 \
        1.3.6.1.2.1.15.3.1.2 2>/dev/null | \
        sed 's/^.1.3.6.1.2.1.15.3.1.2.//g' | \
        awk '{print $1","$4}' | grep "[0-9]\.[0-9]"`

ADMINSTATUSES=`/usr/bin/snmpwalk -v 2c -On -c "$2" $1 \
        1.3.6.1.2.1.15.3.1.3 2>/dev/null | \
        sed 's/^.1.3.6.1.2.1.15.3.1.3.//g' | \
        awk '{print $1","$4}' | grep "[0-9]\.[0-9]"`

echo "$PEERSTATUSES" > $BASE_DIR/bgp_peerstatuses.tmp
echo "$ADMINSTATUSES" > $BASE_DIR/bgp_adminstatuses.tmp

# Join the two together on Peer IP address to
# ensure consistency.
OUTPUT=`join -t, -j1 1 -j2 1 -o 1.1 1.2 2.2 $BASE_DIR/bgp_peerstatuses.tmp $BASE_DIR/bgp_adminstatuses.tmp`

# Figure out how many normal, abnormal and shutdown peerings there are.
RESULT=`echo "$OUTPUT" | \
 awk -F"," '
  BEGIN{normal=0;abnormal=0;shutdown=0}
  $2 == "6" && $3 == "2" {normal+=1;next}
  $2 == "1" && $3 == "1" {shutdown+=1;next}
  $2 >= "1" && $2 <= "5" && $3 == "2" {abnormal+=1;badpeers=$badpeers","$1;next}
  END {total=normal+abnormal+shutdown;
       print normal","abnormal","shutdown","total;
  }'`

if [ "$RESULT" ]; then
   PEEROK=`echo "$RESULT" | awk -F"," '{print $1}'`
   PEERBAD=`echo "$RESULT" | awk -F"," '{print $2}'`
   PEERSHUT=`echo "$RESULT" | awk -F"," '{print $3}'`
   PEERTOTAL=`echo "$RESULT" | awk -F"," '{print $4}'`

   # Set blocks of HTML to different colours based on each value
   # Do this in 4 separate if statements to avoid having to cover
   # Every possible eventuality and to give flexibility down the line.
   # Colour definitions are in main block.

   # HTML for Normal Peers
   if [ "$PEEROK" -eq "0" ]; then
      PEEROK_HTML="<td bgcolor=$COLRD>$PEEROK</td>"
   else
      PEEROK_HTML="<td bgcolor=$COLGN>$PEEROK</td>"
   fi

   # HTML for Abnormal Peers
   if [ "$PEERBAD" -ge "1" ]; then
      PEERBAD_HTML="<td bgcolor=$COLRD>$PEERBAD</td>"
   else
      PEERBAD_HTML="<td bgcolor=$COLGR>$PEERBAD</td>"
   fi

   # HTML for Shut Peers
   if [ "$PEERSHUT" -ge "1" ]; then
      PEERSHUT_HTML="<td bgcolor=$COLYL>$PEERSHUT</td>"
   else
      PEERSHUT_HTML="<td bgcolor=$COLGR>$PEERSHUT</td>"
   fi

   # HTML for Total Peers
   if [ "$PEERTOTAL" -eq "0" ]; then
      PEERTOTAL_HTML="<td bgcolor=$COLRD>$PEERTOTAL</td>"
   else
      PEERTOTAL_HTML="<td bgcolor=$COLGR>$PEERTOTAL</td>"
   fi

   printf "$PEEROK_HTML $PEERBAD_HTML $PEERSHUT_HTML $PEERTOTAL_HTML"

# Otherwise, if there was no result returned,
# print an error snippet. Shouldn't happen.
#
else
   printf "<td>ERR</td><td>ERR</td><td>ERR</td><td>ERR</td>"
fi

}

######## START SCRIPT ########
#
# Main Vars
BASE_DIR=/home/netscripts/
LISTFILE=bgplist.csv
MAILSENDER="networkops@$HOSTNAME"
MAILRECIPIENT=me@mycompany.com
MAILSUBJECT="BGP Peering Status"
COMMUNITY=SNMPCOMMUNITY
#
# Colour Definitions.
# Let's try a nice pastel scheme. :)
COLRD="#FFBDBD"      # Red
COLYL="#FFF485"      # Yellow
COLGN="#E1F7D5"      # Green
COLBU="#D7E7F8"      # Blue (Table headers/dividers)
COLGR="#DDDDDD"      # Grey

# See if an argument was added to use a different file
if [ "$1" ]; then
 LISTFILE=$1
fi
# Check bgplist file is readable
#
if [ ! -r $BASE_DIR/$LISTFILE ]; then
   echo "$LISTFILE not found or readable"
   exit 1
fi

# Echo out mail header along with beginning of HTML
# document  so that sendmail can be used to send
# the output via email. We use sendmail instead of
# mailx as mailx is a major pain to send HTML email
# with. Style sheet in here for tables.
#
echo "From: $MAILSENDER
To: $MAILRECIPIENT
MIME-Version: 1.0
Subject: $MAILSUBJECT
Content-Type: text/html
<!DOCTYPE html>
<html>
<body>
<style media="all" type="text/css">
table, th,td {
     font-family: "arial";
     font-style: normal;
     font-size: 12px;
     border: 1px solid;
     border-color: rgb(180,180,180);
     border-collapse: collapse;
     border-spacing: 0;
     padding: 3px;
}
h1 {
     font-family:arial;
     font-size: 18px;
     font-weight:bold;
}
</style>

<h1>$MAILSUBJECT</h1>
<table>
<tr bgcolor=$COLBU><th align=left>HOST</th><th>Host Description</th><th>Normal</th><th>Abnormal</th><th>Shutdown</th><th>Total</th></tr>
"
# Loop through each line in the listfile, ignoring blank
# lines or lines that are hashed out. In the case of a line
# starting with --- then print a divider table row to separate
# sections. Any text after --- gets printed in 2nd column minus spaces.
egrep -v "^$|^#" $BASE_DIR/$LISTFILE |\
while read LINE;
do
   HOST=`echo "$LINE" | awk -F"," '{print $1}' | tr -d ' '`
   if [[ "$HOST" == "---"* ]]; then
      # Clip out the --- and special characters so we can put in a section header if it exists
      # Special chars can break printf so we get rid of them and put the tag in the second column
      SECTIONTAG=`echo "$HOST" | sed 's/^---//g' | tr -dc '[:alnum:]\_\n\r'`
      printf "<tr bgcolor=$COLBU><td></td><td><b>"$SECTIONTAG"</b></td><td></td><td></td><td></td><td></td></tr>\n"
   else
      DESCR=`echo "$LINE" | awk -F"," '{print $2}'`
      CUSTOMSTR=`echo "$LINE" | awk -F"," '{print $3}'`
      if [ "$CUSTOMSTR" ]; then
         BGPQUERY=`bgpquery $HOST $CUSTOMSTR`
         unset CUSTOMSTR
      else
         BGPQUERY=`bgpquery $HOST $COMMUNITY`
      fi
      echo "$BGPQUERY" |\
      while read INPUT; do
         printf "<tr><td>$HOST</td><td>$DESCR</td>$INPUT</tr>\n"
      done
   fi
done

# Close off the table, body and HTML to complete the
# HTML document
printf "<tr bgcolor=$COLBU><td></td><td>file: $LISTFILE</td><td></td><td></td><td></td><td></td></tr>\n"
printf "</table>\n</body>\n</html>\n"

# Now this script should be piped through to /usr/sbin/sendmail -t
# or whatever other mechanism you want to use to send email with.
# No need for recipient address as it's added in the
# header already.

HIBP Password Breach Bash Script

Another challenge – leverage the basic haveibeenpwned.com web API to see if passwords have been leaked in breaches.

Pretty simple criteria here so it’s not much of a problem. Openssl does what we want and bash script is posted below.

When I was writing this, it didn’t work at first because I was being an idiot and not accounting for the newline “\n” which was completely changing the submitted hash. Just a pointer for anyone else that is tempted to use echo in this sort of application. :)

#!/bin/bash
#
# HaveIBeenPwned Password/Hash Checker
# v1.1 - Mark M
# 
# Check password or hash against hibp.
#
# -p	Prompt for password to check
# -h	Use hash [hash] to check
#
# Simple func to leverage API
gethash(){
curl -A "hibp_checkverv1" -X GET https://api.pwnedpasswords.com/range/$TRUNC 2>/dev/null |\
 awk -F":" '{print "'$TRUNC'"$1" Count:"$2}'
}

USAGE="
`basename $0` [-f filename] [-h sha1-hash] [-p]
-h [sha1hash] checks given hash against HIBP
-p option will prompt for password"

optstring=h:p
while getopts $optstring opt
do
   case $opt in
      h)   MYHASH=$OPTARG;;
      p)   printf "Enter Password: "
           stty -echo
           read MYPASS
           stty echo ;;
      *)   echo "$USAGE.";exit 1;;
   esac
done

if [ "$MYPASS" ] && [ "$MYHASH" ]; then
   echo "File and Hash set. Only use one or the other."
   exit 1
elif [ ! "$MYPASS" ] && [ ! "$MYHASH" ]; then
   echo "No parameters."
   exit 1
fi


# Set required Vars
MYPASS=$(printf $MYPASS | tr -d '\n')

# Only hash with sha1 if password option was specified
if [ "$MYPASS" ]; then
   MYHASH=$(printf $MYPASS | openssl sha1 | awk '{print toupper($2)}')
fi

# Get first 5 chars into $TRUNC 
TRUNC=$(printf $MYHASH | cut -c 1-5)

# MYPASS no longer needed. Unset it.
unset MYPASS

# Run func and checks
printf "\nCheck HIBP for $TRUNC... Full hash is \033[33m$MYHASH\n\033[0m"
HASHLIST=$(gethash)
if [ "$HASHLIST" == "" ]; then
   printf "\nError retreiving Data from Web API\n"
   exit 1
fi

printf "Does it appear in list? "
HASHCHK=$(echo "$HASHLIST" | grep -o "$MYHASH")

if [ "$HASHCHK" == "$MYHASH" ]; then
   printf "\033[31;1m< YES >\n\033[0m"
else
   printf "\033[32;1m< NO >\n\033[0m"
fi

Bash multi-threading – parallel SNMP polls

Bit of a misleading title, that. It’s really not possible and there are issues trying to set vars from the output of background child processes. However, it seems it’s possible to fake it if you’re willing to fudge it a bit with temporary files.

I got annoyed with SNMP polls across a large number of targets being very slow so decided to write something to get around it. Actually, one of the biggest issues is the default retry value of snmp commands which is set at 5. See the man page for snmpcmd which shows this.

Here is a script for grabbing the first line of SNMP get output from the specified OID from a large number of devices. There is no maximum limit here unlike my previous batch script, so if it’s a huge list, run at your own risk. It’s good for checking for things like devices still set to public read string. Beware that it uses temporary files given bash limitations, so bear in mind your user file limits.

snmphosts.txt should contain an IP address or resolvable hostname on each line.

#!/bin/bash
#
# Parallel SNMP Query for BASH - Who needs multithreading? ;)
#
# Version: V1.0 - Mark M (sol@subnetzero.org)
# Date:    15/05/2019
#
# The intention of this script is to get around how slow it is to poll
# a large number of SNMP hosts sequentially. This is achieved by a loop
# which sends each poll to the background which writes its output to
# a unique file suffixed by .$i in folder $OUTDIR. It is not possible
# to populate variables with the results of background child processes 
# in BASH so this is one workaround.
#
# Once complete, awk is used to pick out the fields of the output files
# to avoid issues with blank responses/lack of newlines. Stderror is redirected 
# to the files so that we can see when a poll failed. We only pick out
# the first line of the result with head -1 in the poll to avoid
# loads of extra lines per host with say, sysDescr for example.
#
# Keeping retries low will speed this up even more.
#
# Caveats: Extremely large lists will generate enough files to hit quotas
# or user max file limits.
#
SNMPVER="2c"
SNMPRETRIES=1
SNMPCOMMUNITY=public
SNMPLIST=snmphosts.txt
OUTDIR=tmpoutdir

SNMPOID=".1.3.6.1.2.1.1.1.0"   # system.sysDescr.0
# Some additional useful SNMP OIDs below that should usually respond.
#SNMPOID=".1.3.6.1.2.1.1.3.0"   # system.sysUpTime.0
#SNMPOID=".1.3.6.1.2.1.1.5.0"	# system.sysName.0

# Create temporary output dir if required
if [ ! -d $OUTDIR ]; then
   mkdir $OUTDIR
   if [ $? -ne 0 ]; then
      echo "Problem creating temp dir. Quitting."
      exit 1
   fi
fi

# Delete any old temp files
rm -f $OUTDIR/snmpitem* 
if [ $? -ne 0 ]; then
   echo "Error deleting old temp files in $OUTDIR. Exiting."
fi

# Init i for loop
i=0

# Loop through each host, sending query to background.
# Assigning each host to an array element for future use.
# Ignore blank lines and commented lines in $SNMPLIST file.
for host in $(cat $SNMPLIST | egrep -iv "^$|^#");
do
   printf "Polling Item $i - $host\n"
   HOSTS[$i]=$host
   printf "$host:" > $OUTDIR/snmpitem.$i

   # This bit is tricky. We have to redirect stderr to stdout in both instances
   # here to ensure we see if we get no response or some other error.
   snmpget -Ov -v$SNMPVER -r $SNMPRETRIES -c $SNMPCOMMUNITY $host $SNMPOID 2>&1 | head -1 >> $OUTDIR/snmpitem.$i 2>&1 &
   i=$(( $i + 1 ))
done
printf "Queries launched. Waiting..."
# Use BASH builtin to wait for child processes to exit.
wait
printf "Done!\n"

# Count total number in array
SNMPCOUNT=$(echo ${#HOSTS[*]})
echo "Host Count: $SNMPCOUNT ($i)"

# Use Awk to pick out fields of all files which will avoid
# formatting errors for failures. This will be in same
# order as an ls statement
awk -F":" '{print $1":"$3}' $OUTDIR/snmpitem.*

# Delete temp files
rm -f $OUTDIR/snmpitem* 

Running tasks in parallel batches in Bash

I had a requirement to run quite a lot of tasks in parallel with varying parameters. Initial experimentation suggested I might end up with a lot of processes running and potentially cause system issues so I looked into creating a script to run things in parallel, albeit in controlled batches.

In this example, I’ve substituted the actual actions I was taking with a random sleep command so that processes will finish at different times. What would probably be best would be to have the actions in another script that will log its output somewhere either by writing to a file or by using logger so syslog deals with the flurry. Typically unix file writes are atomic up to 4KB so having several processes writing at the same time isn’t a huge issue.

Bash below:

#!/bin/bash
#
# Loop through items in word list to run actions
# and process in parallel batches to avoid having
# too many processes.
#
# sol@subnetzero.org v1.0 7/5/2019
#
USAGE="`basename $0` /path/to/wordlist {batch size}"
WORDLIST=$1
BATCHSIZE=$2
if [ ! $BATCHSIZE ]; then
    BATCHSIZE=10
fi

if [ ! $1 ] || [ ! $2 ]; then
    echo "$USAGE"
    exit 1
fi

echo "Using wordlist $WORDLIST in batches of $BATCHSIZE"
i=0
for word in `cat $WORDLIST`; do
    if [ $(( $i % $BATCHSIZE )) -eq 0 ] && [ $i -ne 0 ]; then
       echo "Batch of $BATCHSIZE done... waiting"
       wait
    fi

    # Take actions here and run as background processes
    SLEEPRND=`echo $(( $RANDOM % 9 + 1 ))`
    echo "Action: $word - Sleeping for $SLEEPRND"
    sleep $SLEEPRND &

    # Increment counter for tracking
    i=$(( $i + 1 ))
done
printf "Waiting..."
wait
printf "all jobs run.\n"

Output:

[me@server ~]$ ./parallel.sh wordlist localhost 5
Using wordlist wordlist against host localhost in batches of 5
Action: a - Sleeping for 2
Action: b - Sleeping for 3
Action: c - Sleeping for 2
Action: d - Sleeping for 2
Action: e - Sleeping for 5
Batch of 5 done... waiting
Action: f - Sleeping for 7
Action: g - Sleeping for 1
Action: h - Sleeping for 1
Action: i - Sleeping for 1
Action: j - Sleeping for 8
Batch of 5 done... waiting
Action: k - Sleeping for 2
Action: l - Sleeping for 1
Waiting...All jobs run.

Quick and Dirty Cisco Hardware Inventory to CSV

I wanted to quickly grab all part numbers and serial numbers of chassis, cards and modules in some Cisco kit but the standard output wasn’t really nice enough to work with very easily.

To this end I knocked up the following script that will give a nice fixed field or comma-separated output that can be dragged into an Excel spreadsheet. Useful for doing support renewals and the like.

This is the first version so it’s a bit rough with error checking and doesn’t like incorrect credentials so type carefully. Tested with a variety of Cisco switches and routers, including Nexus 5K. Requires expect/tcl/tk

If you have ancient hardware that only supports prehistoric ssh key exchange algorithms, this may fail under expect. Good luck fixing it :)

$ ./hwaudit -d switch1
Tacacs user:me
Tacacs pass:
Using command line specified hosts: switch1
Device        Name                  Desc                  PID                 VID      Serial
switch1     "1"                     "WS-C3560-24TS"       WS-C3560-24TS-E     V02      CAT0XXXXXXX
switch1     "GigabitEthernet0/1"    "1000BaseSX SFP"      Unspecified                  AGMXXXXXX3P

$ ./hwaudit -cd switch2
CSV Mode
Tacacs user:me
Tacacs pass:
Using command line specified hosts: switch1
Device,Name,Desc,PID,VID,Serial
switch2,"1","ME-C3750-24TE",ME-C3750-24TE-M,V05,CATXXXXXXXX
switch2,"GigabitEthernet1/0/1","1000BaseSX SFP",Unspecified,,AGMXXXXXXX
switch2,"GigabitEthernet1/0/2","1000BaseSX SFP",Unspecified,,FNSXXXXXXX

Output is in combined_audit.txt

BASH Script:

#!/bin/bash
# hwaudit v1.1
# Mark M (sol@subnetzero.org)
#
# Script to go and get part numbers/serial numbers
# from Cisco devices.
#
# To do: write additional methods for telnet_acs and telnet_pwonly
#
# Examples:
# ./hwaudit -cd device1    <- CSV delimited output for device1
# ./hwaudit -f list.txt    <- Fixed field output for all in list.txt
#
# Expect functions to log on to Cisco Devices
ssh_acs() {
/usr/bin/expect << EOF
set timeout $PROMPT_TIMEOUT
spawn ssh $USNAME@$DEVICE
expect {
        "continue connecting (yes/no)?" {send "yes\r" ; exp_continue}
        "assword:" { send "$PASS\r" }
}
expect -re "(>|#)" {send "term len 0\r"}
expect -re "(>|#)" {send "show inventory\r"}
expect -re "(>$|#$)" {send "exit\r\n"}
close $spawn_id
EOF
}

PROMPT_TIMEOUT=5
USNAME=""
PASS=""
DEVICE=""
METHOD=ssh_acs
USAGE="$0 { -f filename } { -c = csv format } { -d devicename }"
FMT="%-13.13s %-32.32s %-38.38s %-22.22s %-8.8s %-12.12s\n"
CSV=0
cat /dev/null > combined_audit.txt

case $1 in
 [a-zA-Z0-9]*) echo "$USAGE\n"
               exit 1;;
esac

optstring=cd:f: #tT
while getopts $optstring opt
do
   case $opt in
      c) echo "CSV Mode";FMT="%s,%s,%s,%s,%s,%s\n";CSV=1;;
      d) DEVICES=$OPTARG;;
      f) INPUTFILE=$OPTARG;;
      # t) METHOD=telnet_acs;;
      # T) METHOD=telnet_pwonly;;
      *) echo "$USAGE"
   esac
done

case $METHOD in
   ssh_acs)   printf "Tacacs user:" ; read USNAME; stty -echo
              printf "Tacacs pass:" ; read PASS ; printf "\n"; stty echo
              ;;
   *)        printf "No connection method set\n"
             exit 1
             ;;
esac

if [ ! -r $INPUTFILE ]; then
   echo "Input file: $INPUTFILE not readable"
   exit 1
fi

if [ -z "$DEVICES" ]; then
   DEVICES=$( grep -iv "^$|^#" "$INPUTFILE" )
   if [ -z "$DEVICES" ]; then
      echo "No list generated. $INPUTFILE"
      exit 1
   fi
else echo "Using command line specified hosts: $DEVICES"

fi

if [ $(tput cols) -lt 128 ] && [ "$CSV" -eq "0" ]; then
   printf "\033[31;1mWARNING - \033[37mTerminal may not be wide enough for clean output in non-csv mode.\033[0m\n"
   sleep 2
fi

for DEVICE in ${DEVICES[@]}; do
   $METHOD | tr -d '\r' > output.$DEVICE.tmp
   sed 's/  \+,/,/g' output.$DEVICE.tmp | \
   awk -F "," 'BEGIN {start=0;printf("'"$FMT"'","Device","Name","Desc","PID","VID","Serial")}
               /show inventory/ { start=1 }
               start==0 { next }
               start==1 && $1 ~ /^NAME:/ {name=substr($1,7)}
               start==1 && $2 ~ /DESCR:/ {desc=substr($2,9)}
               start==1 && $1 ~ /^PID:/  {pid=substr($1,6)}
               start==1 && $2 ~ /VID:/   {vid=substr($2,7)}
               start==1 && $3 ~ /SN:/    {ser=substr($3,6)}
               $1 ~ /^$/ {printf ("'"$FMT"'","'$DEVICE'",name,desc,pid,vid,ser)
               name=NULL;desc=NULL;pid=NULL;vid=NULL;ser=NULL} ' |\
               egrep -v ",,,,," | tee -a combined_audit.txt
done

echo "Output is in combined_audit.txt"
rm output.*.tmp