Uncategorized - SubnetZero.org

Requirement: Run extended pings from a number of cisco devices to generate a simple report on current latency and loss without using IP SLA commands. Allow the specification of source addresses where needed. SNMP write access is not available.

Solution: Expect and a shell script.

Funnily enough I found that this also ended up being quite useful for testing ISDN lines. You see some packet loss while the call is being set up but the SUCCESS message is what’s important there.

Configuration file: pingdevices.csv

#Router,Target IP,Description,Source IP (if desired)
10.0.0.254,10.0.0.101,Lab Gateway to Lab Server
10.0.0.253,192.168.1.64,LAB Switch to User PC,10.0.0.253
10.0.0.254,10.0.0.252,Lab 8kbps link router

Output:

[networks@nettools ~]$ ./pingcheck.sh
Timeout for Expect ping test is 30 seconds
Data will be read from pingdevices.csv in current directory (/home/networks) for tests.

Tacacs user:cisco
Tacacs pass:
Working on 10.0.0.254, Target IP 10.0.0.101.

[truncated expect output here to assist in debugging issues]

Time     Device          Target IP       Status   PKT% Min/Avg/Max  Description
--------+---------------+---------------+--------+----+------------|-----------------------------
15:33:19|10.0.0.254     |10.0.0.101     |SUCCESS |100 |1/9/16      |Lab Gateway to Lab Server
15:33:19|10.0.0.253     |192.168.1.64   |SUCCESS |100 |20/28/40    |LAB Switch to User PC
15:33:21|10.0.0.254     |10.0.0.252     |SUCCESS |90  |4/9/20      |Lab Gateway to R7 router
--------+---------------+---------------+--------+----+------------+-----------------------------

Script: pingcheck.sh

#!/bin/bash
#
# Extended ping tester.
#
# v3.1 sol@subnetzero.org
#
# This script allows you to run extended pings across a large
# amount of devices, resulting in a report table once complete.
#
# Create a pingdevices.csv config file to use with this script
# in the following format:
#
#   #Device,Ping-IP,Description#Source_Address(Optional)
#   testrtr1,172.17.1.1,TEST-1,192.168.10.221
#   testrtr2,10.10.10.1,TEST-2
#
# DO NOT USE COMMAS in the description field, or this will break
# any source address configuration in the final field.

pingexp_ssh(){
/usr/bin/expect << EOF
spawn ssh $USNAME@$DEVICE
#login handles cases:
#   user/pass
#   login with keys (first time verification)
expect {
        "continue connecting (yes/no)?" {send "yes\r" ; exp_continue}
        "assword:" { send "$PASS\r" }
}
expect -re "(>|#)" {send "term len 0\r"}
expect -re "(>|#)" {send "enable\r"}
expect "assword:" { send "$PASS\r" }
expect -re "(>|#)" {send "ping\r"}
expect {
        "otocol"      {send "ip\r"}
        timeout       {puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\tNA\tERROR 01 - CMDERROR"
                       exit}
}
expect {
        "address:"   {send "$DESTIP\r"}
         timeout      {puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\tNA\tERROR 02 - CMDERROR"
                      exit}
}

expect "count"     {send "$PKTCNT\r"}
expect "size"      {send "$PKTLEN\r"}
expect "seconds"   {send "3\r"}
expect "commands"  {send "n\r"}
expect "range"     {send "n\r"}

# Check for success marks
set timeout 30
expect {
        -re "Success rate is.*\n" {puts "latency"

             puts \$expect_out(0,string)
             puts [lindex \$expect_out(0,string) 3]
             puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tSUCCESS\t[lindex \$expect_out(0,string) 3]\t[lindex \$expect_out(0,string) 9]\t$DESCRP"
             puts "End of Output"
        }
        timeout {puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\t$DESCRP"}
        }

set timeout 7
# Logout
expect {
        "#"     { send "exit\r" }
        timeout { exit }
        }
EOF
}

pingexp_ssh_srcip(){
/usr/bin/expect << EOF
spawn ssh $USNAME@$DEVICE
#login handles cases:
#   user/pass
#   login with keys (first time verification)
expect {
        "continue connecting (yes/no)?" {send "yes\r" ; exp_continue}
        "assword:" { send "$PASS\r" }
}
expect -re "(>|#)" {send "term len 0\r"}
expect -re "(>|#)" {send "enable\r"}
expect "assword:" { send "$PASS\r" }
expect -re "(>|#)" {send "ping\r"}
expect {
        "otocol"      {send "ip\r"}
        timeout       {puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\tNA\tERROR 03 - CMDERROR"
                       exit}
}
expect {
        "address:"   {send "$DESTIP\r"}
         timeout      {puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\tNA\tERROR 04 - CMDERROR"
                      exit}
}

expect "count"               {send "$PKTCNT\r"}
expect "size"                {send "$PKTLEN\r"}
expect "seconds"             {send "3\r"}
expect "commands"            {send "y\r"}
expect "ource address"       {send "$SRCADD\r"}
expect "ype of service"      {send "\r"}
expect "DF bit in IP header" {send "\r"}
expect "alidate reply data"  {send "\r"}
expect "ata pattern"         {send "\r"}
expect "erbose"              {send "\r"}
expect "range"               {send "n\r"}

# Check for success marks
set timeout 30
expect {
        -re "Success rate is.*\n" {puts "latency"

             puts \$expect_out(0,string)
             puts [lindex \$expect_out(0,string) 3]
             puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tSUCCESS\t[lindex \$expect_out(0,string) 3]\t[lindex \$expect_out(0,string) 9]\t$DESCRP"
             puts "End of Output"
        }
        timeout {puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\t$DESCRP"}
        }

set timeout 7
# Logout
expect {
        "#"     { send "exit\r" }
        timeout { exit }
        }
EOF
}


########### SET VARS ##########
#
# Number of and size of packets
#
PKTCNT=10
PKTLEN=100
#
# Set inut file to be used
if [ "$1" ]; then
   if [ -r "$1" ]; then
      DEVICEFILE=$1
   else
      echo "Input file $1 not found or is not readable."
      exit 1
   fi
else
   DEVICEFILE=pingdevices.csv
fi

########### Sanity ############
#
if [ ! -w . ]; then
   echo "Error: Can't write to current directory."
   exit 1
fi

########### Cleanup ###########
#
cat /dev/null > pingcheck.log


######### USER INPUT ##########
#
printf "Data will be read from $DEVICEFILE in current directory ($PWD) for tests.\n\n"
printf "Tacacs user:" ; read USNAME; stty -echo
printf "Tacacs pass:" ; read PASS ; printf "\n" ; stty echo

# Set IFS to newline so we can read whole lines as elements
#
IFS="
"

# Loop through each line and run the ping check
#
for DATALINE in `egrep -v "^$|^#" $DEVICEFILE`; do
   TSTAMP=`date +%H:%M:%S`
   DEVICE=`echo $DATALINE | awk -F"," '{print $1}' | egrep -v "^#|^$"`
   DESTIP=`echo $DATALINE | awk -F"," '{print $2}'`
   DESCRP=`echo $DATALINE | awk -F"," '{print $3}'`
   SRCADD=`echo $DATALINE | awk -F"," '{print $4}' |\
           egrep '^[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}$'`
   ERRMSG="\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL"
   printf "Working on $DEVICE, Target IP $DESTIP.\n"
   ping -c1 -w1 $DEVICE > /dev/null
   if [ "$?" -eq "0" ]; then
      if [ "$SRCADD" ]; then
         pingexp_ssh_srcip | tee -a pingcheck.log
      else
         pingexp_ssh | tee -a pingcheck.log
      fi
   else
      printf "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\tNA\t$DESCRP\n" >> pingcheck.log
   fi
done

echo "
Time     Device          Target IP       Status   PKT% Min/Avg/Max  Description
--------+---------------+---------------+--------+----+------------|-----------------------------" > pingresults.txt

grep "^RESULT" pingcheck.log |\
 cut -d":" -f2- | \
 awk '{tail = substr ($0,length($1 $2 $3 $4 $5 $6) +7)}
 {printf ("%8.8s|%-15.15s|%-15.15s|%-8.8s|%-4.4s|%-12.12s|%-30.30s\n",$1,$2,$3,$4,$5,$6,tail);
 }'>> pingresults.txt
echo "--------+---------------+---------------+--------+----+------------+-----------------------------" >> pingresults.txt
clear
cat pingresults.txt

Do you have massive syslog files that take an age to grep or run your awk scripts through?

Splunk will be your saviour. It’s easy to define field extractions and follow a transaction all the way through and make pretty reports. I had a large script that hammered CPU every morning to report on system outages and the environment, but with SPLUNK you can run huge queries in seconds. It also allows you to pair or correlate syslog events by common key. Once your network syslogs hit around 100MB per day or more, it doesn’t make much sense to use the usual UNIX tools anymore.

Unfortunately, the enterprise licences for Splunk are very expensive (it’d put me off buying if I were a small company to be honest), but it’s easy enough to build usable, functional dashboards with the free version. You can’t send emails or SNMP traps from the free one but if you want to, it’s easy enough to get around this by writing a syslog monitoring script to cover the extra functionality that’s missing in most cases.

I’ll update this post with examples later, but for anyone who is struggling with huge volumes of syslogs, I seriously recommend that you look into it. Imagine a combination of grep and SQL on steroids and being able to say “show me the interfaces that have gone down in the last 24 hour period and haven’t come back up yet” or “show me the time periods where I’m getting more than 100 hits a minute on this webpage”. You also don’t need to change all your syslog config if you don’t want to; you can run a splunk forwarder on your existing syslog servers.

URL: http://www.splunk.com

The community is pretty damned decent as well. Lots of keen people out there who are willing to help out.

Category Archives: Uncategorized

iptables error: iptables: Setting chains to policy ACCEPT: security raw nat[FAILED]filter

Cisco Extended Pinger and Reporter

Why you should try Splunk

Recent Posts

Recent Comments

Archives

Categories

April 2019
M	T	W	T	F	S	S
« Nov
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30