BGP Peer Summary Reporting

After a previous BGP scheduled report script had become a bit unwieldy by way of the massive amounts of output, I decided to rewrite it to give a summary of each router and its peer statuses.

The script’s function is to poll each BGP router in a list, grab peer state and admin state for each peering, figure out how many peers are OK, abnormal or shutdown, then give a summary we can work with to generate a table on.

Note that for the MIB used, only IPv4 works properly. IPv6 support is on the “to do” list.

Variables that need setting are BASE_DIR, MAILRECIPIENT, COMMUNITY. Tweak others if desired.

Input file format is:
host1.mydomain.com,description text,{optional SNMP string}

For Juniper routers, we only see peerings for the main system. We can target logical systems with [logical-system-name]/[routing-instance]@community – for example: LSYSNAME/default@MYCOMMUNITY – this could be placed as a custom SNMP string in the input file.

Any lines starting with — will pull out the following text and add that as a divider bar.

I like to run this as a cron job twice a day, eg:

0 7,17 * * * /home/netscripts/peersumary.sh | /usr/sbin/sendmail -t

Basically the end result is a HTML email like this:

Of course, the email part could be stripped out and the HTML output could be dumped into a folder somewhere and served as a web page. I don’t see much merit in that given that your NMS should be doing your polling – this is just meant as a morning and afternoon nudge to deal with any outstanding issues.

Bear in mind each poll runs in sequence so longer lists may take a while to complete. I’ve not felt the need to parallel-ise this yet.

#!/bin/bash
#
# Script: peersummary.sh
# Function: Poll BGP peers and create HTML summary report
# Author: Mark M (sol@subnetzero.org)
# Verion: 1.2 18/05/2019
#
# Input file format: resolvable-hostname,text-description,optional-snmp-string
#
# lines beginning with --- will add dividers, eg: ---USA_Peers 
# Don't use spaces in divider titles - use underscore instead.
#
# Change BASE_DIR to the directory this script resides in. This is to
# avoid any issues when running from cron given that we have to
# generate some temporary files.
#

bgpquery() {

PEERSTATUSES=`/usr/bin/snmpwalk -v 2c -On -c "$2" $1 \
        1.3.6.1.2.1.15.3.1.2 2>/dev/null | \
        sed 's/^.1.3.6.1.2.1.15.3.1.2.//g' | \
        awk '{print $1","$4}' | grep "[0-9]\.[0-9]"`

ADMINSTATUSES=`/usr/bin/snmpwalk -v 2c -On -c "$2" $1 \
        1.3.6.1.2.1.15.3.1.3 2>/dev/null | \
        sed 's/^.1.3.6.1.2.1.15.3.1.3.//g' | \
        awk '{print $1","$4}' | grep "[0-9]\.[0-9]"`

echo "$PEERSTATUSES" > $BASE_DIR/bgp_peerstatuses.tmp
echo "$ADMINSTATUSES" > $BASE_DIR/bgp_adminstatuses.tmp

# Join the two together on Peer IP address to
# ensure consistency.
OUTPUT=`join -t, -j1 1 -j2 1 -o 1.1 1.2 2.2 $BASE_DIR/bgp_peerstatuses.tmp $BASE_DIR/bgp_adminstatuses.tmp`

# Figure out how many normal, abnormal and shutdown peerings there are.
RESULT=`echo "$OUTPUT" | \
 awk -F"," '
  BEGIN{normal=0;abnormal=0;shutdown=0}
  $2 == "6" && $3 == "2" {normal+=1;next}
  $2 == "1" && $3 == "1" {shutdown+=1;next}
  $2 >= "1" && $2 <= "5" && $3 == "2" {abnormal+=1;badpeers=$badpeers","$1;next}
  END {total=normal+abnormal+shutdown;
       print normal","abnormal","shutdown","total;
  }'`

if [ "$RESULT" ]; then
   PEEROK=`echo "$RESULT" | awk -F"," '{print $1}'`
   PEERBAD=`echo "$RESULT" | awk -F"," '{print $2}'`
   PEERSHUT=`echo "$RESULT" | awk -F"," '{print $3}'`
   PEERTOTAL=`echo "$RESULT" | awk -F"," '{print $4}'`

   # Set blocks of HTML to different colours based on each value
   # Do this in 4 separate if statements to avoid having to cover
   # Every possible eventuality and to give flexibility down the line.
   # Colour definitions are in main block.

   # HTML for Normal Peers
   if [ "$PEEROK" -eq "0" ]; then
      PEEROK_HTML="<td bgcolor=$COLRD>$PEEROK</td>"
   else
      PEEROK_HTML="<td bgcolor=$COLGN>$PEEROK</td>"
   fi

   # HTML for Abnormal Peers
   if [ "$PEERBAD" -ge "1" ]; then
      PEERBAD_HTML="<td bgcolor=$COLRD>$PEERBAD</td>"
   else
      PEERBAD_HTML="<td bgcolor=$COLGR>$PEERBAD</td>"
   fi

   # HTML for Shut Peers
   if [ "$PEERSHUT" -ge "1" ]; then
      PEERSHUT_HTML="<td bgcolor=$COLYL>$PEERSHUT</td>"
   else
      PEERSHUT_HTML="<td bgcolor=$COLGR>$PEERSHUT</td>"
   fi

   # HTML for Total Peers
   if [ "$PEERTOTAL" -eq "0" ]; then
      PEERTOTAL_HTML="<td bgcolor=$COLRD>$PEERTOTAL</td>"
   else
      PEERTOTAL_HTML="<td bgcolor=$COLGR>$PEERTOTAL</td>"
   fi

   printf "$PEEROK_HTML $PEERBAD_HTML $PEERSHUT_HTML $PEERTOTAL_HTML"

# Otherwise, if there was no result returned,
# print an error snippet. Shouldn't happen.
#
else
   printf "<td>ERR</td><td>ERR</td><td>ERR</td><td>ERR</td>"
fi

}

######## START SCRIPT ########
#
# Main Vars
BASE_DIR=/home/netscripts/
LISTFILE=bgplist.csv
MAILSENDER="networkops@$HOSTNAME"
MAILRECIPIENT=me@mycompany.com
MAILSUBJECT="BGP Peering Status"
COMMUNITY=SNMPCOMMUNITY
#
# Colour Definitions.
# Let's try a nice pastel scheme. :)
COLRD="#FFBDBD"      # Red
COLYL="#FFF485"      # Yellow
COLGN="#E1F7D5"      # Green
COLBU="#D7E7F8"      # Blue (Table headers/dividers)
COLGR="#DDDDDD"      # Grey

# See if an argument was added to use a different file
if [ "$1" ]; then
 LISTFILE=$1
fi
# Check bgplist file is readable
#
if [ ! -r $BASE_DIR/$LISTFILE ]; then
   echo "$LISTFILE not found or readable"
   exit 1
fi

# Echo out mail header along with beginning of HTML
# document  so that sendmail can be used to send
# the output via email. We use sendmail instead of
# mailx as mailx is a major pain to send HTML email
# with. Style sheet in here for tables.
#
echo "From: $MAILSENDER
To: $MAILRECIPIENT
MIME-Version: 1.0
Subject: $MAILSUBJECT
Content-Type: text/html
<!DOCTYPE html>
<html>
<body>
<style media="all" type="text/css">
table, th,td {
     font-family: "arial";
     font-style: normal;
     font-size: 12px;
     border: 1px solid;
     border-color: rgb(180,180,180);
     border-collapse: collapse;
     border-spacing: 0;
     padding: 3px;
}
h1 {
     font-family:arial;
     font-size: 18px;
     font-weight:bold;
}
</style>

<h1>$MAILSUBJECT</h1>
<table>
<tr bgcolor=$COLBU><th align=left>HOST</th><th>Host Description</th><th>Normal</th><th>Abnormal</th><th>Shutdown</th><th>Total</th></tr>
"
# Loop through each line in the listfile, ignoring blank
# lines or lines that are hashed out. In the case of a line
# starting with --- then print a divider table row to separate
# sections. Any text after --- gets printed in 2nd column minus spaces.
egrep -v "^$|^#" $BASE_DIR/$LISTFILE |\
while read LINE;
do
   HOST=`echo "$LINE" | awk -F"," '{print $1}' | tr -d ' '`
   if [[ "$HOST" == "---"* ]]; then
      # Clip out the --- and special characters so we can put in a section header if it exists
      # Special chars can break printf so we get rid of them and put the tag in the second column
      SECTIONTAG=`echo "$HOST" | sed 's/^---//g' | tr -dc '[:alnum:]\_\n\r'`
      printf "<tr bgcolor=$COLBU><td></td><td><b>"$SECTIONTAG"</b></td><td></td><td></td><td></td><td></td></tr>\n"
   else
      DESCR=`echo "$LINE" | awk -F"," '{print $2}'`
      CUSTOMSTR=`echo "$LINE" | awk -F"," '{print $3}'`
      if [ "$CUSTOMSTR" ]; then
         BGPQUERY=`bgpquery $HOST $CUSTOMSTR`
         unset CUSTOMSTR
      else
         BGPQUERY=`bgpquery $HOST $COMMUNITY`
      fi
      echo "$BGPQUERY" |\
      while read INPUT; do
         printf "<tr><td>$HOST</td><td>$DESCR</td>$INPUT</tr>\n"
      done
   fi
done

# Close off the table, body and HTML to complete the
# HTML document
printf "<tr bgcolor=$COLBU><td></td><td>file: $LISTFILE</td><td></td><td></td><td></td><td></td></tr>\n"
printf "</table>\n</body>\n</html>\n"

# Now this script should be piped through to /usr/sbin/sendmail -t
# or whatever other mechanism you want to use to send email with.
# No need for recipient address as it's added in the
# header already.

Quick and Dirty Cisco Hardware Inventory to CSV

I wanted to quickly grab all part numbers and serial numbers of chassis, cards and modules in some Cisco kit but the standard output wasn’t really nice enough to work with very easily.

To this end I knocked up the following script that will give a nice fixed field or comma-separated output that can be dragged into an Excel spreadsheet. Useful for doing support renewals and the like.

This is the first version so it’s a bit rough with error checking and doesn’t like incorrect credentials so type carefully. Tested with a variety of Cisco switches and routers, including Nexus 5K. Requires expect/tcl/tk

If you have ancient hardware that only supports prehistoric ssh key exchange algorithms, this may fail under expect. Good luck fixing it :)

$ ./hwaudit -d switch1
Tacacs user:me
Tacacs pass:
Using command line specified hosts: switch1
Device        Name                  Desc                  PID                 VID      Serial
switch1     "1"                     "WS-C3560-24TS"       WS-C3560-24TS-E     V02      CAT0XXXXXXX
switch1     "GigabitEthernet0/1"    "1000BaseSX SFP"      Unspecified                  AGMXXXXXX3P

$ ./hwaudit -cd switch2
CSV Mode
Tacacs user:me
Tacacs pass:
Using command line specified hosts: switch1
Device,Name,Desc,PID,VID,Serial
switch2,"1","ME-C3750-24TE",ME-C3750-24TE-M,V05,CATXXXXXXXX
switch2,"GigabitEthernet1/0/1","1000BaseSX SFP",Unspecified,,AGMXXXXXXX
switch2,"GigabitEthernet1/0/2","1000BaseSX SFP",Unspecified,,FNSXXXXXXX

Output is in combined_audit.txt

BASH Script:

#!/bin/bash
# hwaudit v1.1
# Mark M (sol@subnetzero.org)
#
# Script to go and get part numbers/serial numbers
# from Cisco devices.
#
# To do: write additional methods for telnet_acs and telnet_pwonly
#
# Examples:
# ./hwaudit -cd device1    <- CSV delimited output for device1
# ./hwaudit -f list.txt    <- Fixed field output for all in list.txt
#
# Expect functions to log on to Cisco Devices
ssh_acs() {
/usr/bin/expect << EOF
set timeout $PROMPT_TIMEOUT
spawn ssh $USNAME@$DEVICE
expect {
        "continue connecting (yes/no)?" {send "yes\r" ; exp_continue}
        "assword:" { send "$PASS\r" }
}
expect -re "(>|#)" {send "term len 0\r"}
expect -re "(>|#)" {send "show inventory\r"}
expect -re "(>$|#$)" {send "exit\r\n"}
close $spawn_id
EOF
}

PROMPT_TIMEOUT=5
USNAME=""
PASS=""
DEVICE=""
METHOD=ssh_acs
USAGE="$0 { -f filename } { -c = csv format } { -d devicename }"
FMT="%-13.13s %-32.32s %-38.38s %-22.22s %-8.8s %-12.12s\n"
CSV=0
cat /dev/null > combined_audit.txt

case $1 in
 [a-zA-Z0-9]*) echo "$USAGE\n"
               exit 1;;
esac

optstring=cd:f: #tT
while getopts $optstring opt
do
   case $opt in
      c) echo "CSV Mode";FMT="%s,%s,%s,%s,%s,%s\n";CSV=1;;
      d) DEVICES=$OPTARG;;
      f) INPUTFILE=$OPTARG;;
      # t) METHOD=telnet_acs;;
      # T) METHOD=telnet_pwonly;;
      *) echo "$USAGE"
   esac
done

case $METHOD in
   ssh_acs)   printf "Tacacs user:" ; read USNAME; stty -echo
              printf "Tacacs pass:" ; read PASS ; printf "\n"; stty echo
              ;;
   *)        printf "No connection method set\n"
             exit 1
             ;;
esac

if [ ! -r $INPUTFILE ]; then
   echo "Input file: $INPUTFILE not readable"
   exit 1
fi

if [ -z "$DEVICES" ]; then
   DEVICES=$( grep -iv "^$|^#" "$INPUTFILE" )
   if [ -z "$DEVICES" ]; then
      echo "No list generated. $INPUTFILE"
      exit 1
   fi
else echo "Using command line specified hosts: $DEVICES"

fi

if [ $(tput cols) -lt 128 ] && [ "$CSV" -eq "0" ]; then
   printf "\033[31;1mWARNING - \033[37mTerminal may not be wide enough for clean output in non-csv mode.\033[0m\n"
   sleep 2
fi

for DEVICE in ${DEVICES[@]}; do
   $METHOD | tr -d '\r' > output.$DEVICE.tmp
   sed 's/  \+,/,/g' output.$DEVICE.tmp | \
   awk -F "," 'BEGIN {start=0;printf("'"$FMT"'","Device","Name","Desc","PID","VID","Serial")}
               /show inventory/ { start=1 }
               start==0 { next }
               start==1 && $1 ~ /^NAME:/ {name=substr($1,7)}
               start==1 && $2 ~ /DESCR:/ {desc=substr($2,9)}
               start==1 && $1 ~ /^PID:/  {pid=substr($1,6)}
               start==1 && $2 ~ /VID:/   {vid=substr($2,7)}
               start==1 && $3 ~ /SN:/    {ser=substr($3,6)}
               $1 ~ /^$/ {printf ("'"$FMT"'","'$DEVICE'",name,desc,pid,vid,ser)
               name=NULL;desc=NULL;pid=NULL;vid=NULL;ser=NULL} ' |\
               egrep -v ",,,,," | tee -a combined_audit.txt
done

echo "Output is in combined_audit.txt"
rm output.*.tmp

F5 External Monitor script not working

I was trying to create a custom external monitor on an F5 LTM today which depended on the output of a tmsh command. I ended up tearing my hair out for a while because the monitor script would run from the command line when supplied with the correct arguments, however, when trying to run it from the monitor on the LTM, it was failing.

As I was sanitizing the output from the tmsh command for my requirements, I never saw the problem.

The cause: REMOTEUSER variable is required to be set when using tmsh in a script used by a monitor.

Just set:

REMOTEUSER=root
export REMOTEUSER

In your script and things should work!

I also learned that as soon as anything is returned to STDOUT, the monitor is assumed to be successful (UP) and your script will be killed off – no further lines will be executed! This was also confusing as I wasn’t getting my log details that were supposed to write to a file at the end! :)

Cisco Extended Pinger and Reporter

Requirement: Run extended pings from a number of cisco devices to generate a simple report on current latency and loss without using IP SLA commands. Allow the specification of source addresses where needed. SNMP write access is not available.

Solution: Expect and a shell script.

Funnily enough I found that this also ended up being quite useful for testing ISDN lines. You see some packet loss while the call is being set up but the SUCCESS message is what’s important there.

Configuration file: pingdevices.csv

#Router,Target IP,Description,Source IP (if desired)
10.0.0.254,10.0.0.101,Lab Gateway to Lab Server
10.0.0.253,192.168.1.64,LAB Switch to User PC,10.0.0.253
10.0.0.254,10.0.0.252,Lab 8kbps link router

Output:

[networks@nettools ~]$ ./pingcheck.sh
Timeout for Expect ping test is 30 seconds
Data will be read from pingdevices.csv in current directory (/home/networks) for tests.

Tacacs user:cisco
Tacacs pass:
Working on 10.0.0.254, Target IP 10.0.0.101.

[truncated expect output here to assist in debugging issues]

Time     Device          Target IP       Status   PKT% Min/Avg/Max  Description
--------+---------------+---------------+--------+----+------------|-----------------------------
15:33:19|10.0.0.254     |10.0.0.101     |SUCCESS |100 |1/9/16      |Lab Gateway to Lab Server
15:33:19|10.0.0.253     |192.168.1.64   |SUCCESS |100 |20/28/40    |LAB Switch to User PC
15:33:21|10.0.0.254     |10.0.0.252     |SUCCESS |90  |4/9/20      |Lab Gateway to R7 router
--------+---------------+---------------+--------+----+------------+-----------------------------

Script: pingcheck.sh

#!/bin/bash
#
# Extended ping tester.
#
# v3.1 sol@subnetzero.org
#
# This script allows you to run extended pings across a large
# amount of devices, resulting in a report table once complete.
#
# Create a pingdevices.csv config file to use with this script
# in the following format:
#
#   #Device,Ping-IP,Description#Source_Address(Optional)
#   testrtr1,172.17.1.1,TEST-1,192.168.10.221
#   testrtr2,10.10.10.1,TEST-2
#
# DO NOT USE COMMAS in the description field, or this will break
# any source address configuration in the final field.

pingexp_ssh(){
/usr/bin/expect << EOF
spawn ssh $USNAME@$DEVICE
#login handles cases:
#   user/pass
#   login with keys (first time verification)
expect {
        "continue connecting (yes/no)?" {send "yes\r" ; exp_continue}
        "assword:" { send "$PASS\r" }
}
expect -re "(>|#)" {send "term len 0\r"}
expect -re "(>|#)" {send "enable\r"}
expect "assword:" { send "$PASS\r" }
expect -re "(>|#)" {send "ping\r"}
expect {
        "otocol"      {send "ip\r"}
        timeout       {puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\tNA\tERROR 01 - CMDERROR"
                       exit}
}
expect {
        "address:"   {send "$DESTIP\r"}
         timeout      {puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\tNA\tERROR 02 - CMDERROR"
                      exit}
}

expect "count"     {send "$PKTCNT\r"}
expect "size"      {send "$PKTLEN\r"}
expect "seconds"   {send "3\r"}
expect "commands"  {send "n\r"}
expect "range"     {send "n\r"}

# Check for success marks
set timeout 30
expect {
        -re "Success rate is.*\n" {puts "latency"

             puts \$expect_out(0,string)
             puts [lindex \$expect_out(0,string) 3]
             puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tSUCCESS\t[lindex \$expect_out(0,string) 3]\t[lindex \$expect_out(0,string) 9]\t$DESCRP"
             puts "End of Output"
        }
        timeout {puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\t$DESCRP"}
        }

set timeout 7
# Logout
expect {
        "#"     { send "exit\r" }
        timeout { exit }
        }
EOF
}

pingexp_ssh_srcip(){
/usr/bin/expect << EOF
spawn ssh $USNAME@$DEVICE
#login handles cases:
#   user/pass
#   login with keys (first time verification)
expect {
        "continue connecting (yes/no)?" {send "yes\r" ; exp_continue}
        "assword:" { send "$PASS\r" }
}
expect -re "(>|#)" {send "term len 0\r"}
expect -re "(>|#)" {send "enable\r"}
expect "assword:" { send "$PASS\r" }
expect -re "(>|#)" {send "ping\r"}
expect {
        "otocol"      {send "ip\r"}
        timeout       {puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\tNA\tERROR 03 - CMDERROR"
                       exit}
}
expect {
        "address:"   {send "$DESTIP\r"}
         timeout      {puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\tNA\tERROR 04 - CMDERROR"
                      exit}
}

expect "count"               {send "$PKTCNT\r"}
expect "size"                {send "$PKTLEN\r"}
expect "seconds"             {send "3\r"}
expect "commands"            {send "y\r"}
expect "ource address"       {send "$SRCADD\r"}
expect "ype of service"      {send "\r"}
expect "DF bit in IP header" {send "\r"}
expect "alidate reply data"  {send "\r"}
expect "ata pattern"         {send "\r"}
expect "erbose"              {send "\r"}
expect "range"               {send "n\r"}

# Check for success marks
set timeout 30
expect {
        -re "Success rate is.*\n" {puts "latency"

             puts \$expect_out(0,string)
             puts [lindex \$expect_out(0,string) 3]
             puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tSUCCESS\t[lindex \$expect_out(0,string) 3]\t[lindex \$expect_out(0,string) 9]\t$DESCRP"
             puts "End of Output"
        }
        timeout {puts "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\t$DESCRP"}
        }

set timeout 7
# Logout
expect {
        "#"     { send "exit\r" }
        timeout { exit }
        }
EOF
}


########### SET VARS ##########
#
# Number of and size of packets
#
PKTCNT=10
PKTLEN=100
#
# Set inut file to be used
if [ "$1" ]; then
   if [ -r "$1" ]; then
      DEVICEFILE=$1
   else
      echo "Input file $1 not found or is not readable."
      exit 1
   fi
else
   DEVICEFILE=pingdevices.csv
fi

########### Sanity ############
#
if [ ! -w . ]; then
   echo "Error: Can't write to current directory."
   exit 1
fi

########### Cleanup ###########
#
cat /dev/null > pingcheck.log


######### USER INPUT ##########
#
printf "Data will be read from $DEVICEFILE in current directory ($PWD) for tests.\n\n"
printf "Tacacs user:" ; read USNAME; stty -echo
printf "Tacacs pass:" ; read PASS ; printf "\n" ; stty echo

# Set IFS to newline so we can read whole lines as elements
#
IFS="
"

# Loop through each line and run the ping check
#
for DATALINE in `egrep -v "^$|^#" $DEVICEFILE`; do
   TSTAMP=`date +%H:%M:%S`
   DEVICE=`echo $DATALINE | awk -F"," '{print $1}' | egrep -v "^#|^$"`
   DESTIP=`echo $DATALINE | awk -F"," '{print $2}'`
   DESCRP=`echo $DATALINE | awk -F"," '{print $3}'`
   SRCADD=`echo $DATALINE | awk -F"," '{print $4}' |\
           egrep '^[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}$'`
   ERRMSG="\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL"
   printf "Working on $DEVICE, Target IP $DESTIP.\n"
   ping -c1 -w1 $DEVICE > /dev/null
   if [ "$?" -eq "0" ]; then
      if [ "$SRCADD" ]; then
         pingexp_ssh_srcip | tee -a pingcheck.log
      else
         pingexp_ssh | tee -a pingcheck.log
      fi
   else
      printf "\nRESULT:$TSTAMP\t$DEVICE\t$DESTIP\tFAIL\tNA\tNA\t$DESCRP\n" >> pingcheck.log
   fi
done

echo "
Time     Device          Target IP       Status   PKT% Min/Avg/Max  Description
--------+---------------+---------------+--------+----+------------|-----------------------------" > pingresults.txt

grep "^RESULT" pingcheck.log |\
 cut -d":" -f2- | \
 awk '{tail = substr ($0,length($1 $2 $3 $4 $5 $6) +7)}
 {printf ("%8.8s|%-15.15s|%-15.15s|%-8.8s|%-4.4s|%-12.12s|%-30.30s\n",$1,$2,$3,$4,$5,$6,tail);
 }'>> pingresults.txt
echo "--------+---------------+---------------+--------+----+------------+-----------------------------" >> pingresults.txt
clear
cat pingresults.txt