nmap ping scan doesn’t populate arp cache

This drove me absolutely bats**t mental. I couldn’t understand why running an nmap -sP 10.0.0.0/24 (locally attached subnet) as root hit all the available hosts, but didn’t populate the local arp cache when you looked at the output of arp -a.

It seems that the -sP parameter (ping only) doesn’t hit the required level of the OS stack to populate the cache, even when running as root, so the required steps are:

1) Ensure that you can run nmap as root via sudo to allow raw socket capability.
2) Add the –send-ip option to the nmap command. This will ensure that the command uses raw sockets and populates the ARP table.

eg:

nmap -sP 10.0.0.0/24 -n --send-ip

Then you will see that the ARP table is updated with:

/sbin/arp -a

The reason I needed this was because I had to enhance the network mac/ip/dns resolution script to use local ARP entries. Without this option, the hosts are shown as up but the OS arp table never gets updated.

Beware that if you run nmap -sP as an unpriveleged user, it’s almost useless as it targets TCP port 80 (according to the man page) rather than using ICMP echoes.

Apparently, fping has an option to support this (-g), but I rarely see it installed as part of a standard build.

Building Smokeping on CentOS and running without root permissions.

I needed smokeping to run some tests, but every guide online alluded to using root for the application user which is really not ideal. I’m still not happy about setuid for fping, but as long as you provide at least basic authentication on the web front end, it shouldn’t be too much of a problem when it comes to audit.

The sections mention swapping between smokeping and root users. This will typically involve a lot of ctrl + D and sudo su – commands. Use common sense in the procedure below for this and use the “id” command to check who you are effectively logged in as. I know you can chown -R but I chose to use the actual user and then lock it down once finished to avoid confusing permission oversights. :p

I have run through this procedure myself and it works on a CentOS minimal install. There was a frustrating issue I had with getting echoping built with ssl support. Running configure and running into library errors usually points to development versions not being installed, NOT the normal, user libraries. In this case, openssl-devel and popt-devel. Thanks to my good friend Cody for pointing this out before I went crazy! :D

smokeping install – CentOS 6
With credit to WeDebugYou.com for the base on which I built this procedure.


# 1) Create smokeping non-priveleged user

sudo su -
useradd -m -s /bin/bash smokeping

# 2) Install prerequisite packages

yum install wget
cd /tmp
wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
rpm -Uvh rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm

# Required
yum install mod_fcgid httpd httpd-devel rrdtool fping wget curl bind-utils gcc make  
yum install perl perl-Net-Telnet perl-Net-DNS perl-LDAP perl-libwww-perl perl-RadiusPerl 
yum install perl-IO-Socket-SSL perl-Socket6 perl-CGI-SpeedyCGI perl-FCGI perl-RRD-Simple
yum install perl-CGI-SpeedyCGI perl-ExtUtils-MakeMaker


# 3) Install smokeping

cd /home/smokeping
mkdir smokeping
chown smokeping:apache smokeping	# Allow apache to read the dir later
chmod 750 smokeping			# Allow apache to read the dir later
su - smokeping
wget http://oss.oetiker.ch/smokeping/pub/smokeping-2.6.9.tar.gz

tar -zxvf smokeping-2.6.9.tar.gz -C /home/smokeping
cd smokeping-2.6.9/setup
./build-perl-modules.sh

cp -r ../thirdparty /home/smokeping/smokeping
cd ..
./configure --prefix=/home/smokeping/smokeping
make install


Create missing folders
cd /home/smokeping/smokeping
mkdir data var cache


# 4) Add startup script

[go back to root with ctrl + d ]

# Copy and paste from the line below into /etc/init.d/smokeping

#!/bin/sh
#
# smokeping    This starts and stops the smokeping daemon
# chkconfig: 345 98 11
# description: Start/Stop the smokeping daemon
# processname: smokeping
# Source function library.
. /etc/rc.d/init.d/functions

SMOKEPING=/home/smokeping/smokeping/bin/smokeping
SMOKEPINGUSER=smokeping
LOCKF=/var/lock/subsys/smokeping
CONFIG=/home/smokeping/smokeping/etc/config

[ -f $SMOKEPING ] || exit 0
[ -f $CONFIG ] || exit 0

RETVAL=0

case "$1" in
  start)
        echo -n $"Starting SMOKEPING: "
        daemon --user $SMOKEPINGUSER $SMOKEPING
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch $LOCKF
        ;;
  stop)
        echo -n $"Stopping SMOKEPING: "
        killproc $SMOKEPING
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && rm -f $LOCKF
        ;;
  status)
        status smokeping
        RETVAL=$?
        ;;
  reload)
        echo -n $"Reloading SMOKEPING: "
        killproc $SMOKEPING -HUP
        RETVAL=$?
        echo
        ;;
  restart)
        $0 stop
        sleep 3
        $0 start
        RETVAL=$?
        ;;
  condrestart)
        if [ -f $LOCKF ]; then
                $0 stop
                sleep 3
                $0 start
                RETVAL=$?
        fi
        ;;
  *)
        echo $"Usage: $0 {start|stop|status|restart|reload|condrestart}"
        exit 1
esac

# Stop copying!

chmod 755 /etc/init.d/smokeping


# 5) Back to smokeping user to set up config (probably ctrl+D rather than sudo su command)

sudo su - smokeping
cd /home/smokeping/smokeping/etc
for foo in *.dist; do cp $foo `basename $foo .dist`; done
vi config

change the following:

owner    = Jean Debogue
contact  = noc@jeandebogue.com
cgiurl   = http://graph.mydomain.com/smokeping/smokeping.cgi

imgcache = /home/smokeping/smokeping/cache
datadir  = /home/smokeping/smokeping/data
piddir  = /home/smokeping/smokeping/var
smokemail = /home/smokeping/smokeping/etc/smokemail.dist
tmail = /home/smokeping/smokeping/etc/tmail.dist

*** Presentation ***

template = /home/smokeping/smokeping/etc/basepage.html.dist

:wq!


# 6) Configure smokeping for apache

sudo su -

cd /home/smokeping/smokeping
ln -s /home/smokeping/smokeping/cache /home/smokeping/smokeping/htdocs/cache
chown -R apache cache
chown -R apache data

Now edit apache config…

#Add these lines into the file /etc/httpd/conf.d/smokeping.conf (ignore the
#Auth stuff and htpasswd if you don't want to password protect the dir)
#I do this out of paranoia.


ScriptAlias /smokeping/smokeping.cgi /home/smokeping/smokeping/htdocs/smokeping.fcgi.dist
Alias /smokeping /home/smokeping/smokeping/htdocs

< Directory "/home/smokeping/smokeping/htdocs">
        Options FollowSymLinks
        AuthType Basic
        AuthName "Smokeping"
        AuthBasicProvider file
        AuthUserFile /etc/httpd/passwd/passwords
        Require valid-user
< /Directory>


#Create Auth password if you need it...
mkdir /etc/httpd/passwd
htpasswd -c /etc/httpd/passwd/passwords webuser

[specify password, eg: smokeping]

chmod 600 /etc/httpd/passwd/passwords


#################### IF YOU HAVEN'T ALREADY SET APACHE UP PROPERLY DO IT NOW #####################
vi /etc/httpd/conf/httpd.conf

# IP below is an example!

Listen 192.168.1.25:80
DocumentRoot "/var/www/html"

And the rest…


# 7) setuid for fping
chown root:root /usr/sbin/fping
chmod u+s /usr/sbin/fping

edit /home/smokeping/smokeping/etc/config

remove the slaves section as we're not using that.

Replace the following from probes section

============================================================

*** Probes ***

+ FPing
binary = /usr/sbin/fping
packetsize = 750
step = 60

+ DNS
 binary = /usr/bin/dig
 server = 8.8.8.8
 pings = 3
 forks = 5

# Use these after you've compiled echoping
#+ EchoPingHttp
# binary = /usr/bin/echoping
# pings = 5
# forks = 5
# offset = 50%
# ipversion = 4
# url = /
#
#+ EchoPingHttps
# binary = /usr/local/bin/echoping
# pings = 5
# forks = 5
# offset = 50%
# ipversion = 4

+ Curl
 # probe-specific variables
 binary = /usr/bin/curl
 pings = 5

# a default for this target-specific variable
urlformat = http://%host%/


*** Targets ***

probe = FPing

menu = Top
title = Network Latency Grapher
remark = Welcome to the SmokePing website of SubnetZero.org!

 + network
 menu = Net latency
 title = Network latency (ICMP pings)

 ++ www1
 host = www.google.com

 + services
 menu = Service latency
 title = Service latency (DNS, HTTP)

 ++ DNS
 probe = DNS
 menu = DNS latency
 title = Service latency (DNS)

 +++ www1
 host = www.google.com

 + HTTP
 probe = Curl
 menu = http full page
 title = HTTP latency

 ++ www1
 host = www.google.com




==============================================

Double check permissions are ok:

ls -l /home/smokeping
drwxr-x---. 11 smokeping apache      4096 Jul  3 22:52 smokeping

cd /home/smokeping/smokeping
ls -l
drwxrwxr-x. 2 smokeping smokeping 4096 Jul  3 22:41 bin
drwxrwxr-x. 6 apache    smokeping 4096 Jul  3 23:55 cache
drwxrwxr-x. 6 apache    smokeping 4096 Jul  3 23:14 data
drwxrwxr-x. 3 smokeping smokeping 4096 Jul  3 23:13 etc
drwxrwxr-x. 3 smokeping smokeping 4096 Jul  3 23:28 htdocs
drwxrwxr-x. 3 smokeping smokeping 4096 Jul  3 22:42 lib
drwxrwxr-x. 3 smokeping smokeping 4096 Jul  3 22:42 share
drwxrwxr-x. 6 smokeping smokeping 4096 Jul  3 22:40 thirdparty
drwxrwxr-x. 2 smokeping smokeping 4096 Jul  3 23:13 var


If necessary..
chown -R apache cache
chown -R apache data

#8) Start smokeping and apache
/etc/init.d/smokeping start
/etc/init.d/httpd start


#### TEST AND IF ALL IS OK.... #######

sudo su -

chkconfig --add smokeping
chkconfig httpd on
chkconfig smokeping on

#lock down smokeping user

usermod -s /bin/false smokeping

======================================================

#9) Optional... build echoping with ssl

# If you locked down smokeping user already then do
usermod -s /bin/bash smokeping

sudo su - 
yum install openssl-devel popt-devel


sudo su - smokeping  
cd /tmp
wget http://downloads.sourceforge.net/project/echoping/echoping/6.0.2/echoping-6.0.2.tar.gz
tar zxvf echoping-6.0.2.tar.gz
cd echoping-6.0.2
./configure --prefix /usr/local/ --enable-icp --with-ssl --without-libidn


sudo su - 
cd /tmp/echoping-6.0.2
make test 
make install

#test echoping

/usr/local/bin/echoping -C -h /dana-na/auth/url_default/welcome.cgi rcseu.rabobank.com


#lock down smokeping user
usermod -s /bin/false smokeping

Note that if you want to use SSL (EchoPingHttps) probes, you MUST refer to the /usr/local/bin/echoping binary now instead
of the one that may already be installed.

This is a fantastic tool but it does have some peculiarities. I don’t agree with the default graph scaling, which you’ll soon see clips out the “smoke” peaks. Changing unison_tolerance for targets can work around this to an extent but you have to remember that graphs scale to the median. You will also probably want to review the RRDTool database aggregation if you want high resolution polling.

Here are my changes in the config file for 1 minute resolution… If I’ve done anything wrong, I’ll be happy to correct this post.

# consfn mrhb steps total

AVERAGE  0.5   1  10080 # 7 days of 1 min = 86400 seconds in a day. So 86400 * 7 / (step value 60) = 10080
AVERAGE  0.5   5  8064  # 4 weeks of 5 min =  2016*5 min aggregates in a week = 8064
    MIN  0.5   5  8064
    MAX  0.5   5  8064
AVERAGE  0.5  60  2016 # 3 months of 1 hour = 24*60min aggregates in a day * 7days * 12 weeks = 2016
    MAX  0.5 144  2016
    MIN  0.5 144  2016

Converting MAC address to another format

It often annoys me that I ask for MAC addresses from other people in other departments, only for them to supply them in a format that I can’t just copy and paste into my terminal for use in a script or cisco CLI. To this end, I came up with this short conversion script:

#!/bin/bash
#
# Function: MAC Address conversion tool
# Author:   sol@subnetzero.org
#
# Usage example: "echo 0000.0000.0000" | macconvert 4d2d

4dto2d () {
awk '{
print substr($0,1,2)"-"\
substr($0,3,2)"-"\
substr($0,6,2)"-"\
substr($0,8,2)"-"\
substr($0,11,2)"-"\
substr($0,13,2)}'
}

4dto4c () {
awk -F"." '{print $1":"$2":"$3}'
}

2dto4d () {
awk -F"-" '{print $1$2"."$3$4"."$5$6}'
}

2cto4d () {
awk -F":" '{print $1$2"."$3$4"."$5$6}'
}

2cto2d () {
sed 's/:/./g'
}

USAGE="$0 [4d2d] [2d4d] [2c4d] [2c2d]
4d2d = Convert xxxx.xxxx.xxxx to xx-xx-xx-xx-xx-xx
4d4d = Convert xxxx.xxxx.xxxx to xxxx:xxxx:xxxx
2d4d = Convert xx-xx-xx-xx-xx-xx to xxxx.xxxx.xxxx
2c4d = Convert xx:xx:xx:xx:xx:xx to xxxx.xxxx.xxxx
2c2d = Convert xx:xx:xx:xx:xx:xx to xx.xx.xx.xx.xx.xx

Requires input stream from redirect or pipe.
"

case $1 in
        4d2d)   4dto2d;;
        4d4c)   4dto4c;;
        2d4d)   2dto4d;;
        2c4d)   2cto4d;;
        2c2d)   2cto2d;;
        *)      echo "$USAGE";
                exit 1;;
esac

Usage… either echo the MAC addess, or create a text file with MAC addresses in it, then pipe it into this script (or redirect it into it) with the required conversion option.

Quickly modifying .ssh/known_hosts

If strict checking ( StrictHostKeyChecking yes in /etc/ssh/ssh_config) is enabled on ssh, it can be a bit of pain if equipment is being upgraded/replaced on a regular basis in your environment. It means that you have to remove the old key from your .ssh/known_hosts after seeing this message:

[sol@testserver .ssh]$ ssh testbox.subnetzero.org
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The RSA host key for testbox.subnetzero.org has changed,
and the key for the corresponding IP address 192.168.100.43
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
eb:bd:d7:21:7b:be:06:f3:97:b9:ac:d7:c0:bd:d3:5f.
Please contact your system administrator.
Add correct host key in /home/sol/.ssh/known_hosts to get rid of this message.
Offending key in /home/sol/.ssh/known_hosts:1
RSA host key for testbox.subnetzero.org has changed and you have requested strict checking.
Host key verification failed.

Solution for the lazy (or for people who hate vi): a simple script to clear out the culprits.

#!/bin/sh
# clear out an unwanted host entry in the known_hosts file
if [ "$1" ]; then
        HOST=`echo $1 | sed 's/\./\\./g'`
        grep -iv "^$HOST[, ]" $HOME/.ssh/known_hosts  > $HOME/.ssh/known_hosts.tmp
        mv $HOME/.ssh/known_hosts.tmp $HOME/.ssh/known_hosts
else
        echo "No Argument supplied. Usage: clearhost [hostname]"
fi

Simply run the script with the IP/hostname as an argument and it’ll be gone from the known_hosts file. Obviously only do this if you know for sure that the change is for legitimate reasons. :)

X11 when using sudo

This is mentioned in a previous article, but I thought it was justified to publish it separately, seeing as this has caused so many headaches in the past.

Often, you are handed a server to build your application on, and are temporarily granted root access via sudo to do what you need. Sometimes, you need to run an X11 app as root for an installation routine. This doesn’t work!

How to work around this:

[user@server ~] xauth list
server.localdomain.com/unix:10  MIT-MAGIC-COOKIE-1  58682a5bb5a4f731ae15c186ff3d68f8
[user@server ~] sudo su -
[root@server ~] xauth add [PASTE xauth list OUTPUT HERE]
[root@server ~] xclock &

X11 forwarding will now work properly, assuming that X11Forwarding is allowed in your sshd_config and you have allowed X11 forwarding in your ssh client.

Beware that X11 these days can be a bit shaky. If your app crashes, it’s probably worth trying a different XServer. From experience, ReflectionX on Windows 7 has caused me a lot of pain. XMing and Exceed still work well.