Firewall reporting excessive ICMP from HP-UX boxes

[Susan Pellerito]

The firewall admins are reporting excessive ICMP requests from the HP-UX servers running 11i. How can I begin to research how and why these ping requests are happening and stop them?

[Steven E. Protter]

Shalom Susan,

I'd like to see a log with some data before getting excited about this. I am not sure what the definition of excessive is.

HP-UX 11i can operate just fine with ICMP disabled. It takes a single ndd command to allow an 11.00 server to operate the same way.

I'd start with as a reseach tool tcpdump or ethereal(now known as wireshark).

You can look at port 8 or 11 (I can't remember which and see if there really are a lot of ping requests coming off the HP-UX boxes.

As a security measure, I'd make sure users can't use ping. Its a adminitrator tool and by default can not be used by unpriviledged users, eg users who are not root.

The tcpdump output will at least let you identify which servers are problematic.

The causes vary and could be due to a hardware or wiring fault or abuse of icmp commands.

The firewall can be configured to block ICMP from HP-UX boxes. This will annoy the admins but not effect normal operation of the systems.

SEP

[Patrick Wallek]

What's excessive? Can the firewall folks show you what the source and destination of the packets is? Is there a common time? Do they happen regularly?

What are you running on the HP-UX machines? Openview? Big Brother? Any other type of monitoring software? Some monitoring software regularly sends pings to determine if the host is up or not. This may be what is happening.

If the pings happen at fixed intervals check cron and see if any cron jobs are doing pings.

[spex]

Hi Susan,

There are many ways to approach this. The first thing I would do is compare the firewall log to the cron log (/var/adm/cron/log). Try to correlate ICMP packet requests with cronjobs that have run. Also see if anything out of the ordinary is running ('top') or scheduled to run ('crontab -l ').

Another method would be to monitor network traffic from your HP-UX servers. If they are linked to your firewall via a hub, you may be able to passively monitor network traffic with a PC attached to the same hub. This will require software such as Wireshark, which is available for Linux and Windows, among other OSes. If this is not an option, you could install tcpdump on your HP-UX servers (available here: http://hpux.connect.org.uk/hppd/hpux/Networking/Admin/tcpdump-3.9.5/) to do the same thing.

With your monitoring software, you will want to look for other packets sent from your server immediately before or after the ICMP packets. Try to determine the source application by analyzing the contents of these packets.

PCS

[Bill Hassell · Accepted solution]

If these ICMP packets occur at a regular rate (every few minutes, there are two possibilities:

1. A long-forgotten cron script that is checking if various IP addresses are 'working'.

2. The networking software is performing a dead gateway detection probe (more likely). The ndd program can show the current state and if necessary, turn off this option. To see the current value:

ndd -get /dev/tcp ip_ire_gw_probe

If you see 1, then the network driver is probing all gateways with an ICMP request to see if they are 'working'. (working is a misnomer -- it really means a reply ws created and has no bearing on functionality except that the power is turned on). Many firewall and router managers will disable ICMP responses in which case, if your setting is 1, within 5 minutes, your link will go down due to dead gateway detection failure.

Turn off this feature with:

ndd -set /dev/tcp ip_ire_gw_probe 1

Then check with the network folks to see if the ICMP packets have stopped. To make this change permanent, edit /etc/rc.config.d/nddconf file. Then test to see that the file is valid with:

ndd -c

NOTE: early versions of ndd failed the -c option. Make sure your ndd is fully patched.

[Susan Pellerito]

Bill, Don't you mean to set the ip_ire_gw_probe to 0? Since the firewall log does not show any consistency in the time between log files, I have ruled out any cron jobs. I only allow root to run cron and see nothing unusual. Therefore, I am setting the ndd param to not probe gateway and will check with firewall admins in the morning.

[Bill Hassell]

Yes, I meant 0. I was looking at a system that has it set to 1 and got a brain freeze -- or maybe it was a defective keyboard...