Sever Sending ARP Broadcasts Once a Second

[Andrew Griffin]

A quick background:

We have an N4000 running HP-UX 11.00 with an HP A4926A Fiber gigabit card, used primarily as an Oracle DB server for CAD/CAM data. Up until about 4 months ago, we also had ~35 C240 unix workstations used by the engineers, which were part of an NIS domain, managed by the above server. We are also running NNM which was monitoring the server and all unix workstations. Those workstations have since been converted to Windows workstations, and all of their IP addresses were given back to the Windows folks for DHCP use.

Here's the problem: The server is sending out what appears to be ARP broadcasts (you can only see the broadcasts through the monitor on the fiber switch itself) every second to ~25 of the IP addresses that were previously used by the unix workstations.

At first, I thought that NIS was searching for these machines, but all of their entrys were taken out of /etc/hosts long ago, and NIS is not configured in to the /etc/nsswitch.conf file. To be sure, I killed the NIS daemons, and the behavior continued.

Then I thought NNM was looking for these machines, even though I know that we have NNM configured to quit looking for nodes if they are missing for more than a week. I checked, and while they're are still on my map (I'll clean that up later), they are all toggled as "REMOVED". To be sure, I killed the OV daemons and the behavior still continued.

I dug through every config file I could find, and I still couldn't find where these workstations were being referenced. It seems to me that there is something on the system sending out these ARP broadcasts trying to get responses from workstations that aren't there any more. Is there a way to map network activity to individual process in order to find out what process is making this noise? Is there something I'm missing?

Oh, by the way, I've checked the ARP table itself and it doesn't contain any of the old addresses.

Any help is appreciated.

[Steven Gillard_2 · Accepted solution]

It is possible to get this information, using a "nettl" trace which tracks thread ids. Once you have the thread id you can use glance to work out what process owns the thread. Here's the steps:

1. Set up a "filter" file to trace only broadcast traffic. This file should contain the following line:

filter dest ff-ff-ff-ff-ff-ff

2. Run the following command to start the trace and capture the appropriate information from the nettl trace:

# nettl -tn pduout -e all -s 1024 | netfmt -lnN -c filter | egrep "ARP|Requesting"

3. When the ARP requests occur, you should see lines similar to the following appear:

================================ ARP REQUEST (outbound -- pid: 26516) =======
Requesting: x.x.x.x
================================ ARP REQUEST (outbound -- [ICS]) ============
Requesting: x.x.x.x
================================ ARP REQUEST (outbound -- [ICS]) ============
Requesting: x.x.x.x

If the "pid" is specified, you know a living process has triggered the ARP request, otherwise its come from the network layer. Note that even though this is labelled "pid", it is actually the thread id, not the process id so you can't simply use ps to find the process.

4. Once you have the thread id, go into glance and select the "Thread List" screen. From this screen type 'S' to select a thread, then enter the thread id as found from nettl. This will bring up a screen which will give you the process info that you require.

5. To terminate your nettl trace, ^C the session and run:

# nettl -tf -e all

The only problem here is that if it is a ping process being run from a script or something it may not be alive very long. You'll have to keep trying until you're able to trap the process in glance.

Good luck!

Regards,
Steve

[Andrew Griffin]

Thank you so much. I figured that nettl was what could get the info I needed. I've been trying various nettl commands the past few days, but admittedly my nettl experience is lacking and I couldn't make heads or tails of the output I was gettting. Turns out is was rpc.statd sending all the ARP requests. Thanks again!