topic Re: Rogue pings in Operating System - HP-UX

Rogue pings

Nicolas_17 — Tue, 05 Jul 2005 09:33:50 GMT

Hi everyone,

I have to find the cause of a server that started to ping repeatedly other nodes on our network. We know of the pings because our firewall is intercepting and dropping the packets so they're not causing any problems, but we don't know what process/app is generating them. I was looking through the netstat manpage to find some way to identify the rogue process/app, but i can't find what i'm looking for.

Does anyone know how to measure outgoing traffic from a unix machine and associate this traffic to a process/app?

Thanks!

Re: Rogue pings

Mahesh Kumar Malik — Wed, 06 Jul 2005 02:43:53 GMT

Hi Nicolas

Use ps -ef | grep ping to know process which is causing ping.

Regards
Mahesh

Re: Rogue pings

Muthukumar_5 — Wed, 06 Jul 2005 03:08:08 GMT

You can use like,

# netstat -nap icmp
# ps -ef | grep ping

to get ping related operations.

hth.

Re: Rogue pings

melvyn burnard — Wed, 06 Jul 2005 03:17:42 GMT

Has someone perhaps installed/configured/started a monitoring application on a node somewhere?
I think OpenView Network Node Manager may do this to discover nodes, for example.

Re: Rogue pings

Nicolas_17 — Wed, 06 Jul 2005 10:33:52 GMT

Thanks for the replies guys,

The ps -ef | grep ping return nothing except my grep.
The netstat -nap icmp returns this:

icmp:
739602 calls to icmp_error
0 errors not generated because old message was icmp
Output histogram:
echo reply: 3803
destination unreachable: 739560
routing redirect: 5
time exceeded: 39
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Input histogram:
echo reply: 32462
destination unreachable: 8163
routing redirect: 35
echo: 3803
time exceeded: 39
3803 message responses generated

Does anything seem abnormal?

As for someone installing a monitoring application, well, our consultants
are somewhat confused and often do strange things...i'll see what i can find out.

Re: Rogue pings

rick jones — Wed, 06 Jul 2005 11:45:15 GMT

If I remember correctly, ICMP echo requests can be generated via a raw IP socket. Perhaps lsof can be used to show which processes on a system have a raw IP socket open rather than a TCP or UDP endpoint. You might then confirm that is the process by using a system call trace such as that from tusc.

Unless the ICMP echo requests are coming at a _very_ high rate and/or are using spoofed source IPs, even if your firewalls were not filtering them, they would not be causing any problems.

Re: Rogue pings

MarkSyder — Thu, 07 Jul 2005 03:40:41 GMT

Have you recently installed Big Brother? This pings servers/workstations to check connectivity.

Mark Syder (like the drink but spelt different)

Re: Rogue pings

Ermin Borovac — Thu, 07 Jul 2005 05:14:39 GMT

# lsof -i | grep ICMP
dced 1359 root 11u IPv4 0x429f0e40 0t0 ICMP *:*
ping 9721 root 6u IPv4 0x4293c040 0t0 ICMP *:*

to see if you can spot any suspects.

Re: Rogue pings

Nicolas_17 — Thu, 07 Jul 2005 09:00:00 GMT

Rick, when you say:

"If I remember correctly, ICMP echo requests can be generated via a raw IP socket."

What does this mean? Because they are exactly that, echo requests, I can see it in my trace report.

Ermin, the output gives me only one process using icmp:

root 12012 13988 0 Jun 22 - 54:29 nim_ether

Does this mean we found the culprit?

Thanks again for the replies!

Nicolas.

Re: Rogue pings

Ron Kinner — Thu, 07 Jul 2005 10:22:23 GMT

An HP server with dead gateway detection turned on will ping its gateways periodically. This feature is on by default in most HPUX 11.0 and higher.

You can turn it off with ndd.

ndd -set /dev/ip ip_ire_gw_probe 0

tho to make it stick after a reboot you need to edit /etc/rc.config.d/nddconf to add:

TRANSPORT_NAME[0]=ip
NDD_NAME[0]=ip_ire_gw_probe
NDD_VALUE[0]=0

Use the next higher integer in the brackets if you already have entries there.

HPUX 10.3 and 11.0 also use an MTU detection scheme based on pings. This can also be changed in NDD.

/usr/bin/ndd -set /dev/ip ip_pmtu_strategy 1

or in nddconf:

TRANSPORT_NAME[1]=ip
NDD_NAME[1]=ip_pmtu_strategy
NDD_VALUE[1]=1

The original default, Option 2 was dropped in 11i.

Ron

Re: Rogue pings

Ermin Borovac — Thu, 07 Jul 2005 20:14:59 GMT

Yes, well that's likely to be the culprit.

You can verify with tusc. If you don't have tusc you can get it from

ftp://ftp.cup.hp.com/dist/networking/tools

Then attach to the process as

# tusc -fv -s sendto

In the output you should see this process making connections to other systems.

Re: Rogue pings

rick jones — Fri, 08 Jul 2005 20:50:56 GMT

I mean it is not a TCP or UDP socket :)

Re: Rogue pings

Nicolas_17 — Mon, 18 Jul 2005 07:00:57 GMT

Thanks for all your help guys!

I'm closing this thread now that we found the process responsible for the pings...and that it seems to be a necessary process for our cluster to work!

Thanks again!