ICMP Destination Unreacheable messages

[Achilles_2]

Hi mate,

Do you have any idea when capture all the ICMP messages by using the icmpinfo tool, and I got thousands of ICMP_Dest_Unreacheable messages from 127.0.0.1 like :

ICMP_Dest_Unreachable[Port] < 127.0.0.1 [localhost] > 127.0.0.1 [localhost] sp=60838 dp=53 seq=0x00330000 sz=79(+20)
0000 : 4506 004F 29F5 0000 FF01 93A2 7F00 0001
0010 : 7F00 0001 0303 7C23 0000 0000 4500 0047
0020 : 29F4 0000 FF11 93AF 7F00 0001 7F00 0001
0030 : EDA6 0035 0033 0000 3FE5 0100 0001 0000
0040 : 0000 0000 0238 3203 3132 3802 3630 02

What is the explanation of the above message? Is it a network problem?

Many thanks

[Michael Steele_2 · Accepted solution]

localhost is a software loopback test for verifying the tcp daemon is up and working. localhost 127.0.0.1 represents your computer. 127.0.0.1 is a reserved ip address. If you are failing then either you're inetd daemon is not up, perhaps you're only in single user mode or only at run-level two instead of run level 3 ( init 3 and who -r ), or your /etc/hosts file is corrupt. Here's the standard test, paste in the results:

ping localhost / ping 127.0.0.1.

Also paste in your /etc/inittab file, I'd like to check the default run level.

[Dave Olker]

Hi mate,

The source and destination IP addresses are 127.0.0.1, which means the packet originated on the local host and it was trying to contact a port on the local host. In other words, this is loopback traffic on the local system.

The source port is 60838, which is just a randomly assigned port in the anonymous port range. The destination port is 53, which means DNS.

So, what does all this mean? It would indicate to me that you've got some process running on your system that is trying to contact a DNS server on the local system (at port 53) and since your system is obviously not a DNS server it returns an ICMP Destination Unreachable because there is no process listening on port 53.

Did you intend to make the local system a DNS server? Does your /etc/resolv.conf point to the local system as being a DNS server? Does your /etc/nsswitch.conf file say to use DNS to resolve hostnames and IP addresses? If so your /etc/resolv.conf file should be pointing to a valid DNS server.

The fact that you're getting thousands of these means some process is trying to contact a DNS server frequently. Are they all from the same source port of 60838? If so, I'd grab a copy of lsof and determine which process is using port 60838 and that will tell you the process that is causing these ICMP messages. Once you know which process is sending requests to port 53 you might be able to configure it to stop doing so.

Hope this helps,

Dave

[Steven E. Protter]

Shalom,

ping on port 53?

Someone has written something non-standard. A normal nslookup would not do this.

This is not the result of the OS or add in products. It appears to be the result of a programmer that needs a little education.

:-()

SEP

[Dave Olker]

Why is this a ping on port 53?

A ping is an ICMP Echo Request/Reply. This above error is an ICMP Destination Unreachable caused by a packet being sent to a port that no process is listening on. This doesn't mean the ping program was used, it could be any application trying to send data to port 53, which is why it's important to figure out which application is using port 60838 in the above error.

If you can figure out which application is sending the packets you can figure out why.

Regards,

Dave

[Achilles_2]

Hi Dave,

Thanks your answers, and I forgot to mention the value of "sp" keeps changing ... it means not only one process is trying to contact a port on the local host? What's the way to terminate these useless processes? Also how to use losf to determine which process is using port either 60838 or other?

ICMP_Dest_Unreachable[Port] < 127.0.0.1 [localhost] > 127.0.0.1 [localhost] sp=61718 dp=53 [domain] seq=0x00300000 sz=76(+20)
0000 : 4500 004C 3D05 0000 FF01 8095 7F00 0001 E..L=...........
0010 : 7F00 0001 0303 7729 0000 0000 4500 0044 ......w)....E..D
0020 : 3D04 0000 FF11 80A2 7F00 0001 7F00 0001 =...............
0030 : F116 0035 0030 0000 4711 0100 0001 0000 ...5.0..G.......
0040 : 0000 0000 0131 0130 0130 0331 .....1.0.0.1

ICMP_Dest_Unreachable[Port] < 127.0.0.1 [localhost] > 127.0.0.1 [localhost] sp=61726 dp=53 [domain] seq=0x00300000 sz=76(+20)
0000 : 4500 004C 3D15 0000 FF01 8085 7F00 0001 E..L=...........
0010 : 7F00 0001 0303 771F 0000 0000 4500 0044 ......w.....E..D
0020 : 3D14 0000 FF11 8092 7F00 0001 7F00 0001 =...............
0030 : F11E 0035 0030 0000 4713 0100 0001 0000 ...5.0..G.......
0040 : 0000 0000 0131 0130 0130 0331 .....1.0.0.1

ICMP_Dest_Unreachable[Port] < 127.0.0.1 [localhost] > 127.0.0.1 [localhost] sp=61734 dp=53 [domain] seq=0x00300000 sz=76(+20)
0000 : 4500 004C 3D25 0000 FF01 8075 7F00 0001 E..L=%.....u....
0010 : 7F00 0001 0303 7715 0000 0000 4500 0044 ......w.....E..D
0020 : 3D24 0000 FF11 8082 7F00 0001 7F00 0001 =$..............
0030 : F126 0035 0030 0000 4715 0100 0001 0000 .&.5.0..G.......
0040 : 0000 0000 0131 0130 0130 0331 .....1.0.0.1

ICMP_Dest_Unreachable[Port] < 127.0.0.1 [localhost] > 127.0.0.1 [localhost] sp=61742 dp=53 [domain] seq=0x00300000 sz=76(+20)
0000 : 4500 004C 3D35 0000 FF01 8065 7F00 0001 E..L=5.....e....
0010 : 7F00 0001 0303 770B 0000 0000 4500 0044 ......w.....E..D
0020 : 3D34 0000 FF11 8072 7F00 0001 7F00 0001 =4.....r........
0030 : F12E 0035 0030 0000 4717 0100 0001 0000 ...5.0..G.......
0040 : 0000 0000 0131 0130 0130 0331 .....1.0.0.1

[Michael Steele_2]

Have you grep'd for those ports in /etc/services?

Using 'lsof' is also an excellent suggestion.

http://gatekeep.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.77/

[Dave Olker]

Since the source port numbers are changing rapidly, I highly doubt you're going to find them in the /etc/services file unless the process just happens to get an anonymous port assignment that "belongs" to a well-known service in /etc/services (i.e. there is nothing that stops one program from grabbing the port number of someone else's port assigned in /etc/services as long as it is not currently in use).

Do the changing port numbers mean this is multiple processes? Not necessarily. It could be a single process that is either launching threads to do the DNS queries or a process that calls fork/vfork to create a child process to do the DNS query.

In any case, I still recommend using lsof to try to determine who is sending these DNS requests to local port 53.

I'm not an lsof power user, so someone else might have a better idea for syntax, but looking at the lsof(8) man page, I think you'd want to use the "-i" and "-P" options to have lsof gather information about open IPV4 "network files" (i.e. connections) and to print their numerical names instead of a resolved name from /etc/services. I'd suggest collecting a few lsof runs at the same time as the icmpinfo command is gathering the ICMP Dest Unreachable messages.

Once you have a few lsof runs collected during the same time as some ICMP errors are logged you will hopefully be able to search the lsof output for the source port numbers in the ICMP messages and figure out which process is sending to port 53.

Good luck, and please let us know if you're able to identify the offending process.

Dave

[Michael Steele_2]

Does this sound like a denial of service attack to anyone else? The only difference is its coming off of you're machine and looping back. If /etc/services won't work then get lsof and locate the culprit process via the ports in use.

[Bob_Vance]

Is 'named' running on this host?
This is a note from the README:

----------------
IMPORTANT NOTE : *If* you machine is running a named (is a name
server) you MUST USE -n when leaving icmpinfo unattended, to avoid
some possible looping, when icmpinfo tries to resolve an ip in an
unreachable domain. [this problem of domain generated icmps has been
worked upon... but I found no satisfary solutions, suggestions welcome,
maybe a solution for v1.12 !]
-----------------

hth
bv