Find process that uses DNS

[H.Merijn Brand (procura]

# lsof -r1 -i:53
or
# lsof -r1 -i:56730

leave gaps of approx 1 sec.

What I want is to find the process that does a DNS query source port 56730 on HP-UX 11.00 dest port 53 to a non-existing client about 4 times a minute. I know this happens from both the DNS log, and ethereal started on linux (which showed the ports)

Somehow I've not (yet) been able to track down the process that does these calls.

named does not run anymore.

/pro/local/share/elvis/elvis.ini

[Mark Greene_1]

telnetd will do dns calls. Maybe nfs, or various other rpc type of calls will, but I'm just guessing there. Also, check the xntp config to see if you've specified the server by name or by IP.

mark

[Stefan Farrelly]

if you look at the manpage for audevent you can see all the calls that would be logged if you swtiched auditing on, but I dont think your dns query is among them, so I dont think auditing will help. The next thing is to use nettl to switch on network tracing, then use netfmt to filter out what you are looking for. If you run it only for a minute you shoudnt get too much output to search through but im sure you will find it.

[John Poff]

Just to follow up on Stefan's comment, the latest versions of Ethereal will work with the output from the HP-UX nettl command, so you can read those files using Ethereal on your Linux box.

JP

[H.Merijn Brand (procura]

telnet(d) won't run while booting, and ethereal finds theses calls already during the boot process. I've already put in some time stamps, but I cannot reboot the server as much as I'd like :)

samba's nmbd doesn't run either.

xntpd uses IP based server address, and that dead server that is searched for has never been a time server.

NFS has een cleaned up completely, and that server is not in any list anymore.

nettl looks more promising, but it reports a process id that in not actively running, so it's probably spawned, and dies. Now if it would report it's parent process id :)

No bunnies yet, because the main question is not to detect the action - I already did that - but I want to know what process is doing it. My quest is thus for the best generic way to track back network access, and not start looking down from all running processes (sorry Mark)

John, ethereal does not show the process that is attached to the action, which is what I want to know. You could get a bunny if you tell me the underdocumented option/function/setting that *does* show the process

Enjoy, have FUN! H.Merijn

[Ron Kinner]

What is the name of the non-existant client? Perhaps that might jog someone's memory.

Ron

[Jeff Schussele]

Hi Merijn,

Other likely culprits I can think of would be

EMS
SNMP
SendMail
OpenView
SD

Rgds,
Jeff

[Jeff Schussele]

I forgot to ask if the host/IP being resolved gives you any clue i.e. is it a mail relay, an SNMP collection host, EMS alarm server. etc?

Rgds,
Jeff

[Geoff Wild]

What about installing an Intrusion Detection System?

Something like Snort?

http://www.snort.org/



Rgds...Geoff

[H.Merijn Brand (procura]

name = "k1.procura.nl", a 9000/K250 which that is uhhh dead, dead as a doornail, as in dismanteled and scrapped :) only connected to our LAN

/etc # grep -w -i k1 *

only gives 'mount', 'named', 'smtpd', 'wtmp', (all program binaries) and several entries in termcap. None of those is a likely candidate to use that 'k1' as a value to do a dns for.

No OpenView, EMS, or SD active. (send)mail related stuff is IIRC in /etc/mail, and

/etc/mail# grep -r -i -w k1 .

gives nothing, and neither does

/var/adm/syslog# grep -i -w k1 *

so I guess it's not a mail forwarding problem. I'm strenghtened in this by seeing that both /var/mail and /var/spool/mqueue are empty

netfmt is rather puzzling for me. Even after removing the files and restarting nettl, it keeps telling me process ID 973 is the process that tries to access k1. THERE IS NO PID 973! Grrrrrr. netfmt -v -N doesn't show me much more than ethereal, except that PID, which does not exist.

Enjoy, have FUN! H.Merijn

[Jeff Schussele]

Hi Merijn,

Any automount stuff still referenced?

ll /etc/auto*

Jeff

[H.Merijn Brand (procura]

autofs has never ever been enabled on any of our systems. I hate that system :)

the log only shows that DHCP is disabled. Yeah!

Might need snort. Does it indeed give more info than ethereal and/or nettl/netfmt?

Enjoy, have FUN! H.Merijn

[Massimo Bianchi]

Just my thoughts:

- any printer ?
lpstat -t | grep k1

- any scheduled job from cron ?

- Whar was the k1 used for ?

- is there any dynamic dhcp-dns the-bunny-knows software on the HPUX11.0 that can do some polling ? Apart from the mentioned EMS, Openview, ....

Massimo

[John Poff]

Massimo has a good point. Do you have any remote printers defined?

JP

[Jeff Schussele]

Has to be in a file somewhere

Try

find /etc /var /opt /usr -exec grep -l k1 {} \;

Problem is it might be in a non-ascii text file. I'm trying to compose a find/exec using strings. Stand by.

Oh be advised that the above will hammer the box for a while, so choose your time appropriately or break it up to a single FS at a time.

Rgds,
Jeff

[Massimo Bianchi]

writing a simple script that does a:

while true
do
UNIX95= ps -efH
done > /tmp/PIDs.txt

(to show process hierarchy)

and let it run for 5 minutes, that cross the information with the output from lsof.


In a try or two you should get the bad guy :)

Massimo

[Geoff Wild]

Snort gives you info like this:

#0-(1-176519) [arachNIDS][snort] ICMP PING CyberKit 2.2 Windows 2003-08-28 23:29:54 64.116.168.91 64.114.228.60 ICMP
#1-(1-176518) [snort] ICMP Destination Unreachable (Communication Administratively Prohibited) 2003-08-28 23:26:52 134.156.101.250 64.114.228.131 ICMP
#2-(1-176517) [arachNIDS][snort] ICMP PING CyberKit 2.2 Windows 2003-08-28 23:23:55 64.114.77.135 64.114.228.60 ICMP
#3-(1-176516) [arachNIDS][snort] ICMP PING CyberKit 2.2 Windows 2003-08-28 23:23:52 64.114.225.28 64.114.228.60 ICMP
#4-(1-176515) [snort] ICMP Destination Unreachable (Communication Administratively Prohibited) 2003-08-28 23:23:32 65.119.179.14 64.114.228.60 ICMP
#5-(1-176514) [arachNIDS][snort] ICMP PING CyberKit 2.2 Windows 2003-08-28 23:21:35 64.114.77.193 64.114.228.60 ICMP
#6-(1-176513) [arachNIDS][snort] ICMP PING CyberKit 2.2 Windows 2003-08-28 23:21:17 64.114.0.141 64.114.228.60 ICMP
#7-(1-176512) [arachNIDS][snort] ICMP PING CyberKit 2.2 Windows 2003-08-28 23:21:06 64.115.22.163 64.114.228.60 ICMP
#8-(1-176511) [arachNIDS][snort] ICMP PING CyberKit 2.2 Windows 2003-08-28 23:20:34 64.115.1.118 64.114.228.60 ICMP
#9-(1-176510) [arachNIDS][snort] ICMP PING CyberKit 2.2 Windows 2003-08-28 23:20:25 64.114.178.84 64.114.228.60 ICMP
#10-(1-176509) [arachNIDS][snort] ICMP PING CyberKit 2.2 Windows 2003-08-28 23:20:12 64.115.72.178 64.114.228.60 ICMP
#11-(1-176508) [arachNIDS][snort] ICMP PING CyberKit 2.2 Windows 2003-08-28 23:19:45 64.114.227.10 64.114.228.60 ICMP
#12-(1-176507) [arachNIDS][snort] ICMP PING CyberKit 2.2 Windows 2003-08-28 23:19:36 64.113.213.84 64.114.228.60 ICMP
#13-(1-176506) [arachNIDS][snort] ICMP PING CyberKit 2.2 Windows 2003-08-28 23:19:05 64.114.100.4 64.114.228.60 ICMP
#14-(1-176505) [arachNIDS][snort] ICMP PING CyberKit 2.2 Windows 2003-08-28 23:18:37 64.115.21.86 64.114.228.60 ICMP


I use it with ACID (Analysis Console for Intrusion Databases ) on my linux box.


It uses Apache, php, mysql - I don't see any guides for HP-UX - but I don't see why it wouldn't work....

Takes a bit of time to setup...

Rgds...Geoff

[Mark Greene_1]

According to iana, port 53 is used for encrypted IP:

http://www.iana.org/assignments/protocol-numbers

I couldn't find 56730 on the list of known used ports:

http://www.chebucto.ns.ca/~rakerman/port-table.html#Table

Are you running ssl or IPv6?

mark

[Jeff Schussele]

Hi Mark,

Port 53 TCP/UDP is DNS - has been for as long as I can remember.

Those high ports are negotiated on the fly.

Cheers,
Jeff

[H.Merijn Brand (procura]

a. No printers. Since this was my first hunch too, I've checked that extensively, and all printers are^Wwere either local or jetdirect.
b. No cron.
c. k1 was used for almost everything you can think off, but remember the we do not have any web service like stuff here, so it has never done any apache or such. It was a 10.20 box when it died.
d. dynamic? well most likely, but WHAT? No dhcp, that has never been enabled on any HP-UX box.
e. I tried
/# find var opt etc usr -type f | xargs grep -i -w -l k1
and mainly found binaries :(
[ Yes, grep -r -w -l -i k1 /opt /var /etc will run forever ]
f. Since lsof does not report anything at all about using that port, the combo with ps will not yield any info either
g. No IPV6
h. I do have ssl enabled on this box, but not during boot. There is no sshd active (yet), and all ssl/ssh activity on this box started way after the k1 died
i. Keep the suggestions coming.

Enjoy, have FUN! H.Merijn

[Massimo Bianchi]

it is enabled on boot.....
what is there in the /etc/inittab ??

Massimo

[John Poff]

Ok, here is what I would try. Since your k1 box is dead and gone, I would bring up a minimal Linux box with the hostname of k1 and using the same IP address that k1 had. Next, I would write a Perl script on the Linux k1 box to listen to port 53 and try to catch it that way. If your Perl script answers whatever process is looking for it, maybe the process on the other end will hang out long enough for you to find it with Glance, lsof, etc. Maybe seeing what response you get on port 53 will give some good clues also.

JP

[Ron Kinner]

Just verifying the situation. Something send a DNS query to the usual DNS server asking for the IP of k1. Is that correct?

And it always uses source port 56730? (That seems quite strange).

Bind is not running on this box is it?

Wonder what would happen if you put an entry for k1 in /etc/hosts and gave it the IP address of the 11.0? At least it would stop asking the DNS server for the address. Perhaps the process would establish a connection which you could see in netstat -an.

Ron

[H.Merijn Brand (procura]

Ron, statement 1 is correct

nettl shows that, as does ethereal. The communication is

1. UDP l1:56730 -> adsl:53
Standard query A k1.procura.nl
2. UDP adsl:53 -> l1:56730
Standard query response, No such name
3. UDP l1:56731 -> adsl:53
Standard query A k1
4. UDP adsl:53 -> l1:56731
Standard query response, No such name

Giving the k1 the IP of the 11.00 (either l1 or a5) would cause trouble for processes that actually need to communicate with the real machine. I like the idea though and will try to boot up an old Win server and give it the IP of the k1

Enjoy, have FUN! H.Merijn

[Jeff Schussele]

Well, in that vein, why don't you just add the k1 hostname as an alias for another system? Maybe even the system that's requesting it. Just add it into the hosts file on itself & make sure files is first in nswitch.conf

Lot easier than firing another system up & setting an IP for it.

My $0.02,
Jeff