/etc/resolv.conf file always be changed automatically

[diaoxin]

Hi TTr,
Sorry for that and thank you for your idea.

I checked the server and there is no .rhosts file existing .

DiaoXin

[TTr]

Check if there are any ftp or ssh connections at 5 am in the syslog (or ftp log if it logs in its own logs)

[diaoxin]

Hi TTr,

I really find there is one user account ssh to the server at 4:00am everyday , and I will try to discuss with the user to deny his ssh connection for test.

Thank you!
DiaoXin

[diaoxin]

Hi TTr,
I discussed with the user about his ssh connection. It is a cronjob for him to collect some information from the server and he only use the normal user permission to do this .
So I think it is not the root cause.

DiaoXin

[Alzhy]

I would bet there is a CFENGINE scheme running. It could be HP's version (forgot the name) or cfengine itself.

Check /var/cfengine.

[TTr]

If you have tcpdump installed, run tcpdump from 4:59am to 5:01am to see who is connecting to the server and at what port. This will tell you if the resolv.conf file is overwritten by an external process or not.

Verify the permissions of the file. It should be owned by root and have 644 (-rw-r--r--) permission mode. Therefore whatever is changing it is running as root.

Check if there is a batch scheduling tool installed other than cron.

Check if you have sudo installed and which users are allowed to run commands via sudo.

[diaoxin]

Hi Alzhy,

But I can not find /var/cfengine in the server.

DiaoXin

[diaoxin]

Hi TTr,
tcpdump is not installed .
I checked the permission of /etc/resolv.conf , it is 644 . and we use sudo in the server , but I find some users in sudoers file and I can not delete them because I am not sure whether they are necessary for some applications .
For the batch scheduler not cronjob , sorry I don't know how to check it in the server . Can you give me any ideas?
Diaoxin

[Anshumali]

Check for root's login with last -R command and see from where there was an attempt to login.
Use HIDS to find out whats happening.
Tcpdump in that time is a good idea.
If its non-prod or if you can afford, take the network down for the said period and see if it happens just to isolate the cause being from n/w or local.
Think when the issue started and if you get a date/time, see what was changed if your change control is good.

[Horia Chirculescu]

>tcpdump is not installed .

You can install it from:

http://hpux.connect.org.uk/hppd/hpux/Networking/Admin/tcpdump-4.1.1/

Best regards,
Horia.

[diaoxin]

Hi Anshumali,
The " last " command can not works ,it shows the error "Invalid record size. Unable to continue ...".


diaoxin

[Horia Chirculescu]

>"Invalid record size. Unable to continue

This means that you have a corrupted wtmp file.

You must do this:

cat /dev/null > /var/adm/wtmp

Horia.

[Horia Chirculescu]

Maybe in 11.23 you have to truncate both /var/adm/wtmps and /var/adm/btmps files

Just figure out which wtmp you have by running ls on /var/adm.

Horia.

[diaoxin]

Hi Horia,
I run the commands as below,

server# cat /dev/null > /var/adm/wtmps
server# last

WTMPS_FILE begins at Thu Jan 1 07:59:59


it works.

diaoxin

[Horia Chirculescu]

You must truncate also btmps in order to keep the system information consistent.

Horia.

[diaoxin]

Hi Horia,
I will try to install tcpdump .

Thanks.
diaoxin

[diaoxin]

Hi Horia,
Ok , I did it .and "last " shows the information now.

diaoxin

[diaoxin]

Hi,
I installed tcpdump in the server , and tried to generate a log file whose contents as below ,
15:22:21.299732 IP 150.236.34.226.1023 > 150.236.34.229.2049: tcp 160
15:22:21.299930 IP 150.236.34.229.2049 > 150.236.34.226.1023: tcp 120
15:22:21.300020 IP 150.236.34.226.1023 > 150.236.34.229.2049: tcp 172
15:22:21.300248 IP 150.236.34.229.2049 > 150.236.34.226.1023: tcp 120
But I have no ideas how to analyse it . Can you help adivse?

diaoxin

[Horia Chirculescu]

As TTr said,

>If you have tcpdump installed, run tcpdump from 4:59am to 5:01am to see who is connecting to the server and at what port. This will tell you if the resolv.conf file is overwritten by an external process or not.

You could start tcpdump by cron at 4:59 and monitor the TCP connections to/from your server.

You should add the following 2 lines into crontab for root user (assuming you have tcpdump in standard place: /usr/sbin, you should change that if inappropriate):

59 4 * * * /usr/sbin/tcpdump -w /root/tcpdump.txt

01 5 * * * /root/kill-tcpdump.sh


Where kill-tcpdump.sh looks something like this:

#!/bin/sh

PID=`ps -e |grep tcpdump |awk '{print $1}'`

kill -9 $PID
########## Done ##########


Next day you should check of course the file /root/tcpdump.txt. You could also, increase verbosity by adding -vv to tcpdump in crontab like in tcpdump -vv -w /root/tcpdump.txt


Best regards,
Horia.

[Horia Chirculescu]

15:22:21.299732 IP 150.236.34.226.1023 > 150.236.34.229.2049: tcp 160

The format is like this:

date Source_IP_Address.Source_PortNo Dest_IP_Address.Dest_PortNo protocol size


So, at this time: 15:22:21.299732, the machine having IP address 150.236.34.226 is making a TCP connection from port 1023 to machine having IP: 150.236.34.229 to port 2049

Read manual page at:

http://www.tcpdump.org/tcpdump_man.html

Which IP address is the one of your server?

In the interval of time TTr suggested, you should check the connections made from other servers to yours.

Analyze those server's logs and try to understand why those are needed to make connections to your server.


Best regards from Romania,
Horia.

[diaoxin]

Hi TTr and Horia ,
Thank you so much !
I have added the script to crontab , and will check it tomorrow morning.

DiaoXin

[Alzhy]

So no cfengine in your system sir? Better double check - it is the only plausible explanation to your woes as that /etc/resolv.conf persistence is one f the classic examples that are usually adpated by cfengine implementers.

HTH.

[Dennis Handly]

>Horia: You must truncate also btmps in order to keep the system information consistent.

Not really, unless you want to match up successful logins and ignore typing mistakes. This is one file you shouldn't truncate, since it has the bad logins.

[diaoxin]

Hi HTH ,
But the server really doesn't have cfengine. I cannot find "cfengine" under /opt , and can not find it from "swlist " output.

diaoxin

[Horia Chirculescu]

@Dennis: I assumed too much (that lastb is not working also).

When I was talking about consistency I was in fact thinking about keeping both information (all/successful and bad login) from a common start point (in time).

DiaoXin: Did you manage to take a decision about connections made to/from your server in the specified time frame?

Horia.