/etc/resolv.conf file always be changed automatically

[diaoxin]

Hi Horia ,
I can not manage it .
I have got a file generated from tcpdump , but there are 415700 lines in the file .I will try to analyse it .

DiaoXin

[diaoxin]

Hi experts,
The tcpdump log file is so big and so many destination IP mentioned , it is very difficult for me to analyse .
Can you give me some ideas?

Thanks!

DiaoXin

[Horia Chirculescu]

You should check all the servers making connections to your server. You have no choice but to manually parse all the recordings from the tcpdump output.

Check what is happening exactly at 5:00 AM.

Look at the source IP address, then connect to the server that have this IP address.

Best regards
Horia.

[TTr]

You need to use grep and grep -v to clean out the tcpdump file from all the lines that you know are not from suspect servers or to non suspect ports.

On another thought is this an oracle server? Oracle can do scheduling similar to cron.

Also search for suid-root files in the server under oracle or under any other subdirectory. Oracle is known to use suid files but it could be anywhere.

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1015047

[diaoxin]

Hi TTr,
There is no oracle installed on this server.

DiaoXin

[diaoxin]

Hi Horia,
But the file is too long which has 415700 lines ,it is very difficult to find the suspect IP from it. But i will try .

DiaoXin

[Dennis Handly]

>it is very difficult to find the suspect IP from it.

That's why you need to do some scripting to isolate the IPs and possibly use "sort -u".

[Alzhy]

One last crack from me...

Check if there is a cfengine process (cfexecd) or do a find on ll processes that start with "cf" or even check your filesystems for any occurence of cfengine

Note, HP as their own cfengine implementation -- I just cannot remember what exactly it is...

[diaoxin]

Hi Alzhy,
Thank you! I checked the processes runing on the server ,but really find anyone like you mentioned. Please find the information as below.

# ps -ef |grep exec
root 219 178 0 10:18:57 pts/3 0:00 grep exec
root 23191 1 0 Jul 6 ? 2:36 /opt/novadigm/radexecd
# ps -ef |grep cf
root 262 178 0 10:19:36 pts/3 0:00 grep cf

DiaoXin

[diaoxin]

Hi Dennis,

I use " sort -u dump.log > dumpnewlog " , but the new log file is the same as the original log .

DiaoXin

[Viktor Balogh]

>I use " sort -u dump.log > dumpnewlog " , but the new log file is the same as the original log .

Show us some lines from your logs, I can write you a one-liner which only prints the relevant information to you.

[Alzhy]

Diaoxin

BINGO!

You have CFENGINE indeed -- albeit the Commercial Version - NOVA!

So there you go -- chase who is the champion behind it sir.

What gave it away is /etc/resolv.conf "
compliance" is actually one of the example "Promises" any CFENGINE HOWTO gives away!

Case Closed.

[Alzhy]

Diaoxin,

Scratch that previous post.. NOVA still installs in /var/cfengine... so that's not it...

But I suggest to search your filesystems fo any incidence of cfengine* cfexec* -- etc.

[TTr]

If you have lsof installed in your server, run this command in cron at 5am

while true
do
ps -ef `lsof /etc/resolv.conf` >> /tmp/lsof-output.log
done

If you are on the server run it at 4:59:55 and kill it at about 5:00:30. Otherwise you need to schedule it in cron and also kill it after one minute.

Note that it might or it might not catch the /etc/resolv.conf file when it is opened but it is worth a try.

[Viktor Balogh]

>root 23191 1 0 Jul 6 ? 2:36 /opt/novadigm/radexecd

this is the Radia remote execution daemon, I think it has nothing to do with cfengine...

[Dennis Handly]

>but the new log file is the same as the original log.

Most likely due to the fact each line is timestamped. You need to grep for certain strings or sort on certain fields, using -k.

[diaoxin]

Hi Alzhy,

I am really sorry , that I type the wrong word . I really find no one process related to cfengine . The process I mentioned above as below which is called " /opt/novadigm/radexecd " is a tool called RADIA which can collect some information such as filesystem usage . And the tool installed on all unix servers (about 60 servers),and have been running for some years . But this problem just happened 2 monthes before.

# ps -ef |grep exec
root 219 178 0 10:18:57 pts/3 0:00 grep exec
root 23191 1 0 Jul 6 ? 2:36 /opt/novadigm/radexecd


Thank you!
diaoxin

[diaoxin]

Hi Viktor Balogh,
Please find some lines of tcpdump log file as below .Thank you!

04:59:00.303284 IP 150.236.34.226.1023 > 150.236.34.229.2049: tcp 140
04:59:00.303481 IP 150.236.28.140.54631 > 150.236.34.226.57840: UDP, length 148
04:59:00.303487 IP 150.236.34.226.57840 > 150.236.28.140.54631: UDP, length 216
04:59:00.303491 IP 150.236.28.140.54631 > 150.236.34.226.57840: UDP, length 148
04:59:00.303496 IP 150.236.34.226.57840 > 150.236.28.140.54631: UDP, length 216
04:59:00.303501 IP 150.236.34.229.2049 > 150.236.34.226.1023: tcp 124
04:59:00.303504 IP 150.236.34.226.1023 > 150.236.34.229.2049: tcp 160
04:59:00.303509 IP 150.236.28.140.54631 > 150.236.34.226.57840: UDP, length 148
04:59:00.303513 IP 150.236.34.226.57840 > 150.236.28.140.54631: UDP, length 216
04:59:00.303517 IP 150.236.28.140.54631 > 150.236.34.226.57840: UDP, length 148

diaoxin

[diaoxin]

Hi Viktor Balogn,
You are right ! it is Radia .

diaoxin

[diaoxin]

Hi TTr and Dennis and Viktor Balogh,

I sort the tcpdump log file using "sort -k " "sort -u " "awk " , and so far , the original file has been filtered from 415700 lines to more than 1000 lines .
So , can I believe that all the IP mentioned in the file are all have the posibility to change /etc/resolv.conf file in the server ?


diaoxin

[diaoxin]

Hi experts,
I filter the tcpdump log file again and so far it only has more than 300 lines . But i have question , shall I ignore the lines whose source IP is this server's IP ? and I only analyse the lines whose dest IP is this server's IP ?

diaoxin

[Viktor Balogh]

Yes, you can ignore the lines, where the source IP is your servers own address. And try to filter the output to list only the 4:59-5:01 timeframe (if you didn't do it so already)

From the remaining lines, you could list only the ip addresses, and do a sort -u (uniq) to get a full list of ip addresses which machines were connected to your server at the given timeframe. Maybe this gives you a clue, or at least it will be cleared on which other servers should you investigate this further.

[Alzhy]

diaosin.

you can also try cavring a script and running it during the window of suspicion. script should be written this way.

while true;do
fuser /etc/resolv.conf
if there is a HIT
get PID of attached process
seek out processname
display process name
exit
end if
end loop | tee /pathlarge/hunter.log

An endless fast loop so it has better chances of catching the culprit... Then you can have anothere job in cron to kill the process ata certain time.

Long shot but I used the same trick in a similar situation (although not the same).

Good LUCK!

[Jim Walls]

diaoxin,

I have looked through this entire thread and nobody has suggested looking in the cron log.

Check /var/adm/cron/log to see if any jobs - which could also be "at" jobs - run at 5am.

Do you use ssh for inward connections to the server? Connection and login messages should be present in one of the syslogs.

grep sshd /var/adm/syslog/*

Also; check the ~/.ssh/Authorized_keys files for all users (especially for root) to see if any of the entries have forced commands. These entries may also give you some clues to who has ssh access to the server.

[diaoxin]

Hi Viktor Balogh,
So far , I filtered the tcpdump file to not more than 100 lines .
I will try to analyse.

Thanks.
diaoxin