topic Re: '/var/log/messages' automatically becomes directory in Operating System - Linux

'/var/log/messages' automatically becomes directory

Maaz — Tue, 29 Sep 2009 10:43:26 GMT

SLES 10 SP2 x864
syslog-ng-1.6.8-20.18
openssh-4.2p1-18.36

this machine is running behind the firewall(i.e on lan, with a single NIC, having private IP), and no Internet access is allowed from this machine.
this machine cant be accessible from Internet

we are running SSH, and VNC services on this machine.

Our firewall is sending its log to this machine(syslog-ng is accepting logs from our firewall).

Problem:
this is second time, instead of a file, we found that there is an empty directory named '/var/log/messages'... and obviously I am not able to check the logs.

I simply delete the '/var/log/messages' directory, and then restart the syslog daemon(rcsyslog restart), and then a new '/var/log/messages' created... and now I can check the logs send be the firewall.

and some strange/additional/non-default empty directories are there too(in /var/log), e.g 'all.log', 'auth.log', 'everything.log', 'messages.log' 'and 'secure'.. and all these directories are owned by root.

what might be the problem ?
is it a virus issue ?(no Anti-Virus installed)
or kind of attack ?
what should I do ? and what to check ?

as I told, its the second time, I noticed this issue.

Regards
Maaz

Re: '/var/log/messages' automatically becomes directory

Steven E. Protter — Tue, 29 Sep 2009 11:06:06 GMT

Shalom Maaz,

I suspect bad software or a bad script.

Every seen this happen, but this has all the marks of human error.

Look for clues such as last access or permissions in this newly created folder.

SEP

Re: '/var/log/messages' automatically becomes directory

Maaz — Tue, 29 Sep 2009 11:27:02 GMT

I ran the 'ps ax' and got the following strange processes (almost 1206 lines output of following lines)

ps aux

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 1359 0.0 0.0 232 208 ? S 02:40 0:00 ps wax
root 1360 0.0 0.0 232 208 ? S 02:40 0:00 ps wax
root 1545 0.0 0.0 232 208 ? S 02:46 0:00 ps wax
root 1546 0.0 0.0 232 208 ? S 02:46 0:00 ps wax
root 1683 0.0 0.0 232 208 ? S 02:50 0:00 grep irq
root 1685 0.0 0.0 232 208 ? S 02:50 0:00 grep irq

.
.
.
root 31424 0.0 0.0 232 208 ? S Sep28 0:00 ps wax
root 31431 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31436 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31441 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31446 0.0 0.0 232 208 ? S Sep28 0:00 ps wax
root 31452 0.0 0.0 232 208 ? S Sep28 0:00 ps wax
root 31458 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31858 0.0 11.2 473044 447564 ? SN Sep06 5:35 /usr/sbin/snmpd -r -A -LF d /var/log/net-snmpd.log -p /var/run/snmpd.pid
root 31952 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31955 0.0 0.0 232 208 ? S Sep28 0:00 ps wax
root 31962 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31967 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31970 0.0 0.0 232 208 ? S Sep28 0:00 ps wax
root 32096 0.0 0.0 42280 2964 ? Ss 17:31 0:00 sshd: root@pts/6
root 32106 0.0 0.0 232 208 ? S 17:31 0:00 grep irq
root 32173 0.0 0.0 13104 2452 pts/6 Ss+ 17:31 0:00 -bash
root 32666 0.0 0.0 232 208 ? S 01:58 0:00 ps wax
root 32667 0.0 0.0 232 208 ? S 01:58 0:00 ps wax

attached is the output of 'ps aux'

Re: '/var/log/messages' automatically becomes directory

Maaz — Tue, 29 Sep 2009 11:33:57 GMT

Hi thanks SEP for reply

>Look for clues such as last access or permissions in this newly created folder.

drwxr-xr-x 2 root root 48 Sep 26 19:00 all.log
drwxr-xr-x 2 root root 48 Sep 26 19:00 auth.log
drwxr-xr-x 2 root root 48 Sep 27 07:09 everything.log
drwxr-xr-x 2 root root 48 Sep 26 19:00 secure

since I myself has access these directories(to check whats inside) thats why last access is reporting the time when I 'ls' the directory e.g

# stat everything.log/
File: `everything.log/'
Size: 48 Blocks: 0 IO Block: 4096 directory
Device: 6803h/26627d Inode: 233648 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2009-09-29 17:15:21.000000000 +0500
Modify: 2009-09-27 07:09:42.000000000 +0500
Change: 2009-09-27 07:09:42.000000000 +0500

# stat secure/
File: `secure/'
Size: 48 Blocks: 0 IO Block: 4096 directory
Device: 6803h/26627d Inode: 65 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2009-09-29 17:15:30.000000000 +0500
Modify: 2009-09-26 19:00:40.000000000 +0500
Change: 2009-09-26 19:00:40.000000000 +0500

but 'Modify' and 'Change' time is different.

Re: '/var/log/messages' automatically becomes directory

kobylka — Tue, 29 Sep 2009 12:03:05 GMT

Hello Maaz!

What is the parent of all those

grep irq
ps wax

?

You could install another syslog-ng version and use it with the same config file. If the problem persists, check config file for possible create_dirs options, macro expansions in filepaths, etc.

Kind regards,

Kobylka

Re: '/var/log/messages' automatically becomes directory

Maaz — Tue, 29 Sep 2009 13:47:13 GMT

>What is the parent of all those

>grep irq
>ps wax

>?
how can I tell you the parent of the process
as there is no 'pstree' command on this machine

# ls /usr/bin/pstree
/bin/ls: /usr/bin/pstree: No such file or directory

# rpm -q psmisc
psmisc-22.1-14.2

# rpm -ql psmisc
/bin/fuser
/usr/bin/killall
/usr/bin/oldfuser
/usr/bin/pstree
/usr/bin/pstree.x11

>You could install another syslog-ng version >and use it with the same config file. If the >problem persists, check config file for >possible create_dirs options, macro >expansions in filepaths, etc

syslog-ng.conf attached

Re: '/var/log/messages' automatically becomes directory

Maaz — Tue, 29 Sep 2009 14:00:12 GMT

I reinstall the 'psmisc-22.1-14.2.x86_64.rpm'

rpm -Uvh --force psmisc-22.1-14.2.x86_64.rpm

and now I have pstree command.

attached is pstree output

Re: '/var/log/messages' automatically becomes directory

kobylka — Tue, 29 Sep 2009 15:03:20 GMT

Hello Maaz!

Well, indeed the problem is configuration specific:

syslog-ng logs 24 kinds of facility codes (facility is a type of message, depending on where it comes from). The problem you are experimenting seems to be related to your syslog-ng not filtering out explicitly those facilities, therefore creating files you would not expect (nothing prevents your log sources from generating messages for any facilities).

Overcome this by creating definitions for all the facilities and marking create_dirs global option to no.

Here is a list of facilities syslog-ng handles:

http://www.balabit.com/dl/html/syslog-ng-v2.0-guide-admin-en.html/ch09s04.html

And here an example of simple definitions:

http://linux.cudeso.be/linuxdoc/syslog-ng.php

Add to your syslog-ng.conf those facilities you do not already have defined and you should be done with the problem.

About the

grep irq
ps wax

problem what is really needed is the ppid of any (if possible all) of those processes. This is to see who created them and where this process is. You should be able to use

ps -el

to see the ppid column of a process.

Kind regards,

Kobylka

Re: '/var/log/messages' automatically becomes directory

Maaz — Thu, 01 Oct 2009 03:54:39 GMT

Hi Thanks kobylka for help

>Well, indeed the problem is configuration specific:
Ok, I am going to learn/understand the syslog-ng.

>About the
>grep irq
>ps wax
>problem what is really needed is the ppid of any (if possible all) of those
>processes. This is to see who created them and where this process is. You should be
>able to use
>ps -el
>to see the ppid column of a process.

Attached is the output of "ps aux" and "ps -el"

Thanks
Regards
needee

Re: '/var/log/messages' automatically becomes directory

Rob Leadbeater — Thu, 01 Oct 2009 04:59:00 GMT

Hi,

I'll guess that some sort of badly written monitoring script is running from the cron.

Check out the cron logs, if they're still there, to see what could have caused things.

Cheers,

Rob

Re: '/var/log/messages' automatically becomes directory

Matti_Kurkela — Thu, 01 Oct 2009 05:57:41 GMT

The fact that so many standard logfiles have been changed into directories is suspicious, but it *might* be a configuration problem with logrotate or similar utility.

But looking at your process listing I noticed that "ps aux" lists many processes as "ps wax" and "grep irq", the "ps -el" lists those same processes as running "tblockd". This is most definitely not normal!

Google does not find any significant hits on "tblockd", so it is not likely to be a normal part of the system.

I also see "pure-ftpd", multiple SSH connections as root, and various processes related to Xen virtualization on this host.

In light of this, the fact that log files are changing into directories becomes very suspicious too. Maybe someone does not want the logs to be there?

I'd say you have been attacked with some significant degree of success: an intruder seems to have root access to your system!

The intruder is probably running some software that tries to mask the intruder's processes running on the system, and has been only partially successful.
These types of software are generally known as "root kits" and are purposefully made to resist removal.

Take backups of all important data on the system *NOW* but don't overwrite any old backups: you may need them too.

Be prepared to re-install the entire operating system: it is the only way to be absolutely sure that all the intruder's malware is gone.

If you wish to analyze what has been done, boot the system with some Linux Live-CD and use it to examine the filesystems or backup them for forensic purposes: the system's own kernel can not be trusted to be truthful any more.

The attack may have come from the internal network too: if someone has brought in a laptop or other machine that was already infected with a worm program, it may have been able to automatically attack your server. Once the machine has been contaminated somehow, an outgoing connection to the Internet is enough to allow an intruder to remotely control it as a part of a botnet.

Maybe your server is behind the firewall now, but was it always so? Was the server installed for its current role, or was it re-purposed without reinstallation? If so, was it less well protected before?

MK

Re: '/var/log/messages' automatically becomes directory

Maaz — Tue, 06 Oct 2009 03:22:49 GMT

Hi Rob Leadbeater
>I'll guess that some sort of badly written monitoring script is running from the
>cron.
>Check out the cron logs, if they're still there, to see what could have caused
>things.
No Sir, no such problem

Hi Matti Kurkela
>logfiles have been changed into directories is suspicious, but it *might* be a
>configuration problem with logrotate or
>similar utility.
No configuration issue ..

>But looking at your process listing I noticed that "ps aux" lists many
>processes as "ps wax" and "grep irq", the "ps -el" lists those same processes
>as running "tblockd". This is most definitely not normal!

>I'd say you have been attacked with some significant degree of success: an
>intruder seems to have root access to your system!
Your guess is absoloutely right, I rebuild the system 3 days before, and found no issue yet.. no "ps wax" and "ps irq" process running now.
This machine is again accepting logs from another machine(a linux box this time).. and yet not found any problem(like changing of important log files into directories)

Thanks Forum for help and support.

Regards
Maaz