'/var/log/messages' automatically becomes directory

[Maaz]

SLES 10 SP2 x864
syslog-ng-1.6.8-20.18
openssh-4.2p1-18.36

this machine is running behind the firewall(i.e on lan, with a single NIC, having private IP), and no Internet access is allowed from this machine.
this machine cant be accessible from Internet

we are running SSH, and VNC services on this machine.

Our firewall is sending its log to this machine(syslog-ng is accepting logs from our firewall).

Problem:
this is second time, instead of a file, we found that there is an empty directory named '/var/log/messages'... and obviously I am not able to check the logs.

I simply delete the '/var/log/messages' directory, and then restart the syslog daemon(rcsyslog restart), and then a new '/var/log/messages' created... and now I can check the logs send be the firewall.

and some strange/additional/non-default empty directories are there too(in /var/log), e.g 'all.log', 'auth.log', 'everything.log', 'messages.log' 'and 'secure'.. and all these directories are owned by root.

what might be the problem ?
is it a virus issue ?(no Anti-Virus installed)
or kind of attack ?
what should I do ? and what to check ?

as I told, its the second time, I noticed this issue.


Regards
Maaz

[Steven E. Protter]

Shalom Maaz,

I suspect bad software or a bad script.

Every seen this happen, but this has all the marks of human error.

Look for clues such as last access or permissions in this newly created folder.

SEP

[Maaz]

I ran the 'ps ax' and got the following strange processes (almost 1206 lines output of following lines)


ps aux

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 1359 0.0 0.0 232 208 ? S 02:40 0:00 ps wax
root 1360 0.0 0.0 232 208 ? S 02:40 0:00 ps wax
root 1545 0.0 0.0 232 208 ? S 02:46 0:00 ps wax
root 1546 0.0 0.0 232 208 ? S 02:46 0:00 ps wax
root 1683 0.0 0.0 232 208 ? S 02:50 0:00 grep irq
root 1685 0.0 0.0 232 208 ? S 02:50 0:00 grep irq

.
.
.
root 31424 0.0 0.0 232 208 ? S Sep28 0:00 ps wax
root 31431 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31436 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31441 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31446 0.0 0.0 232 208 ? S Sep28 0:00 ps wax
root 31452 0.0 0.0 232 208 ? S Sep28 0:00 ps wax
root 31458 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31858 0.0 11.2 473044 447564 ? SN Sep06 5:35 /usr/sbin/snmpd -r -A -LF d /var/log/net-snmpd.log -p /var/run/snmpd.pid
root 31952 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31955 0.0 0.0 232 208 ? S Sep28 0:00 ps wax
root 31962 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31967 0.0 0.0 232 208 ? S Sep28 0:00 grep irq
root 31970 0.0 0.0 232 208 ? S Sep28 0:00 ps wax
root 32096 0.0 0.0 42280 2964 ? Ss 17:31 0:00 sshd: root@pts/6
root 32106 0.0 0.0 232 208 ? S 17:31 0:00 grep irq
root 32173 0.0 0.0 13104 2452 pts/6 Ss+ 17:31 0:00 -bash
root 32666 0.0 0.0 232 208 ? S 01:58 0:00 ps wax
root 32667 0.0 0.0 232 208 ? S 01:58 0:00 ps wax


attached is the output of 'ps aux'

[Maaz]

Hi thanks SEP for reply

>Look for clues such as last access or permissions in this newly created folder.

drwxr-xr-x 2 root root 48 Sep 26 19:00 all.log
drwxr-xr-x 2 root root 48 Sep 26 19:00 auth.log
drwxr-xr-x 2 root root 48 Sep 27 07:09 everything.log
drwxr-xr-x 2 root root 48 Sep 26 19:00 secure


since I myself has access these directories(to check whats inside) thats why last access is reporting the time when I 'ls' the directory e.g

# stat everything.log/
File: `everything.log/'
Size: 48 Blocks: 0 IO Block: 4096 directory
Device: 6803h/26627d Inode: 233648 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2009-09-29 17:15:21.000000000 +0500
Modify: 2009-09-27 07:09:42.000000000 +0500
Change: 2009-09-27 07:09:42.000000000 +0500

# stat secure/
File: `secure/'
Size: 48 Blocks: 0 IO Block: 4096 directory
Device: 6803h/26627d Inode: 65 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2009-09-29 17:15:30.000000000 +0500
Modify: 2009-09-26 19:00:40.000000000 +0500
Change: 2009-09-26 19:00:40.000000000 +0500


but 'Modify' and 'Change' time is different.

[kobylka]

Hello Maaz!


What is the parent of all those

grep irq
ps wax

?


You could install another syslog-ng version and use it with the same config file. If the problem persists, check config file for possible create_dirs options, macro expansions in filepaths, etc.


Kind regards,

Kobylka

[Maaz]

>What is the parent of all those

>grep irq
>ps wax

>?
how can I tell you the parent of the process
as there is no 'pstree' command on this machine

# ls /usr/bin/pstree
/bin/ls: /usr/bin/pstree: No such file or directory

# rpm -q psmisc
psmisc-22.1-14.2

# rpm -ql psmisc
/bin/fuser
/usr/bin/killall
/usr/bin/oldfuser
/usr/bin/pstree
/usr/bin/pstree.x11


>You could install another syslog-ng version >and use it with the same config file. If the >problem persists, check config file for >possible create_dirs options, macro >expansions in filepaths, etc

syslog-ng.conf attached

[Maaz]

I reinstall the 'psmisc-22.1-14.2.x86_64.rpm'

rpm -Uvh --force psmisc-22.1-14.2.x86_64.rpm

and now I have pstree command.

attached is pstree output

[kobylka]

Hello Maaz!


Well, indeed the problem is configuration specific:

syslog-ng logs 24 kinds of facility codes (facility is a type of message, depending on where it comes from). The problem you are experimenting seems to be related to your syslog-ng not filtering out explicitly those facilities, therefore creating files you would not expect (nothing prevents your log sources from generating messages for any facilities).

Overcome this by creating definitions for all the facilities and marking create_dirs global option to no.

Here is a list of facilities syslog-ng handles:

http://www.balabit.com/dl/html/syslog-ng-v2.0-guide-admin-en.html/ch09s04.html

And here an example of simple definitions:

http://linux.cudeso.be/linuxdoc/syslog-ng.php

Add to your syslog-ng.conf those facilities you do not already have defined and you should be done with the problem.


About the

grep irq
ps wax

problem what is really needed is the ppid of any (if possible all) of those processes. This is to see who created them and where this process is. You should be able to use

ps -el

to see the ppid column of a process.


Kind regards,

Kobylka

[Maaz]

Hi Thanks kobylka for help

>Well, indeed the problem is configuration specific:
Ok, I am going to learn/understand the syslog-ng.

>About the
>grep irq
>ps wax
>problem what is really needed is the ppid of any (if possible all) of those
>processes. This is to see who created them and where this process is. You should be
>able to use
>ps -el
>to see the ppid column of a process.

Attached is the output of "ps aux" and "ps -el"

Thanks
Regards
needee

[Rob Leadbeater]

Hi,

I'll guess that some sort of badly written monitoring script is running from the cron.

Check out the cron logs, if they're still there, to see what could have caused things.

Cheers,

Rob