/etc/cron.daily/00-logwatch hangs

[kenny chia]

My RHEL4 update 3 server 00-logwatch cron.daily job runs everyday at 4am

But yesterday I did a process check and found that this cron job hanged on May 25th. On May 26th to May29 yesterday, additional 00-logwatch cron job was lauched at 4am everyday and they all hanged. Please see attached for the process list

I tried to kill the processes but I could not kill the /bin/cat /var/log/messages. these processes are in lock_p and their state os 'D'

4 D root 19907 19906 0 76 0 - 12475 lock_p May25 ? 00:00:00 /bin/cat /var/log/message
4 D root 19907 19906 0 76 0 - 12475 lock_p May26 ? 00:00:00 /bin/cat /var/log/message
4 D root 19907 19906 0 76 0 - 12475 lock_p May27 ? 00:00:00 /bin/cat /var/log/message
4 D root 19907 19906 0 76 0 - 12475 lock_p May28 ? 00:00:00 /bin/cat /var/log/message
4 D root 19907 19906 0 76 0 - 12475 lock_p May29 ? 00:00:00 /bin/cat /var/log/message

What could have caused this problem?

[Stuart Browne]

You want to kill off the following PID's, as:

kill -TERM 7712 19895 14244 490 30698

You want to then check the 'size' of your '/var/log/messages' file.

You also want to check the available space in your '/tmp' directory:

df -vh /tmp

I'm picking the messages file has gotten large or your root/tmp file system is filling up.

[kenny chia]

Could a huge message file caused the problem? Mine is at least 180MB per file

-rw------- 1 root root 17717388 May 30 11:21 /var/log/messages
-rw------- 1 root root 2915121 May 29 19:29 /var/log/messages.1
-rw------- 1 root root 187376019 May 20 04:02 /var/log/messages.2
-rw------- 1 root root 187346970 May 13 04:02 /var/log/messages.3
-rw------- 1 root root 187306013 May 6 04:02 /var/log/messages.4

[Stuart Browne]

Well logwatch will combine all of those files (so about 700MB all up), then analyse the results.

It can get a bit out of control if you keep large logs :)