Linux server hang

[tom quach_1]

Hi All,
I have proliant 380 with 2cpu 3.2GH and 4Gb Ram.
OS RedHatlinux Enterprise version 3.0
my linux server for some reason, keeps hanging on me i could ping the IP but that was about it, had to do a hard reboot. Last month when it hung, checked in /var/log/messages and found some web cache error. So that was a known issues with oracle. applied the patch and it was ok for a month. Now it happens again, check in the file messages, this time nothing in there.
My question:
Where else can i check, which log file ?
How to troubleshoot this issue.

Regards,
Tom
?

[Bejoy C Alias]

Is there any hdd activity ? Is the server heavily loaded ? Try appending the load average value to a file for further monitoring by puting 'cat /proc/loadavg' in ur cron . R u able to get login to any other prompts like telnet , ftp , rlogin etc. ?

[tom quach_1]

Thanks Bejoy for the reply.
the server was still running. The only thing i could do was pinging the ip address. The Console, telnet , ftp nothing were working. And have to do a hard reboot.
any other suggestions.

Regards,
Tom

[dirk dierickx]

use sar to see if there is a performance problem on the machine, heavy swapping could be a problem.

[Ivan Ferreira]

You can enable the magic sysrq in /etc/sysctl.conf and use special key combinations to generate a memory dump, sync the disks and reboot the server when it freezes.

/etc/sysctl.conf

kernel.sysrq = 1

See the kernel documentation linux/Documentation/sysrq.txt.

[Gopi Sekar]

if your system is not heavily loaded then the next best thing to check for is memory.

Also ensure that you have applied all the latest patches and updates related to kernel

[Bejoy C Alias]

Check the values of load average . If there is no load , then u may have a failing hdd . Check for any hdd activities while the system is hanging.

[Andrea Rossi]

did you try to disable all unecessary modules, first of all hp agents?

Andrea

[Tvs]

hi.

any web server (apache) server is running in u r system...?

normally a mis-configured configuration file of apache can make the system slow.

try to decrease the no:of processor setting in /etc/httpd/conf/httpd.conf

[Vitaly Karasik_1]

Do you already have the latest (5) update for RHEL3?

[Durgaprasad Ala]

I faced this issue on couple of servers running update 3. Actually this is partial kernel panic and couple of hours after system will reboot itself(if you leave the server in the partial panic state).

I fixed it going to Update 4 and
now you can go to update 5

The bug was with kswapd

[Yipeng CHEN]

Tom,
Can you please post your solution to share. Thanks.

We have a Production database server running Oracle 9.2.0.4 instance under RHEL3 Update 4.
The server experienced hang on problem, that is:
"The server was still running. The only thing i could do was pinging the ip address. The Console, telnet , ftp nothing were working. And have to do a hard reboot."

From Durgaprasad note, it seems that Kswap bug was fixed in Update 4? WHY our server still experienced this HANG ON issue?
Tom, appreciate that if you can share any light for us. Thanks.

[tom quach_1]

Sorry Yipeng,
I do not have a solution for my RedHat server problem. Because it is a low priority in our ENV and do not have time to fix it, so i sill do a hard reboot when it happens.

Thanks,
Tom

[tom quach_1]

Thank you all,

Tom

[Yipeng CHEN]

Thanks, Tom.
Do someone have this fix experiences? Please share with us. Thanks.

[Anindra]

Tom,

1. Is the server a NFS client? If yes, could you pls post here your /etc/fstab.

2. Is it running any Veritas software? If yes, what and which version?

3. Search a line "kernel_sysrq = 0" in /etc/sysctl.conf and replace the 0 with 1. Then run "sysctl -p". This will basically throw more debug messages into the /var/log/messages file to provide you some clues.

4. You could also add the following to /etc/syslog.conf:-

*.debug;mail.none;authpriv.none;cron.none /var/log/messages

Again, it will provide you more debugging messages in /var/log/messages.

Regards,
Anindra