topic Re: Linux server hang in Operating System - Linux

Linux server hang

tom quach_1 — Wed, 06 Jul 2005 17:12:13 GMT

Hi All,
I have proliant 380 with 2cpu 3.2GH and 4Gb Ram.
OS RedHatlinux Enterprise version 3.0
my linux server for some reason, keeps hanging on me i could ping the IP but that was about it, had to do a hard reboot. Last month when it hung, checked in /var/log/messages and found some web cache error. So that was a known issues with oracle. applied the patch and it was ok for a month. Now it happens again, check in the file messages, this time nothing in there.
My question:
Where else can i check, which log file ?
How to troubleshoot this issue.

Regards,
Tom
?

Re: Linux server hang

Bejoy C Alias — Wed, 06 Jul 2005 23:54:55 GMT

Is there any hdd activity ? Is the server heavily loaded ? Try appending the load average value to a file for further monitoring by puting 'cat /proc/loadavg' in ur cron . R u able to get login to any other prompts like telnet , ftp , rlogin etc. ?

Re: Linux server hang

tom quach_1 — Thu, 07 Jul 2005 01:02:50 GMT

Thanks Bejoy for the reply.
the server was still running. The only thing i could do was pinging the ip address. The Console, telnet , ftp nothing were working. And have to do a hard reboot.
any other suggestions.

Regards,
Tom

Re: Linux server hang

dirk dierickx — Thu, 07 Jul 2005 01:06:45 GMT

use sar to see if there is a performance problem on the machine, heavy swapping could be a problem.

Re: Linux server hang

Ivan Ferreira — Thu, 07 Jul 2005 01:43:41 GMT

You can enable the magic sysrq in /etc/sysctl.conf and use special key combinations to generate a memory dump, sync the disks and reboot the server when it freezes.

/etc/sysctl.conf

kernel.sysrq = 1

See the kernel documentation linux/Documentation/sysrq.txt.

Re: Linux server hang

Gopi Sekar — Thu, 07 Jul 2005 01:44:27 GMT

if your system is not heavily loaded then the next best thing to check for is memory.

Also ensure that you have applied all the latest patches and updates related to kernel

Re: Linux server hang

Bejoy C Alias — Thu, 07 Jul 2005 04:45:28 GMT

Check the values of load average . If there is no load , then u may have a failing hdd . Check for any hdd activities while the system is hanging.

Re: Linux server hang

Andrea Rossi — Thu, 07 Jul 2005 07:28:01 GMT

did you try to disable all unecessary modules, first of all hp agents?

Andrea

Re: Linux server hang

Tvs — Thu, 18 Aug 2005 06:26:33 GMT

hi.

any web server (apache) server is running in u r system...?

normally a mis-configured configuration file of apache can make the system slow.

try to decrease the no:of processor setting in /etc/httpd/conf/httpd.conf

Re: Linux server hang

Vitaly Karasik_1 — Thu, 18 Aug 2005 07:04:50 GMT

Do you already have the latest (5) update for RHEL3?

Re: Linux server hang

Durgaprasad Ala — Thu, 18 Aug 2005 13:20:57 GMT

I faced this issue on couple of servers running update 3. Actually this is partial kernel panic and couple of hours after system will reboot itself(if you leave the server in the partial panic state).

I fixed it going to Update 4 and
now you can go to update 5

The bug was with kswapd

Re: Linux server hang

Yipeng CHEN — Mon, 19 Sep 2005 09:07:02 GMT

Tom,
Can you please post your solution to share. Thanks.

We have a Production database server running Oracle 9.2.0.4 instance under RHEL3 Update 4.
The server experienced hang on problem, that is:
"The server was still running. The only thing i could do was pinging the ip address. The Console, telnet , ftp nothing were working. And have to do a hard reboot."

From Durgaprasad note, it seems that Kswap bug was fixed in Update 4? WHY our server still experienced this HANG ON issue?
Tom, appreciate that if you can share any light for us. Thanks.

Re: Linux server hang

tom quach_1 — Wed, 21 Sep 2005 12:41:54 GMT

Sorry Yipeng,
I do not have a solution for my RedHat server problem. Because it is a low priority in our ENV and do not have time to fix it, so i sill do a hard reboot when it happens.

Thanks,
Tom

Re: Linux server hang

tom quach_1 — Wed, 21 Sep 2005 12:45:27 GMT

Thank you all,

Tom

Re: Linux server hang

Yipeng CHEN — Wed, 21 Sep 2005 12:53:38 GMT

Thanks, Tom.
Do someone have this fix experiences? Please share with us. Thanks.

Re: Linux server hang

Anindra — Thu, 19 Oct 2006 04:48:54 GMT

Tom,

1. Is the server a NFS client? If yes, could you pls post here your /etc/fstab.

2. Is it running any Veritas software? If yes, what and which version?

3. Search a line "kernel_sysrq = 0" in /etc/sysctl.conf and replace the 0 with 1. Then run "sysctl -p". This will basically throw more debug messages into the /var/log/messages file to provide you some clues.

4. You could also add the following to /etc/syslog.conf:-

*.debug;mail.none;authpriv.none;cron.none /var/log/messages

Again, it will provide you more debugging messages in /var/log/messages.

Regards,
Anindra