Mysterious RedHat Hangs...

[Lee Harris_5]

Hi, I was just wondering if anyone else has seen a similar problem to this. We have three servers running RHEL-AS-3 update 3. All of them run an Oracle 9 database and all of them occasionally (about every one or two weeks) completely stop responding to logins.

If we try to ssh to the servers, it just hangs. They do respond to pings still. If we connect to the iLO we aren't even able to login at the console.

The only way to recover the server is to power cycle the box. There is nothing obvious showing the /var/log/messages or dmesg on any other boxes. There's nothing showing up in the HP System Managemet Homepage (by the way these are Proliant DL380s).

The only thing I can see, is that all of the boxes report that they only have around 20Meg of free physical RAM when running the "free" command. Not sure if this could be causing the issues.

Anyone seen anything like this before or have any ideas?

Regards - Lee

[Ivan Ferreira]

This seems to be a problem to open a call. You should enable the magic sysrq key and try to force a memory dump.

When I had a similar problem I configured a remote syslog server because the system was hang and cannot write to disk, but was able to send the message over the network and more infor was obtained to troubleshoot the problem.

System hangs can be caused by a kernel module o driver, but without any message is very hard to identify the source of the problem.

[Steven E. Protter]

Shalom Lee,

I've seen this behavior with sudden kernel stops.

Some of the times I've seen it I was forced to upgrade the kernel.

Other times, I was forced to tone downt he agressiveness with which my iptables firewall was updating itself (with the help of snort) to deal with hack attempts.

It depends on many factors what the actual cause is. Since these are Oracle Servers, I'd check and see if there are new Oracle patches or required OS patches for Oracle that might need to be installed.

Regards,

SEP

[Vitaly Karasik_1]

I'll suggest you to update you RHEL3 to the latest (7) update.
May be make sense to use HP-recommended NIC drivers.

As for "20Meg of free physical RAM when running the "free" command"

it's OK,

see for example my answer http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1029490

[Alan_152]

Any messages on the stderr and stout consoles prior to you rebooting?

[George Liu_4]

It may have various causes. It could be file system lost due to kernel bug.
Strongly suggest to test different kernels.