RedHat freeze

[Jean-Christophe Valiere]

Hello,

On Saturday a RedHat 2.1AS server freeze. We couldn't access the machine from ssh or localy so we reboot it. Now I'm wondering what happens as it the second times it happens. In /var/log/messages I get regularly messages from IPTABLE (all input is logged), but when the machine freeze I get no more message anywhere. Does somebody know where I should look at or what should I do to see where the probleme come from ??
PS: The server is a DL-380 G3 with Hyper-Threading.

Thanks,

[Steven E. Protter]

I got the very same behavior on a ES 3.0 server on Saturday, interestingly enough.

The only possible issue with that server was the var filesystem was on its way to getting full.

The behavior was the same, no messages, no nothing. I'd check bugzilla for this issue becasue mine is a Dell Poweredge and has few smililarities to yours in the hardware realm.

Interesting.

SEP

[Don_89]

Here is a good article on troubleshooting this..

http://www.linuxdevcenter.com/pub/a/linux/2001/11/01/postmortem.html

Since Steve had the same issue, maybe it was an up2date package. I checked my /var/log/up2date.1 and didn't see anything being updated.. Could be different for you since it depends on what packages are installed..

[Jean-Christophe Valiere]

What is curious is that hpasm is running and, from one second to the other the Insight Console change the state of the server from "OK" to "not accessible".
Is it possible that the Hyper-Threading could have some problem with linux ?
I will take a look at bugzilla too.

[Don_89]

I've had nothing but problems with version 7.0 of the HPASM drivers including automatic system reboots (ASR's). We went back to 6.40 and never had a problem since on 20+ servers.

[Jean-Christophe Valiere]

As I see you are using hpasm I will ask one just last question about ASR: Do you let it enabled on your servers ??
I disable it because some of my servers are sunnning Oracle and they can't be rebooted when they want.

Thanks for all information.

[Jan Sladky]

hi Jean,
in this situation I would consider allowing to log all messages by syslog:
/etc/syslog.conf

*.* -/var/log/allmessages

maybe you will find there the cause for next freezing (I hope it will not come again, but for certainty)

br Jan

[Jean-Christophe Valiere]

Thanks,

I will try that too.

[Darrin St. Amant]

RH does not suggest because of performance using Hyper-Threding turned on for RH2.1

could be the culprit.

cheers!

[Vitaly Karasik_1]

Darrin, can you please point me to this RH recommendation?

10x, Vitaly

[Jean-Christophe Valiere]

I am interrested in too.

[David Sullivan_7]

One of the best ways to troubleshoot complete system freezes is to use the "NMI watchdog" feature. Your machine needs to either be SMP or have an enabled APIC (if uniprocessor). If you're running the SMP kernel with a single Xeon, that works, too.

To enable it, append "nmi_watchdog=1" to the boot line in GRUB, or add an 'append="nmi_watchdog=1' if you're using LILO. Once this is done, reboot and you're all set.

The NMI watchdog generates interrupts periodically that will cause the kernel to panic if the processor freezes. You can then use the aforementioned debugging tools to find out what is going on. I used this recently to detect system freezes caused by an errant driver.

Hope this helps!

[Jean-Christophe Valiere]

Many Thanks,

While talking about that, I wanted to stop hpasm on a server (running about still 70 days), and the /etc/init.d/hpasm stop failed for the 2 following processus: cevtd and casmd. They are now totally lost in the space and take each 100% of a CPU. As I have 4 the server is still up but really unhappy to work so much for nothing ;).
BTW I think Ingight Manager 7 got very big trouble... Where should I make some feedback to HP ??