topic Hung Server - kdump enabled - Can Dump be forced on Reboot? in Operating System - Linux

Hung Server - kdump enabled - Can Dump be forced on Reboot?

Alzhy — Wed, 11 Aug 2010 14:10:16 GMT

We've a new SmartIron RHEL 5.5 server.

It just suddenly hung.

Would it be possible to capture kernel dump on next boot?

TIA.

Re: Hung Server - kdump enabled - Can Dump be forced on Reboot?

Viktor Balogh — Thu, 12 Aug 2010 12:55:15 GMT

Hi,

Here is a Red Hat specific article:

http://magazine.redhat.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/

You can either choose diskdump or netdump.

Regards,
Viktor

Re: Hung Server - kdump enabled - Can Dump be forced on Reboot?

Alzhy — Thu, 12 Aug 2010 23:26:28 GMT

Let's make it simple:

Linux Hangs
Poor Admin has no choice but to reset/Post the Poor X86 Box.
On other "platforms" - one can TOC so on next reboot -- it forces a dump.

Is there something similar in Linux?

Re: Hung Server - kdump enabled - Can Dump be forced on Reboot?

Viktor Balogh — Fri, 13 Aug 2010 14:05:02 GMT

Hi Alzhy,

on a reboot AFAIK you can't force a dump, but from the os you can. Here is the procedure for kdump, which we use on SuSE:

1.) edit menu.lst, add a crashkernel parameter like this:

crashkernel=64M@16M

2.) install kexec-tools and kdump

3.) enable sysrq in case it is turned off:

# echo 1 > /proc/sys/kernel/sysrq

4.) you should have a kernel compiled with sysrq support, (CONFIG_MAGIC_SYSRQ=yes)
Check it if it is enabled in /proc/config or /proc/config.gz

5.) with the following command you can trigger an instant reboot with a crashdump:

# sync; sync; echo c > /proc/sysrq-trigger

or you can also trigger it with the following key combination:

Alt+SysRq(PrintScreen)+C

6.) you'll find the dump in /var/log/dump

hope this helps,
Viktor

Re: Hung Server - kdump enabled - Can Dump be forced on Reboot?

Matti_Kurkela — Mon, 16 Aug 2010 09:50:33 GMT

As far asI understand, you have the dump mechanism configured; you just need a way to trigger it manually now.

The Alt+SysRq+C key combination (as mentioned by Viktor) on the system console might work even if the console is otherwise hung.

The rest is hardware dependent. Some servers have a button which can be used to force a NMI interrupt: I think that can be used to trigger a Linux crash dump in some situations. Others have configurable "watchdog" hardware features (like ASR on HP Proliant hardware) which can automatically trigger a crash reboot if the userspace is hung for more than a pre-determined time... but that wouldn't help you in your current situation because it must be configured in advance.

I'm not at all familiar with SmartIron hardware.

MK