11.0, Unresponsive system, root login hangs via MP

[imrichar]

Had a pingable "alive" system that stopped performing it duties, ssh was unresponsive, monitoring agents ceased to function, etc. Logged in via the MP and received console, logged in as root and it just hung, was never prompted for the password. Waited a considerable amount of time with no encouraging signs, so I issued a RST from the MP. System came back up clean, but now I'm at a loss for what really happened. Syslog shows normal activity, and then just stops. New syslog picks up at the time of reboot. Filesystems have adequate free space, no crash logs, tombstones clean, etc. Any thoughts for finding the root cause?

[Torsten. · Accepted solution]

Maybe too late now.

If you reset the system via "TC" from GSP it will write a dump that can be analyzed, but "RS" will simply rest the system.

If there is nothing in the (old) logs you can maybe inspect the GSP logs.

[melvyn burnard]

I have seen this happen for a number of reasons, one of them being that /var and/or /var/tmp are full.
You may want to check on how much disc space is free in these directories.

[imrichar]

Nothing of significance in the MP logs.

/var was my first thought actually, but my monitoring agent shows that it was ~ 29% free before the system become unresponsive.

Load, memory utilization, and process count were all within normal ranges as well.

[melvyn burnard]

Then as Torsten stated, a crash dump might have helped. Simply doing a Reset wipes any trace that may have been there.

If it happens again, do a TOC and get the dump analysed.

[imrichar]

In the future, how much space will the dump take? There is 7 gig of memory in this system and /var (I'm assuming the dump will reside here) is ~ 1.6 gig free. Will it just dump the kernel?

[melvyn burnard]

depends how your system is set up to take a dump.
You need to know you have enough dump space, and also what it will dump.
Use crashconf -v to see what pages will get dumped and an idea of how much space will be needed.

[imrichar]

Thanks for your help, at least I can be better prepared in the future. :)