topic one node hung reason in Operating System - HP-UX

one node hung reason

Shivkumar — Fri, 16 Jun 2006 14:25:41 GMT

Hi,

We have a 2 node serviceguard cluster running oracle 9i rac. Our sys admin team found one node in hung state
and rebooted the server manually. I think that the hung server could have gone for TOC and rebooted itself.

Can someone explain what could be this hung situation and why it had not gone for crash dump and rebooting ?

Thanks,
Shiv

Re: one node hung reason

Torsten. — Fri, 16 Jun 2006 14:41:07 GMT

Not every "hang" is the same.

You can feel the system is hung, but the system doesn't - one scenario.

The system is hang and will crash - even in non-sg environment.

Your control script is not working as expected.

And some more...

You need to analyze the logs to get the reason.

Re: one node hung reason

Deoncia Grayson_1 — Fri, 16 Jun 2006 15:47:09 GMT

The node could have panic for any reason as Torsten stated, you have to investigate the logs to find exactly why. You can look at the /etc/shutdownlog and also look in your syslogs, you also check /var/adm/crash to see if created a crash log.

Deonia

Re: one node hung reason

Bill Hassell — Fri, 16 Jun 2006 19:55:19 GMT

Thyere is no reason to assume that the hang condition will cause a TOC/panic. You have to determine the state of the system. When you say 'hung', is that being measured with a network connection? The system may be running fine but a network router has failed so the link between your device (like a PC) is broken. When the system hangs, you must go to the console so you can bypass the networking. If you can get logged in, you'll have to determine if local networking is still working.

If the console login fails, there may be a hardware failure and your cluster failover is not correctly setup.

Re: one node hung reason

inventsekar_1 — Sat, 17 Jun 2006 11:19:28 GMT

i have a question SHiv,
how ur sys admins team found one node in hung state? is there any command?
i am curious to know that..

and what is meant by "crash" in unix?
sometimes my windows system crashed and when we install windows again we missed only the c drive datas. what will happen in unix crash? when we install unix os again, what datas we can get or recover?

Re: one node hung reason

Shivkumar — Sat, 17 Jun 2006 11:43:37 GMT

Sekar,
Someone miscommunicated to us that it was in hung state. In fact the system panicked and rebooted itself. So it seems to be an expected behaviour.

Thanks,
Shiv

Re: one node hung reason

Bill Hassell — Sat, 17 Jun 2006 12:05:21 GMT

Actually, a system panic is not a 'normal' state. Your systems need up to date patches. It is not unusual for HP-UX systems to run for years and never have a panic and reboot. But these systems should be fully patched ever 4 to 6 months.

Re: one node hung reason

Michael Steele_2 — Sat, 17 Jun 2006 16:23:58 GMT

A panic and reboot is not the same as a 'hung' system. See Mr. Hassell's description.

You should investigate your logs files:

/var/adm/tombstones/ts98, ts99 (* HPMC's?*)
OLDsyslog.log (* Read from bottom up *)

SERVICE GUARD
cmreadlog /var/opt/cmon/cmomd.g
cmreadlog /var/opt/sgmgr/929917sgmgr.log
cmscancl -n node -o outputfile

SAVECRASH
/var/adm/crash/*

GSP > sl > e