topic Re: Node hanging when shutdown in Operating System - HP-UX

Node hanging when shutdown

AnthonySN — Sun, 03 Jun 2007 07:56:27 GMT

hi,
We are implementing a two node cluster on itanium servers (HP-UX 11i v2) for oracle 10g RAC using SGeRAC.
problem is when we try to shutdown one node the oracle instance is shifting from that node to the other but the node goes in a hang status, attached a srn sht of it and also cluster log and config file.

Re: Node hanging when shutdown

skt_skt — Sun, 03 Jun 2007 08:35:53 GMT

Is that two separate boxes or two partitions on the same hardware?

We had a little similar problem on 10g with 11.11 Check the cluster patch level(get HP if possible).

What do you see in the chassis logs when it happens? What you see in the crs logs for both nodes?

Re: Node hanging when shutdown

AnthonySN — Sun, 03 Jun 2007 11:11:21 GMT

there are two separate servers accessing one common storage (MSA30)
attached is the crs log (both servers have the sam info).
Latest quality pack (0307) is installed.

Re: Node hanging when shutdown

Mridul Shrivastava — Sun, 03 Jun 2007 20:38:40 GMT

I got a feeling that you have executed the shutdown before halting the package and node... since package did not halt successfully..

Stopping resources.
Error while stopping resources. Possible cause: CRSD is down.
Stopping CSSD.
Unable to communicate with the CSS daemon.

So this could be a reason of system hang since it was trying to halt the package and package halt script did not run successfully as some of the services were already stopped by the time halt script run.

Re: Node hanging when shutdown

AnthonySN — Sun, 03 Jun 2007 23:56:57 GMT

Hi,
We are testing a live case scenario where something happens to the node suddenly and shuts down, in this case there would no time to halt the package and node.
also if u see the attached scrn shot it is hanging while stopping the highly available cluster.

Re: Node hanging when shutdown

AnthonySN — Sun, 03 Jun 2007 23:57:59 GMT

Re: Node hanging when shutdown

Mridul Shrivastava — Sun, 03 Jun 2007 23:59:16 GMT

In that scenario this node will panic.. so that you could try testing by removing the heartbeat lan or killing clusterdaemon or killing monitoring script or my be by removing the power cable ....

the best way would be to kill cmcld / monitoring daemon and check

Re: Node hanging when shutdown

AnthonySN — Mon, 04 Jun 2007 02:11:48 GMT

ok,
but the problem is we are not able to even halt the cluster it says halting and remains stuck

Re: Node hanging when shutdown

Mridul Shrivastava — Mon, 04 Jun 2007 02:23:37 GMT

what's the version of the MCSg installed on the box.. What are the error message in syslog.log ?? If there is none then check the flight recorder logs located under /var/adm/cmclsuter ( It will be there only if Sg version is 11.15 or above)..

How long it takes and how do you notice that it is hung ... what steps do u take to overcome this situation?

If possible then do a TOC while system is hung so there will be crashdump generated and you may be able to find out where halt process got hung.

Re: Node hanging when shutdown

AnthonySN — Mon, 04 Jun 2007 08:16:42 GMT

SGe Rac version installed is 11.17.
am attachin the fr logs also.
When we kill the cluster deamon then the node proceeds to shutdown properly.

Re: Node hanging when shutdown

Mridul Shrivastava — Mon, 04 Jun 2007 23:39:53 GMT

I have checked the flight recorder logs and did not find anything wrong.

system is shuting down properly after killing the cmcld or it panics after doing so?? If cluster is up and we kill cmcld on one node then it should panic and generate the crashdump??

If this is not the case then can you try halting the node while cluster is up and If it hangs then TOC it.

You could also try the ways mentioned earlier to test the package failover and let me know the outcome..

Re: Node hanging when shutdown

AnthonySN — Sun, 10 Jun 2007 02:41:42 GMT

Hi,
it seems we need to apply two patches for MC/SG. will let you know the status after applying the patches.

Re: Node hanging when shutdown

AnthonySN — Tue, 02 Oct 2007 08:58:13 GMT

thanks