Node hanging when shutdown

[AnthonySN]

hi,
We are implementing a two node cluster on itanium servers (HP-UX 11i v2) for oracle 10g RAC using SGeRAC.
problem is when we try to shutdown one node the oracle instance is shifting from that node to the other but the node goes in a hang status, attached a srn sht of it and also cluster log and config file.

[skt_skt]

Is that two separate boxes or two partitions on the same hardware?

We had a little similar problem on 10g with 11.11 Check the cluster patch level(get HP if possible).

What do you see in the chassis logs when it happens? What you see in the crs logs for both nodes?

[AnthonySN]

there are two separate servers accessing one common storage (MSA30)
attached is the crs log (both servers have the sam info).
Latest quality pack (0307) is installed.

[Mridul Shrivastava]

I got a feeling that you have executed the shutdown before halting the package and node... since package did not halt successfully..

Stopping resources.
Error while stopping resources. Possible cause: CRSD is down.
Stopping CSSD.
Unable to communicate with the CSS daemon.

So this could be a reason of system hang since it was trying to halt the package and package halt script did not run successfully as some of the services were already stopped by the time halt script run.

[AnthonySN]

Hi,
We are testing a live case scenario where something happens to the node suddenly and shuts down, in this case there would no time to halt the package and node.
also if u see the attached scrn shot it is hanging while stopping the highly available cluster.

[AnthonySN]

Hi,
We are testing a live case scenario where something happens to the node suddenly and shuts down, in this case there would no time to halt the package and node.
also if u see the attached scrn shot it is hanging while stopping the highly available cluster.

[Mridul Shrivastava]

In that scenario this node will panic.. so that you could try testing by removing the heartbeat lan or killing clusterdaemon or killing monitoring script or my be by removing the power cable ....

the best way would be to kill cmcld / monitoring daemon and check

[AnthonySN]

ok,
but the problem is we are not able to even halt the cluster it says halting and remains stuck

[Mridul Shrivastava]

what's the version of the MCSg installed on the box.. What are the error message in syslog.log ?? If there is none then check the flight recorder logs located under /var/adm/cmclsuter ( It will be there only if Sg version is 11.15 or above)..

How long it takes and how do you notice that it is hung ... what steps do u take to overcome this situation?

If possible then do a TOC while system is hung so there will be crashdump generated and you may be able to find out where halt process got hung.

[AnthonySN]

SGe Rac version installed is 11.17.
am attachin the fr logs also.
When we kill the cluster deamon then the node proceeds to shutdown properly.

[Mridul Shrivastava]

I have checked the flight recorder logs and did not find anything wrong.

system is shuting down properly after killing the cmcld or it panics after doing so?? If cluster is up and we kill cmcld on one node then it should panic and generate the crashdump??

If this is not the case then can you try halting the node while cluster is up and If it hangs then TOC it.

You could also try the ways mentioned earlier to test the package failover and let me know the outcome..

[AnthonySN]

Hi,
it seems we need to apply two patches for MC/SG. will let you know the status after applying the patches.

[AnthonySN]

thanks