Serviceguard hangs up the system

[Srinikalyan]

Hi,
We are facing some strange problem in starting the serviceguard RAC cluster. When we start the cluster, both the nodes hangs up and being shuts down. So we started the system by login into console and started the vpars, then again the system hung. I found the following error in the syslog:
Aug 18 13:40:35 testsrv vmunix: GAB WARNING V-15-1-20115 Port d registration failed, GAB not configured
Aug 18 13:40:35 testsrv vmunix: vxgms: GAB_API_REGISTER error=261
Aug 18 13:40:35 testsrv vmunix: ODM WARNING V-41-6-5 odm_gms_api_start_msgs fails
Aug 18 13:40:36 testsrv inetd[2984]: registrar/tcp: Connection from localhost (127.0.0.1) at Mon Aug 18 13:40:36 2008
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3019]: Setting STREAMS-HEAD high water value to 131072.
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3021]: nfsd do_one mpctl succeeded: ncpus = 1.
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3021]: nfsd do_one pmap 2
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3021]: nfsd do_one pmap 3
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3021]: nfsd do_one bind 0
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3021]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3022]: nfsd 0 0 sock 4
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3027]: nfsd 0 1 sock 4
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3031]: nfsd 0 2 sock 4
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3033]: nfsd 0 3 sock 4
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3037]: nfsd 0 4 sock 4
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3041]: nfsd 0 5 sock 4
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3044]: nfsd 0 6 sock 4
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3046]: nfsd 0 7 sock 4
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3048]: nfsd 0 8 sock 4
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3050]: nfsd 0 9 sock 4
Aug 18 13:40:38 testsrv su: + tty?? root-root
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3053]: nfsd 0 10 sock 4
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3055]: nfsd 0 11 sock 4
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3059]: nfsd 0 12 sock 4
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3061]: nfsd 0 13 sock 4
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3063]: nfsd 0 14 sock 4
Aug 18 13:40:38 testsrv su: + tty?? root-root
Aug 18 13:40:38 testsrv /usr/sbin/nfsd[3021]: nfsd 0 15 sock 4
Aug 18 13:40:38 testsrv syslog: Oracle Cluster Ready Services disabled by administrator.
Aug 18 13:40:39 testsrv sfd[3178]: starting the daemon.
Aug 18 13:40:39 testsrv krsd[3176]: Delay time is 300 seconds
Aug 18 13:40:39 testsrv inetd[3206]: registrar/tcp: Connection from testsrv (10.189.39.46) at Mon Aug 18 13:40:39 2008
Note:
Service guard version: A.11.17
HP Serviceguard Storage Management Suite for RAC: A.01.00 (including Serviceguard A11.17 and Veritas 4.1)
Oracle10g R2 10.2.0.3. And also installed the SG and CFS patches:
Serviceguard 11.17
PHCO_32426
PHCO_35048
PHSS_33838
PHSS_33839
PHSS_35371
PHKL_34213
PHKL_35420
CFS/CVM/VxVM 4.1 patches:
PHCO_33081
PHCO_33082
PHCO_33522
PHCO_33691
PHCO_35431
PHCO_35476
PHCO_35518
PHKL_33566
PHKL_33620
PHKL_35334
PHKL_35430
PHKL_35477
PHKL_34741
PHNE_34664
PHNE_33723
PHNE_35353

Can you please help me on this?

[Steven E. Protter]

Shalom,

It looks like you are trying to run SG in a vm environment.

It is either frakking(messing up) your NFS or you are trying to do a high availability NFS environment and failing.

You have configuration problems and need to check your package scripts logs and configuration from step 1.

A brief description of what you want this cluster to do for you.

SEP

[melvyn burnard · Accepted solution]

Hmm, not sure where Steven gets the idea there are VM's involved here, as you are using Vpars.
From what you have posted, this looks more like a Veritas issue, as the GAB is part of the Veritas side of the SMS software, along with LLT.
Revisit all of the configuration steps, and make sure that all is set correctly for this.
This may be one where you need to log a call with HP Response Centre.

[Srinikalyan]

Hi,

In this cluster, we want to start the RAC instance. I suspect there is a problem with the CFS.
Aug 18 13:40:35 testsrv vmunix: GAB WARNING V-15-1-20115 Port d registration failed, GAB not configured
Aug 18 13:40:35 testsrv vmunix: vxgms: GAB_API_REGISTER error=261
Aug 18 13:40:35 testsrv vmunix: ODM WARNING V-41-6-5 odm_gms_api_start_msgs fails
Aug 18 13:40:38 testsrv syslog: Oracle Cluster Ready Services disabled by administrator.

Is there any patches required to resolve this issues?

[Duncan Edmonstone]

Hi,

I don't think you can assume the Veritas errors are actually your problem - these are expected and can be ignored as documented here:

http://docs.hp.com/en/T2771-90028/T2771-90028_R4.pdf

See p38

I'd take a look at other log files such as package logs etc for some more guidance on this.

HTH

Duncan

[Srinikalyan]

Please find the CFS log file.
Thanks,
Srini

[Deepak Kr]

For fixing the errors related to GAB (from CVM) log a case with symantec and ask them what are the patches required for this configuration.

[Srinikalyan]

Hi,
We are not sure that the problem is in with GAB. Even the logs are not helping. We reset the hanged Vpars and rebuilt the cluster. It works fine until we stopped the cluster.Again the vpars hangs up. Any clue on what may be the problem?

Srini

[Srinikalyan]

Closed. thanks for the info.