unable to start package

[irshad ali]

Hi,

I have 02 node cluster with shared VG. on OS version 11.23. database-oracle RAC
on node 01 we down the package and now trying to start the package with below commannd-

rumsla31:/root # cmrunpkg ldwhdb1
cmrunpkg : Node rumsla31 is currently unable to run package ldwhdb1.
Check the syslog on node rumsla31 and pkg log files for more detailed information.

VG status on this node.
rumsla31:/root # vgdisplay /dev/vg_ldwhdb
vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "/dev/vg_ldwhdb".

No error was found in syslog and package log file.
plz help how to start the package.

[Michal Kapalka (mikap)]

hi,

check the package log directly in cluster directory :

/etc/cmcluster/PKG_NAME/PKG_NAME.cntl.log

mikap

[irshad ali]

hi Michel,

Thanks for instant reply. There is no pkg.cntl.log file persent on server.

rumsla31:/etc/cmcluster/ldwhdb1 # cmviewcl

CLUSTER STATUS
rumsla30 up

NODE STATUS STATE
rumsla31 up running

PACKAGE STATUS STATE AUTO_RUN NODE
adsdb1 up running enabled rumsla31
condb1 up running enabled rumsla31
odsdb1 up running enabled rumsla31
dwhdb1 up running disabled rumsla31
sibdb1 up running enabled rumsla31
listener1 up running enabled rumsla31
gsd1 up running enabled rumsla31

NODE STATUS STATE
rumsla32 up running

PACKAGE STATUS STATE AUTO_RUN NODE
adsdb2 up running enabled rumsla32
condb2 up running enabled rumsla32
dwhdb2 up running enabled rumsla32
odsdb2 up running enabled rumsla32
sibdb2 up running enabled rumsla32
listener2 up running enabled rumsla32
gsd2 up running enabled rumsla32
ldwhdb2 up running disabled rumsla32

UNOWNED_PACKAGES

PACKAGE STATUS STATE AUTO_RUN NODE
ldwhdb1 down halted disabled unowned
rumsla31:/etc/cmcluster/ldwhdb1 # cmviewcl -v -p ldwhdb1

UNOWNED_PACKAGES

PACKAGE STATUS STATE AUTO_RUN NODE
ldwhdb1 down halted disabled unowned

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback automatic

Script_Parameters:
ITEM STATUS NODE_NAME NAME
Subnet up rumsla31 192.168.0.0

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up disabled rumsla31
rumsla31:/etc/cmcluster/ldwhdb1 # cd /etc/cmcluster/ldwhdb1
rumsla31:/etc/cmcluster/ldwhdb1 # ll
total 848
-rw-r--r-- 1 root sys 8308 Feb 12 2009 haoracle.conf
-r-xr-xr-x 1 root sys 2742 Feb 12 2009 haoracle.mon
-r-xr-xr-x 1 root sys 12802 Feb 12 2009 haoracle.sh
-r-xr-xr-x 1 root sys 3669 Feb 12 2009 haoracle_sql.sh
-r--r--r-- 1 root root 501 Feb 12 2009 ldwhdb1.config
-r--r--r-- 1 root root 13365 Feb 12 2009 ldwhdb1.sdf
-r-xr--r-- 1 root root 54828 Feb 12 2009 ldwhdb1.sdf.sh
-rw-r--r-- 1 root root 182140 Aug 20 08:42 ldwhdb1.sdf.sh.log
-r-xr-xr-x 1 root sys 2226 Feb 12 2009 toolkit.sh
rumsla31:/etc/cmcluster/ldwhdb1 #

plz reply soon what to do? Thanks

[Prashanth Waugh]

Hi ,

It seems that vg is not activated. U need to activate the vg.
activate the vg using
vgchnage -a e vgname

can u paste the syslog

Regards,
Prashant

[Asif Sharif]

Salam Irshad,

run the following commands,
#vgchange -a e vg_ldwhdb
#cmrunpkg -e ldwhdb1

Regards,
Asif Sharif

[Matti_Kurkela · Accepted solution]

Something seems to have been written into the log file /etc/cmcluster/ldwhdb1/ldwhdb1.sdf.sh.log recently.

The file looks rather big. Please look into it and see if there are any messages related to your latest package start-up attempt.

Your "cmviewcl -v -p ldwhdb1" Node_Switching_Parameters indicates "STATUS" is "disabled" for this package. This means something has gone wrong in some previous Serviceguard operation (the most common cause would be that the package did not shut down cleanly). Serviceguard remembers this and does not allow the package to run on this node again until it has been told the problem is fixed. You can do it with this command:

cmmodpkg -e -n rumsla31 ldwhdb1

> rumsla31:/root # vgdisplay /dev/vg_ldwhdb
> vgdisplay: Volume group not activated.
> vgdisplay: Cannot display volume group "/dev/vg_ldwhdb".

This is normal and expected behavior when the package is not up on this node.

MK

[irshad ali]

Thanks all for quick responce.
package has been made up (not by me) But I want to know the command and procedure to activate it. I am pesting latest syslog below-
Aug 20 08:25:38 rumsla31 CM-CMD[3567]: cmrunpkg ldwhdb1
Aug 20 08:25:38 rumsla31 CM-CMD[3567]: Request from root on node rumsla31 to start the package
Aug 20 08:25:38 rumsla31 cmcld: Unable to start package ldwhdb1. Node rumsla31 not enabled to run it.
Aug 20 08:37:32 rumsla31 cmgmsd[2441]: validate_pid: pstat_getfile2 returned -1 (2, No such file or directory)
Aug 20 08:38:06 rumsla31 above message repeats 97 times
Aug 20 08:38:27 rumsla31 cmgmsd[2441]: validate_pid: pstat_getfile2 returned -1 (2, No such file or directory)
Aug 20 08:39:38 rumsla31 cmomd[18945]: Request to connect as root from skutin@10.160.3.22.
Aug 20 08:39:52 rumsla31 sshd[19189]: error: PAM: Authentication failed
Aug 20 08:38:27 rumsla31 cmgmsd[2441]: validate_pid: pstat_getfile2 returned -1 (2, No such file or directory)
Aug 20 08:39:58 rumsla31 above message repeats 5 times
Aug 20 08:39:58 rumsla31 sshd[19189]: Accepted keyboard-interactive/pam for oracle from 10.160.3.22 port 10135 ssh2
Aug 20 08:40:22 rumsla31 cmgmsd[2441]: validate_pid: pstat_getfile2 returned -1 (2, No such file or directory)
Aug 20 08:40:58 rumsla31 LVM[20959]: vgchange -a s /dev/vg_ldwhdb
Aug 20 08:41:23 rumsla31 cmclconfd[2442]: Updated file /etc/cmcluster/cmclconfig.tmp for node rumsla31 (length = 183728).
Aug 20 08:41:24 rumsla31 cmclconfd[2442]: Updated file /etc/cmcluster/cmclconfig.tmp for node rumsla31 (length = 0).
Aug 20 08:41:24 rumsla31 cmclconfd[2408]: Updated file /etc/cmcluster/cmclconfig for node rumsla31 (length = 92636).
Aug 20 08:41:15 rumsla31 cmgmsd[2441]: validate_pid: pstat_getfile2 returned -1 (2, No such file or directory)
Aug 20 08:41:28 rumsla31 above message repeats 13 times
Aug 20 08:41:28 rumsla31 cmclconfd[2442]: Updated file /etc/cmcluster/cmclconfig.tmp for node rumsla31 (length = 185272).
Aug 20 08:41:28 rumsla31 cmclconfd[2408]: Updated file /etc/cmcluster/cmclconfig for node rumsla31 (length = 93408).
Aug 20 08:41:28 rumsla31 cmclconfd[2408]: Updated file /etc/cmcluster/cmclconfig for node rumsla31 (length = 94180).
Aug 20 08:41:28 rumsla31 cmclconfd[2442]: Updated file /etc/cmcluster/cmclconfig.tmp for node rumsla31 (length = 185272).
Aug 20 08:41:35 rumsla31 cmgmsd[2441]: validate_pid: pstat_getfile2 returned -1 (2, No such file or directory)
Aug 20 08:41:28 rumsla31 cmclconfd[2442]: Updated file /etc/cmcluster/cmclconfig.tmp for node rumsla31 (length = 0).
Aug 20 08:41:39 rumsla31 above message repeats 2 times
Aug 20 08:41:39 rumsla31 cmclconfd[2442]: Updated file /etc/cmcluster/cmclconfig.tmp for node rumsla31 (length = 188360).
Aug 20 08:41:39 rumsla31 cmclconfd[2442]: Updated file /etc/cmcluster/cmclconfig.tmp for node rumsla31 (length = 0).
Aug 20 08:41:39 rumsla31 cmclconfd[2408]: Updated file /etc/cmcluster/cmclconfig for node rumsla31 (length = 94956).
Aug 20 08:41:35 rumsla31 cmgmsd[2441]: validate_pid: pstat_getfile2 returned -1 (2, No such file or directory)
Aug 20 08:42:06 rumsla31 above message repeats 4 times
Aug 20 08:42:06 rumsla31 cmomd[22422]: Request to connect as root from skutin@10.160.3.22.
Aug 20 08:42:07 rumsla31 cmomd[22422]: Request to modify node package switching for ldwhdb1 on cluster rumsla30.
Aug 20 08:42:07 rumsla31 cmcld: Request from skutin on node 10.160.3.22 connected to rumsla31 as root via cmomd(pid 22422) to enable node rumsla31 for switch
ing of package ldwhdb1.
Aug 20 08:42:07 rumsla31 cmcld: Request from node rumsla31 to enable node switching for package ldwhdb1 on node rumsla31.
Aug 20 08:42:07 rumsla31 cmcld: Enabled package ldwhdb1 on node rumsla31.
Aug 20 08:42:07 rumsla31 cmomd[22422]: Successfully enabled package ldwhdb1 from switching on node rumsla31.
Aug 20 08:42:08 rumsla31 cmomd[22422]: Request to start package ldwhdb1 on node rumsla31.
Aug 20 08:42:08 rumsla31 cmcld: Request from skutin on node 10.160.3.22 connected to rumsla31 as root via cmomd(pid 22422) to start package ldwhdb1 on node r
umsla31.
Aug 20 08:42:08 rumsla31 cmcld: Request from node rumsla31 to start package ldwhdb1 on node rumsla31.
Aug 20 08:42:08 rumsla31 cmcld: Executing '/etc/cmcluster/ldwhdb1/ldwhdb1.sdf.sh start' for package ldwhdb1, as service PKG*49679.
Aug 20 08:42:08 rumsla31 LVM[22633]: vgchange -a s /dev/vg_ldwhdb
Aug 20 08:42:08 rumsla31 su: + tty?? root-oracle
Aug 20 08:42:08 rumsla31 CM-ldwhdb1[22699]: cmrunserv -r 50 ldwhdb1 >> /etc/cmcluster/ldwhdb1/ldwhdb1.sdf.sh.log 2>&1 /etc/cmcluster/ldwhdb1/toolkit.sh monit
or
Aug 20 08:42:08 rumsla31 cmcld: Service PKG*49679 terminated due to an exit(0).
Aug 20 08:42:08 rumsla31 cmcld: Started package ldwhdb1 on node rumsla31.
Aug 20 08:42:08 rumsla31 cmomd[22422]: Successfully started package ldwhdb1 on node rumsla31.

++++++++++++++++++++++++++++++
plz help, thanks

[melvyn burnard]

Well the clues and answer re all there in amongst the jumble.
The answers you were given saying that the vg was noactivated as the cause are completely incorrect.
From your data supplied:

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up disabled rumsla31

This tells you that the node has been told it CANNOT run this package.
Look here:
Aug 20 08:25:38 rumsla31 CM-CMD[3567]: cmrunpkg ldwhdb1
Aug 20 08:25:38 rumsla31 CM-CMD[3567]: Request from root on node rumsla31 to start the package
Aug 20 08:25:38 rumsla31 cmcld: Unable to start package ldwhdb1. Node rumsla31 not enabled to run it.

This would have occurred as a result of you halting the package manually, probably using:
cmhaltpkg ldwhdb1

Your problem is that you then first need to re-enable the node to allow it to sytart the package with cmmodpkg -e,
and THEN try to run it.

In fact, this was done as per your last log info (far too much "noise" in the data).
And looking at this data, someone used Serviceguard Manager to do this, running it on a node with the IP address and user name:
Aug 20 08:42:06 rumsla31 cmomd[22422]: Request to connect as root from skutin@10.160.3.22

They then issued the cmmodpkg request:

Aug 20 08:42:07 rumsla31 cmomd[22422]: Request to modify node package switching for ldwhdb1 on cluster rumsla30.
Aug 20 08:42:07 rumsla31 cmcld: Request from skutin on node 10.160.3.22 connected to rumsla31 as root via cmomd(pid 22422) to enable node rumsla31 for switch
ing of package ldwhdb1.
Aug 20 08:42:07 rumsla31 cmcld: Request from node rumsla31 to enable node switching for package ldwhdb1 on node rumsla31.
Aug 20 08:42:07 rumsla31 cmcld: Enabled package ldwhdb1 on node rumsla31.
Aug 20 08:42:07 rumsla31 cmomd[22422]: Successfully enabled package ldwhdb1 from switching on node rumsla31.

and then requested the package to start, which it did:

Aug 20 08:42:08 rumsla31 cmomd[22422]: Request to start package ldwhdb1 on node rumsla31.
Aug 20 08:42:08 rumsla31 cmcld: Request from skutin on node 10.160.3.22 connected to rumsla31 as root via cmomd(pid 22422) to start package ldwhdb1 on node r
umsla31.
Aug 20 08:42:08 rumsla31 cmcld: Request from node rumsla31 to start package ldwhdb1 on node rumsla31.
Aug 20 08:42:08 rumsla31 cmcld: Executing '/etc/cmcluster/ldwhdb1/ldwhdb1.sdf.sh start' for package ldwhdb1, as service PKG*49679.
Aug 20 08:42:08 rumsla31 LVM[22633]: vgchange -a s /dev/vg_ldwhdb
Aug 20 08:42:08 rumsla31 su: + tty?? root-oracle
Aug 20 08:42:08 rumsla31 CM-ldwhdb1[22699]: cmrunserv -r 50 ldwhdb1 >> /etc/cmcluster/ldwhdb1/ldwhdb1.sdf.sh.log 2>&1 /etc/cmcluster/ldwhdb1/toolkit.sh monit
or
Aug 20 08:42:08 rumsla31 cmcld: Service PKG*49679 terminated due to an exit(0).
Aug 20 08:42:08 rumsla31 cmcld: Started package ldwhdb1 on node rumsla31.
Aug 20 08:42:08 rumsla31 cmomd[22422]: Successfully started package ldwhdb1 on node rumsla31.