Cluster is running but package is not up

[Jestin John Chacko]

Dear to all..

I am facing with a strange problem while i rebooted my production servers for monthly maintenance purpose after rebooting i had started the cluster service,but when i start the package
using this

cmrunpkg -v -n


the package script for failed with no restart should not be restarted unable to run package on the node.
check the syslog and pkg log files for more detailed information

cmrunpkg :unable to start some packages or package instance


what i should do to make my cluster up with the corresponding packages....

please help me in this issue...

[Steven E. Protter]

Shalom,

There is a log file in the package directory that will help you find the problem.

Also check /var/adm/syslog/syslog.log for warning or error messages.

Without looking at the logs, all I can do is guess.

SEP

[Rita C Workman]

Go out to your package cntl logfile and see what happened when your package went down.

My guess....a mount point got hung, thus hanging the volume group and not properly de-activating it. Check your logfile and simply do
vgchange -c n /dev/vgname
vgchange -a n /dev/vgname

Then with the package down I personall like to:
vgchange -a y /dev/vgname
vgdisplay /dev/vgname

..and when all looks okay I give it back to the cluster:
vgchange -a n /dev/vgname
vgchange -c y /dev/vgname

And to kick it off to run I tend to type:
cmmodpkg -e package-name

the -e option will start on the pkg primary node and set the pkg status to enabled.

Hope this gives you a place to start,
Rgrds,
Rita

[Vishu]

Hi,

well, the best practice is always check your package.cntl.log file for the errors and also check the syslog.log. This will help in proper resolution of the problem.

everyone is making just a guess. So, better if you paste your error here then it can be proceeded on a right path.

[Jestin John Chacko]

Dear to all

Thanks for your quick replys.....

Actually i had tried it for the same i am attaching the log files checking this please give me a solution

[Steven E. Protter]

Shalom,

Nov 6 01:48:50 kmcedb1 LVM[6522]: vgchange -a y vgerpdata
Nov 6 01:49:00 kmcedb1 LVM[6525]: vgchange -a y vgerparch
Nov 6 01:49:07 kmcedb1 LVM[6538]: vgchange -a y vgerpdata_bkup
Nov 6 01:51:03 kmcedb1 ftpd[7183]: Data port : 20
Nov 6 01:51:03 kmcedb1 ftpd[7183]: FTP server (Revision 1.1 Version wuftpd-2.6.1(PHNE_34698) Fri Nov 10 10:21:03 GMT 2006) ready.
Nov 6 01:51:03 kmcedb1 ftpd[7183]: FTP LOGIN FROM kmcedb2.kmcgov.in [172.16.66.53], applprod
Nov 6 01:51:04 kmcedb1 CM-CMD[7184]: cmruncl -v
Nov 6 01:51:05 kmcedb1 cmclconfd[7185]: Request from root on node kmcedb1 to start the cluster on this node
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Logging level changed to level 0.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Daemon Initialization - Maximum number of packages supported for this incarnation is 150.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Global Cluster Information:
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Heartbeat Interval is 1.00 seconds.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Logging level changed to level 0.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Node Timeout is 5.00 seconds.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Network Polling Interval is 5.00 seconds.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Auto Start Timeout is 600.00 seconds.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Failover Optimization is disabled.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Information Specific to node kmcedb1:
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Cluster lock disk: /dev/dsk/c14t0d0.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: lan900 0x001a4b064756 172.16.66.52 bridged net:1
Nov 6 01:51:05 kmcedb1 cmcld[7191]: lan901 0x0019bbeb3dc6 10.1.1.5 bridged net:2
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Heartbeat Subnet: 172.16.66.0
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Heartbeat Subnet: 10.1.1.4
Nov 6 01:51:05 kmcedb1 cmcld[7191]: The maximum # of concurrent local connections to the daemon that will be supported is 24546.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: rcomm health: Initializing timeout to 120000000 microseconds
Nov 6 01:51:06 kmcedb1 cmcld[7191]: Total allocated: 35108680 bytes, used: 2017600 bytes, unused 33091072 bytes
Nov 6 01:51:06 kmcedb1 cmcld[7191]: Starting cluster management protocols.
Nov 6 01:51:06 kmcedb1 cmcld[7191]: Attempting to form a new cluster
Nov 6 01:51:06 kmcedb1 cmcld[7191]: Beginning standard election
Nov 6 01:51:06 kmcedb1 cmcld[7191]: New node kmcedb2 is joining the cluster
Nov 6 01:51:06 kmcedb1 cmcld[7191]: Clearing Cluster Lock
Nov 6 01:51:08 kmcedb1 cmcld[7191]: Turning on safety time protection
Nov 6 01:51:08 kmcedb1 cmcld[7191]: 2 nodes have formed a new cluster, sequence #1
Nov 6 01:51:08 kmcedb1 cmcld[7191]: The new active cluster membership is: kmcedb1(id=1), kmcedb2(id=2)
Nov 6 01:51:09 kmcedb1 cmlvmd[7197]: Clvmd initialized successfully.
Nov 6 01:51:09 kmcedb1 cmcld[7191]: Request from node kmcedb1 to start package erpdb on node kmcedb1.
Nov 6 01:51:09 kmcedb1 cmcld[7191]: Executing '/etc/cmcluster/ERP/erpdb.cntl start' for package erpdb, as service PKG*21505.
Nov 6 01:51:13 kmcedb1 cmcld[7191]: Request from node kmcedb1 to disable global switching for package erpapps.
Nov 6 01:51:13 kmcedb1 cmcld[7191]: Disabled switching for package erpapps.
Nov 6 01:51:15 kmcedb1 cmcld[7191]: Service PKG*21505 terminated due to an exit(1).
Nov 6 01:51:15 kmcedb1 cmcld[7191]: Package erpdb run script exited with NO_RESTART.
Nov 6 01:51:15 kmcedb1 cmcld[7191]: Examine the file /etc/cmcluster/ERP/erpdb.cntl.log for more details.

Package failed to start. Package was set to no restart.

Please look at this log file:

etc/cmcluster/ERP/erpdb.cntl.log

SEP

[Jestin John Chacko]

Dear steven

I had gone through that file which is added with the log file but i cant find solution from that...so what i have to do next to recover from this issue..

please give me some replys...

[Vishu · Accepted solution]

Hi Jestin,

>>>>>>>>>
Nov 6 01:48:50 kmcedb1 LVM[6522]: vgchange -a y vgerpdata
Nov 6 01:49:00 kmcedb1 LVM[6525]: vgchange -a y vgerparch
Nov 6 01:49:07 kmcedb1 LVM[6538]: vgchange -a y vgerpdata_bkup
<<<<<<<<<<<<<<

you can see here that your VG is activated with "-a y" while booting.

The error you are getting is

>>>>>>>>>>>>>>
Nov 6 02:17:14 - Node "kmcedb1": Activating volume group vgerpdata with exclusive option.
vgchange: Activation mode requested for the volume group "/dev/vgerpdata" conflicts with configured mode.
ERROR: Function activate_volume_group
ERROR: Failed to activate vgerpdata
Nov 6 02:17:14 - Node "kmcedb1": Deactivating volume group vgerpdata
vgchange: Couldn't deactivate volume group "vgerpdata":
Device busy
<<<<<<<<<<<<<<<<<<<<<

so, the problem lies in activating the VG as VG is configured to active in exclusive mode but they are activated with normal read/write mode.

you need to deactivate these three VGs after umounting their FS.

# vgchange -a n

and now try to run the package with cmrunpkg command and see it is starting up.

according you will have to edit your /etc/lvmrc file not to activate your VGs while server startup. these VGs are cluster aware so they will be active by cluster startup only.

[Jestin John Chacko]

Dear Vishu...


Thanks for you valuable support...i had manged to solve this issue with your help...

Thanks a lot