HPE Community read-only access December 15, 2018
This is a maintenance upgrade. You will be able to read articles and posts, but not post or reply.
Hours:
Dec 15, 4:00 am to 10:00 am UTC
Dec 14, 10:00 pm CST to Dec 15, 4:00 am CST
Dec 14, 8:00 pm PST to Dec 15, 2:00 am PST
System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Cluster is running but package is not up

 
SOLVED
Go to solution
Jestin John Chacko
Regular Advisor

Cluster is running but package is not up

Dear to all..

I am facing with a strange problem while i rebooted my production servers for monthly maintenance purpose after rebooting i had started the cluster service,but when i start the package
using this

cmrunpkg -v -n


the package script for failed with no restart should not be restarted unable to run package on the node.
check the syslog and pkg log files for more detailed information

cmrunpkg :unable to start some packages or package instance


what i should do to make my cluster up with the corresponding packages....

please help me in this issue...

8 REPLIES
Steven E. Protter
Exalted Contributor

Re: Cluster is running but package is not up

Shalom,

There is a log file in the package directory that will help you find the problem.

Also check /var/adm/syslog/syslog.log for warning or error messages.

Without looking at the logs, all I can do is guess.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Rita C Workman
Honored Contributor

Re: Cluster is running but package is not up

Go out to your package cntl logfile and see what happened when your package went down.

My guess....a mount point got hung, thus hanging the volume group and not properly de-activating it. Check your logfile and simply do
vgchange -c n /dev/vgname
vgchange -a n /dev/vgname

Then with the package down I personall like to:
vgchange -a y /dev/vgname
vgdisplay /dev/vgname

..and when all looks okay I give it back to the cluster:
vgchange -a n /dev/vgname
vgchange -c y /dev/vgname

And to kick it off to run I tend to type:
cmmodpkg -e package-name

the -e option will start on the pkg primary node and set the pkg status to enabled.

Hope this gives you a place to start,
Rgrds,
Rita
Vishu
Trusted Contributor

Re: Cluster is running but package is not up

Hi,

well, the best practice is always check your package.cntl.log file for the errors and also check the syslog.log. This will help in proper resolution of the problem.

everyone is making just a guess. So, better if you paste your error here then it can be proceeded on a right path.
Jestin John Chacko
Regular Advisor

Re: Cluster is running but package is not up

Dear to all

Thanks for your quick replys.....

Actually i had tried it for the same i am attaching the log files checking this please give me a solution
Steven E. Protter
Exalted Contributor

Re: Cluster is running but package is not up

Shalom,

Nov 6 01:48:50 kmcedb1 LVM[6522]: vgchange -a y vgerpdata
Nov 6 01:49:00 kmcedb1 LVM[6525]: vgchange -a y vgerparch
Nov 6 01:49:07 kmcedb1 LVM[6538]: vgchange -a y vgerpdata_bkup
Nov 6 01:51:03 kmcedb1 ftpd[7183]: Data port : 20
Nov 6 01:51:03 kmcedb1 ftpd[7183]: FTP server (Revision 1.1 Version wuftpd-2.6.1(PHNE_34698) Fri Nov 10 10:21:03 GMT 2006) ready.
Nov 6 01:51:03 kmcedb1 ftpd[7183]: FTP LOGIN FROM kmcedb2.kmcgov.in [172.16.66.53], applprod
Nov 6 01:51:04 kmcedb1 CM-CMD[7184]: cmruncl -v
Nov 6 01:51:05 kmcedb1 cmclconfd[7185]: Request from root on node kmcedb1 to start the cluster on this node
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Logging level changed to level 0.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Daemon Initialization - Maximum number of packages supported for this incarnation is 150.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Global Cluster Information:
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Heartbeat Interval is 1.00 seconds.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Logging level changed to level 0.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Node Timeout is 5.00 seconds.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Network Polling Interval is 5.00 seconds.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Auto Start Timeout is 600.00 seconds.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Failover Optimization is disabled.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Information Specific to node kmcedb1:
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Cluster lock disk: /dev/dsk/c14t0d0.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: lan900 0x001a4b064756 172.16.66.52 bridged net:1
Nov 6 01:51:05 kmcedb1 cmcld[7191]: lan901 0x0019bbeb3dc6 10.1.1.5 bridged net:2
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Heartbeat Subnet: 172.16.66.0
Nov 6 01:51:05 kmcedb1 cmcld[7191]: Heartbeat Subnet: 10.1.1.4
Nov 6 01:51:05 kmcedb1 cmcld[7191]: The maximum # of concurrent local connections to the daemon that will be supported is 24546.
Nov 6 01:51:05 kmcedb1 cmcld[7191]: rcomm health: Initializing timeout to 120000000 microseconds
Nov 6 01:51:06 kmcedb1 cmcld[7191]: Total allocated: 35108680 bytes, used: 2017600 bytes, unused 33091072 bytes
Nov 6 01:51:06 kmcedb1 cmcld[7191]: Starting cluster management protocols.
Nov 6 01:51:06 kmcedb1 cmcld[7191]: Attempting to form a new cluster
Nov 6 01:51:06 kmcedb1 cmcld[7191]: Beginning standard election
Nov 6 01:51:06 kmcedb1 cmcld[7191]: New node kmcedb2 is joining the cluster
Nov 6 01:51:06 kmcedb1 cmcld[7191]: Clearing Cluster Lock
Nov 6 01:51:08 kmcedb1 cmcld[7191]: Turning on safety time protection
Nov 6 01:51:08 kmcedb1 cmcld[7191]: 2 nodes have formed a new cluster, sequence #1
Nov 6 01:51:08 kmcedb1 cmcld[7191]: The new active cluster membership is: kmcedb1(id=1), kmcedb2(id=2)
Nov 6 01:51:09 kmcedb1 cmlvmd[7197]: Clvmd initialized successfully.
Nov 6 01:51:09 kmcedb1 cmcld[7191]: Request from node kmcedb1 to start package erpdb on node kmcedb1.
Nov 6 01:51:09 kmcedb1 cmcld[7191]: Executing '/etc/cmcluster/ERP/erpdb.cntl start' for package erpdb, as service PKG*21505.
Nov 6 01:51:13 kmcedb1 cmcld[7191]: Request from node kmcedb1 to disable global switching for package erpapps.
Nov 6 01:51:13 kmcedb1 cmcld[7191]: Disabled switching for package erpapps.
Nov 6 01:51:15 kmcedb1 cmcld[7191]: Service PKG*21505 terminated due to an exit(1).
Nov 6 01:51:15 kmcedb1 cmcld[7191]: Package erpdb run script exited with NO_RESTART.
Nov 6 01:51:15 kmcedb1 cmcld[7191]: Examine the file /etc/cmcluster/ERP/erpdb.cntl.log for more details.

Package failed to start. Package was set to no restart.

Please look at this log file:

etc/cmcluster/ERP/erpdb.cntl.log

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Jestin John Chacko
Regular Advisor

Re: Cluster is running but package is not up

Dear steven

I had gone through that file which is added with the log file but i cant find solution from that...so what i have to do next to recover from this issue..

please give me some replys...
Vishu
Trusted Contributor
Solution

Re: Cluster is running but package is not up

Hi Jestin,

>>>>>>>>>
Nov 6 01:48:50 kmcedb1 LVM[6522]: vgchange -a y vgerpdata
Nov 6 01:49:00 kmcedb1 LVM[6525]: vgchange -a y vgerparch
Nov 6 01:49:07 kmcedb1 LVM[6538]: vgchange -a y vgerpdata_bkup
<<<<<<<<<<<<<<

you can see here that your VG is activated with "-a y" while booting.

The error you are getting is

>>>>>>>>>>>>>>
Nov 6 02:17:14 - Node "kmcedb1": Activating volume group vgerpdata with exclusive option.
vgchange: Activation mode requested for the volume group "/dev/vgerpdata" conflicts with configured mode.
ERROR: Function activate_volume_group
ERROR: Failed to activate vgerpdata
Nov 6 02:17:14 - Node "kmcedb1": Deactivating volume group vgerpdata
vgchange: Couldn't deactivate volume group "vgerpdata":
Device busy
<<<<<<<<<<<<<<<<<<<<<

so, the problem lies in activating the VG as VG is configured to active in exclusive mode but they are activated with normal read/write mode.

you need to deactivate these three VGs after umounting their FS.

# vgchange -a n

and now try to run the package with cmrunpkg command and see it is starting up.

according you will have to edit your /etc/lvmrc file not to activate your VGs while server startup. these VGs are cluster aware so they will be active by cluster startup only.
Jestin John Chacko
Regular Advisor

Re: Cluster is running but package is not up

Dear Vishu...


Thanks for you valuable support...i had manged to solve this issue with your help...

Thanks a lot