cluster package is not stopping

[Ankit Kr Gupta]

Dear All,
I am getting the following error while stopping the cluster.

#cmhaltpkg dbci
cmhaltpkg : script failed with no restart : dbci should not be restarted.
Check the syslog & pkg log files for more detailed information.

after this, when i am checking the cmviewcl, it shows me that it is down.
Plz let me know, what should i check. After this , I have restarted the server. Then on MP logs i found the below warning/error.
Also i want to know the referd warning by the Mp log code
***********************************************
276 SFW 1 *3 0x768002AC01E01220 0000000000000000 MC_RENDEZ_PAL_CANT_ESCALATE 12 Apr 2009 06:15:15

&

281 OSA *5 0x2149E1876D0212C0 FF0F016F00200300 Type-02 206f01 2125569 12 Apr 2009 06:17:17

***********************************************
Plz help me. Thanks in advance.

Ankit

[Prashanth Waugh]

Hi,

Check the /etc/cmccluster//pkg.cntl.log File


Reagrds
Prashant

[Prashanth Waugh]

Hi,

have you done in modification in pkg control file.
check it.

/etc/cmcluster/pkg name/pkg.cntl

Regards
Prsahnt

[Prashanth Waugh]

Hi,

without stopping the pkg successfully u restarted the node right?

Regards
Prashnat

[Suraj K Sankari]

Hi,
>>Check the syslog & pkg log files for more detailed information

Did you check these logs ?
/var/adm/syslog/syslog.log
and
/etc/cmcluster//pkg.cntl.log


Suraj

[R.K. #]

Hi Ankit,
As mentioned, the best place to check out the reason are log files:

1. /etc/cmcluster/package/package.cntl.log
2. syslog.log

Have a look on this as well:
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1030589

Regds,
R.K.

[Ankit Kr Gupta]

Thanks to all of u for suggestions. I read the logs from /etc/cmcluster/pkg/pkgname.cntl.log. I found these errors..

***********************************
umount: cannot unmount /usr/sap/PRD/DVEBMGS00 : Device busy
umount: return error 1.
WARNING: Running fuser to remove anyone using the file system directly.
/dev/vg02/lvol1: 23775co(prdadm) 23773c(prdadm) 17683mcto(prdadm) 4336mcto(prdadm) 13782mcto(prdadm) 4334mcto(prdadm) 13783mcto(prdadm) 17682mcto(prdadm) 23803co(prdadm) 4325mcto(prdadm) 4335mcto(prdadm) 17672mcto(prdadm) 13773mcto(prdadm) 13784mcto(prdadm) 7216co(prdadm)

umount: cannot unmount /usr/sap/PRD/DVEBMGS00 : Device busy
umount: return error 1.
ERROR: Function umount_fs
ERROR: Failed to unmount /dev/vg02/lvol1
************
vgchange: Couldn't deactivate volume group "vg02":
Device busy
ERROR: Function deactivate_volume_group
ERROR: Failed to deactivate vg02

*************************************


and from syslog i found these

*************************************
Apr 12 11:41:07 MSLCIDB cmcld: (MSLAPP) Halted package apPRD01 on node MSLAPP.
Apr 12 11:41:41 MSLCIDB CM-CMD[23556]: cmhaltpkg msldbci
Apr 12 11:41:41 MSLCIDB CM-CMD[23556]: Request from root on node MSLCIDB to halt the package
Apr 12 11:41:41 MSLCIDB cmcld: Request from node MSLCIDB to disable global switching for package msldbci.
Apr 12 11:41:41 MSLCIDB cmcld: Request from node MSLCIDB to halt package msldbci on node MSLCIDB.
Apr 12 11:41:41 MSLCIDB cmcld: Executing '/etc/cmcluster/PRD/msldbci.cntl stop' for package msldbci, as service PKG*42753.
Apr 12 11:41:41 MSLCIDB CM-msldbci[23576]: cmmodnet -r -i 192.168.193.3 192.168.193.0
Apr 12 11:42:00 MSLCIDB LVM[23848]: vgchange -a n vglock
Apr 12 11:42:00 MSLCIDB LVM[23856]: vgchange -a n vg01
Apr 12 11:42:01 MSLCIDB LVM[23872]: vgchange -a n vg03
Apr 12 11:42:02 MSLCIDB LVM[23880]: vgchange -a n vg04
Apr 12 11:42:02 MSLCIDB LVM[23888]: vgchange -a n vg05
Apr 12 11:42:03 MSLCIDB LVM[23896]: vgchange -a n vg06
Apr 12 11:42:05 MSLCIDB LVM[23904]: vgchange -a n vg07
Apr 12 11:42:05 MSLCIDB cmcld: Service PKG*42753 terminated due to an exit(1).
Apr 12 11:42:05 MSLCIDB cmcld: Halted package msldbci on node MSLCIDB.
Apr 12 11:42:05 MSLCIDB cmcld: Package msldbci halt script exited with NO_RESTART.
Apr 12 11:42:05 MSLCIDB cmcld: Examine the file /etc/cmcluster/PRD/msldbci.cntl.log for more details.
Apr 12 11:42:05 MSLCIDB cmcld: Switching disabled on package msldbci.
Apr 12 11:42:05 MSLCIDB cmcld: Package msldbci did not halt cleanly. Owning node encountered an error during halt.
Apr 12 11:42:39 MSLCIDB vmunix: NFS server msldbci not responding still trying
Apr 12 11:43:15 MSLCIDB CM-CMD[24050]: /usr/sbin/cmhaltnode -vf
Apr 12 11:43:15 MSLCIDB cmcld: Request from root on node MSLCIDB to halt the cluster on this node
Apr 12 11:43:15 MSLCIDB cmcld: Request from node MSLCIDB to disable node switching for package msldbci on node MSLCIDB.
Apr 12 11:43:15 MSLCIDB cmcld: Disabled package msldbci on node MSLCIDB.
Apr 12 11:43:15 MSLCIDB cmcld: Request from node MSLCIDB to disable node switching for package apPRD01 on node MSLCIDB.
Apr 12 11:43:15 MSLCIDB cmcld: Disabled package apPRD01 on node MSLCIDB.
Apr 12 11:43:15 MSLCIDB cmlvmd[1772]: Volume group vg02 is still active.
Apr 12 11:43:25 MSLCIDB /usr/sbin/envd[1238]: terminated by signal 15
Apr 12 11:43:25 MSLCIDB diagmond[1235]: Exit due to user requested abort
Apr 12 11:43:26 MSLCIDB sshd[674]: Received signal 15; terminating.
Apr 12 11:43:30 MSLCIDB cimserver[1134]: HP WBEM Services for HP-UX stopped.
Apr 12 11:43:41 MSLCIDB inetd[760]: Going down on signal 15
Apr 12 11:43:42 MSLCIDB rpcbind: terminate: rpcbind terminating on signal. Restart with "rpcbind -w"
Apr 12 11:43:43 MSLCIDB syslogd: going down on signal 15
************************************

I am unable to find the solution for this problem ( if it will occur next time). Plz suggest what should i do. also plz tell me that what MP log coding is saying.

Ankit

[R.K. #]

Hi Ankit,

I am not sure if there is anything to do with MP logs, but its for sure that it CANNOT unmount /usr/sap/PRD/DVEBMGS00 i.e. /dev/vg02/lvol1 and this is the point of concern here.

We can try following command manually also to kill any process related to this mount point.

#fuser -cu /usr/sap/PRD/DVEBMGS00 #to check the running processes related to this fs

and

#fuser -ku /usr/sap/PRD/DVEBMGS00 #to kill

Regds,
R.K.

[sujit kumar singh]

Hi Ankit What RK has said is true, Pakcage halt failed as DVEBMGS could not be unmounted.
This is showing the Device busy, some process sure had been running using this FS but was someone most probably prdadm in one of the directories.

Well nneed to check wahy the FS were not unmounted and apart from that ensure that you always uf nay changes applied from UNIX side or using the SAP modifications, every time you do a proper failover test of teh ckuster.

ASK SAP admin also to have a check.
Regards
Sujit

[gb karki]

Hi Ankit,

As per my understanding there SAP is running, if this is the case then whenever you going to stop the ci package before that you need to stop the DVEBMGS and then you stop the ci package.

As per error log, before restarting the server you need to be check the package file systems are mounted or not.

if there any package file systems are mounted then you need to unmount the file system then deactive the volume group.


Regards
Karki