Cluster plg down with showing cmrunpkg: Unable to start some package or package instances

[Aungshuman Paul]

Hi,

Can anyone help me regarding the following error. My Cluster pkg down with showing the following error.
# cmrunpkg -n erpdb1 dbciPRD
Running package dbciPRD on node erpdb1
The package script for dbciPRD failed with no restart. dbciPRD should not be restarted
Unable to run package dbciPRD on node erpdb1
Check the syslog and pkg log files for more detailed information
cmrunpkg: Unable to start some package or package instances


Aungshu

[R.K. #]

Hi Anugshu,

Need to see syslog.log and /etc/cmcluster/package_name/package_name.cntl.log to get more info.

[Aungshuman Paul]

Hi,

FInd the O/P below :

# tail -n 50 /var/adm/syslog/syslog.log
Dec 6 11:05:47 erpdb2 cmcld[1964]: Attempting to form a new cluster
Dec 6 11:05:53 erpdb2 above message repeats 21 times
Dec 6 11:05:53 erpdb2 cmcld[1964]: Turning on safety time protection
Dec 6 11:05:47 erpdb2 cmcld[1964]: Beginning standard election
Dec 6 11:05:53 erpdb2 above message repeats 21 times
Dec 6 11:05:53 erpdb2 cmcld[1964]: 2 nodes have formed a new cluster, sequence #1
Dec 6 11:05:50 erpdb2 syslog: /usr/sbin/cmrunnode -v
Dec 6 11:05:53 erpdb2 cmcld[1964]: The new active cluster membership is: erpdb2(id=2), erpdb1(id=1)
Dec 6 11:05:53 erpdb2 cmlvmd[1981]: Clvmd initialized successfully.
Dec 6 11:06:47 erpdb2 cmcld[1964]: Request from node erpdb2 to disable global switching for package dbciPRD.
Dec 6 11:14:22 erpdb2 vmunix: NFS getattr failed for server prddbci: error 5 (RPC: Timed out)
Dec 6 11:21:05 erpdb2 vmunix: NFS getattr failed for server prddbci: error 5 (RPC: Timed out)
Dec 6 11:21:16 erpdb2 above message repeats 21 times
Dec 6 11:21:25 erpdb2 vmunix: NFS getattr failed for server prddbci: error 5 (RPC: Timed out)
Dec 6 11:35:39 erpdb2 esmd: Essential Services Monitor daemon started
Dec 6 11:35:39 erpdb2 esmd: Started monitoring the EVM daemon
Dec 6 11:35:40 erpdb2 krsd[4489]: Delay time is 300 seconds
Dec 6 11:35:39 erpdb2 vmunix: NFS getattr failed for server prddbci: error 5 (RPC: Timed out)
Dec 6 11:35:40 erpdb2 above message repeats 41 times
Dec 6 11:35:40 erpdb2 sfd[4490]: started 'insf' to create device special files for newly found devices.
Dec 6 11:35:45 erpdb2 sfd[4490]: execution of 'insf' completed.
Dec 6 11:36:01 erpdb2 vmunix: NFS getattr failed for server prddbci: error 5 (RPC: Timed out)
Dec 6 11:36:01 erpdb2 SAP_00[4479]: Cannot open Profile /usr/sap/PRD/SYS/profile/START_D00_erpdb2. (Error 2 No such file or directory) [ntservmainux.cpp 548]
Dec 6 12:08:40 erpdb2 cmcld[1964]: Request from node erpdb2 to start package dbciPRD on node erpdb2.
Dec 6 12:08:40 erpdb2 cmcld[1964]: Executing '/etc/cmcluster/PRD/dbciPRD.control.script start' for package dbciPRD, as service PKG*60929.
Dec 6 12:08:40 erpdb2 LVM[5385]: vgchange -a e vgerpprdraid1
Dec 6 12:08:40 erpdb2 LVM[5390]: vgchange -a e vgerpprdraid5
Dec 6 12:08:44 erpdb2 syslog: cmmodnet -a -i 10.254.20.40 10.254.20.0
Dec 6 12:08:46 erpdb2 su: + tty?? root-prdadm
Dec 6 12:08:48 erpdb2 su: + tty?? root-oraprd
Dec 6 12:09:02 erpdb2 sshd[6934]: Accepted keyboard-interactive/pam for root from 192.168.30.126 port 1815 ssh2
Dec 6 12:09:26 erpdb2 syslog: cmmodnet -r -i 10.254.20.40 10.254.20.0
Dec 6 12:09:32 erpdb2 LVM[9450]: vgchange -a n vgerpprdraid1
Dec 6 12:09:20 erpdb2 su: + tty?? root-oraprd
Dec 6 12:09:32 erpdb2 above message repeats 5 times
Dec 6 12:09:32 erpdb2 LVM[9454]: vgchange -a n vgerpprdraid5
Dec 6 12:09:26 erpdb2 su: + tty?? root-prdadm
Dec 6 12:09:32 erpdb2 above message repeats 9 times
Dec 6 12:09:32 erpdb2 cmcld[1964]: Service PKG*60929 terminated due to an exit(1).
Dec 6 12:09:32 erpdb2 cmcld[1964]: Package dbciPRD run script exited with NO_RESTART.
Dec 6 12:09:32 erpdb2 cmcld[1964]: Examine the file /etc/cmcluster/PRD/dbciPRD.control.script.log for more details.
Dec 6 12:09:32 erpdb2 cmcld[1964]: Switching disabled on package dbciPRD.
Dec 6 12:09:32 erpdb2 cmcld[1964]: Request from node erpdb2 to disable global switching for package dbciPRD.
Dec 6 12:09:32 erpdb2 cmcld[1964]: Unable to start package dbciPRD. Node erpdb2 is not able to run it.
Dec 6 12:13:36 erpdb2 automountd[1075]: server prddbci not responding
Dec 6 12:13:36 erpdb2 automountd[1075]: server prddbci not responding
Dec 6 12:25:27 erpdb2 cmcld[1964]: Request from node erpdb2 to disable global switching for package dbciPRD.
Dec 6 12:25:27 erpdb2 cmcld[1964]: Unable to start package dbciPRD. Node erpdb1 is not able to run it.
Dec 6 12:35:06 erpdb2 cmcld[1964]: Request from node erpdb2 to enable global switching for package dbciPRD.
Dec 6 12:35:06 erpdb2 cmcld[1964]: Enabled switching for package dbciPRD.

root@erpdb2 [/var/adm/syslog]
#

#############################################
# tail -n 50 /etc/cmcluster/PRD/dbciPRD.control.script.log
HANFS -- Dec 6 12:09:26 - Node "erpdb2": Restarting rpc.lockd
Dec 6 12:09:26 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvASCS
Dec 6 12:09:27 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvusrsapPRD
Dec 6 12:09:27 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvsapmntPRD
Dec 6 12:09:27 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvusrsaptrans
Dec 6 12:09:28 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid5/lvoraclePRDsapdata1
umount: cannot unmount /dev/vgerpprdraid5/lvoraclePRDsapdata1 : Device busy
umount: return error 1.
/dev/vgerpprdraid5/lvoraclePRDsapdata1 in use by:
6947: ora_ckpt_PRD
6945: ora_lgwr_PRD
6943: ora_dbw0_PRD
WARNING: Running fuser to remove anyone using the file system directly.
/dev/vgerpprdraid5/lvoraclePRDsapdata1: 6947o(prdadm) 6945o(prdadm) 6943o(prdadm)

Dec 6 12:09:28 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid5/lvoraclePRDsapdata2
Dec 6 12:09:28 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid5/lvoraclePRDsapdata3
Dec 6 12:09:28 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid5/lvoraclePRDsapdata4
Dec 6 12:09:28 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid5/lvdump
Dec 6 12:09:28 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid5/lvoraclePRDsapcheck
Dec 6 12:09:29 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid5/lvoraclePRDsapbackup
Dec 6 12:09:29 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid5/lvoraclePRDsapreorg
Dec 6 12:09:29 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvoraclePRDoriglogA
Dec 6 12:09:29 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvoraclePRDoriglogB
Dec 6 12:09:29 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvoraclePRDmirrlogA
Dec 6 12:09:29 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvoraclePRDmirrlogB
Dec 6 12:09:29 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvoraclePRDoraarch
Dec 6 12:09:30 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvoracleclient
Dec 6 12:09:30 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvoraclestage102_64
Dec 6 12:09:30 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvoraclePRD102_64
umount: cannot unmount /dev/vgerpprdraid1/lvoraclePRD102_64 : Device busy
umount: return error 1.
/dev/vgerpprdraid1/lvoraclePRD102_64 in use by:
6937: ora_pmon_PRD
WARNING: Running fuser to remove anyone using the file system directly.
/dev/vgerpprdraid1/lvoraclePRD102_64: 6937mcto(prdadm)

umount: cannot unmount /dev/vgerpprdraid1/lvoraclePRD102_64 : Device busy
umount: return error 1.
Dec 6 12:09:30 PM - Unmount /dev/vgerpprdraid1/lvoraclePRD102_64 failed, trying again.
/dev/vgerpprdraid1/lvoraclePRD102_64:
Dec 6 12:09:32 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvoraclePRD
Dec 6 12:09:32 PM - Node "erpdb2": Unmounting filesystem on /dev/vgerpprdraid1/lvoracle
Dec 6 12:09:32 PM - Node "erpdb2": Deactivating volume group vgerpprdraid1
Deactivated volume group in Exclusive Mode.
Volume group "vgerpprdraid1" has been successfully changed.
Dec 6 12:09:32 PM - Node "erpdb2": Deactivating volume group vgerpprdraid5
Deactivated volume group in Exclusive Mode.
Volume group "vgerpprdraid5" has been successfully changed.
###### Node "erpdb2": Package start failed at Sun, Dec 6, 2009 12:09:32 PM ######

root@erpdb2 [/var/adm/syslog]
#


Aungshu

[R.K. #]

Hi ..

Is this mounted on any of the nodes:
/dev/vgerpprdraid1/lvoraclePRD102_64

[Aungshuman Paul]

Hi,

No it is not mounted on any system. Find the below O/P :

# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 401968 644056 38% /
/dev/vg00/lvol1 1835008 471424 1352992 26% /stand
/dev/vg00/lvol8 8912896 1868552 6991184 21% /var
/dev/vg00/lvol7 4653056 3098704 1542232 67% /usr
/dev/vg00/lvusrsapPRD
8388608 1436188 6517928 18% /usr/sap/PRD
/dev/vg00/lvtmp 7340032 2511667 4526831 36% /tmp
/dev/vg00/lvol6 8880128 5483984 3369640 62% /opt
/dev/vg00/lvol5 131072 6512 123648 5% /home
DevFS 6 6 0 100% /dev/deviceFileSystem
NFS getattr failed for server prddbci: error 5 (RPC: Timed out)
NFS fsstat failed for server prddbci: error 5 (RPC: Timed out)
bdf: /sapmnt/PRD: Connection timed out

################
root@erpdb2 [/var/adm/syslog]
# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 450816 593216 43% /
/dev/vg00/lvol1 1835008 471256 1353168 26% /stand
/dev/vg00/lvol8 8912896 1324936 7530648 15% /var
/dev/vg00/lvol7 4653056 3099088 1541856 67% /usr
/dev/vg00/lvusrsapPRD
8388608 1757431 6216870 22% /usr/sap/PRD
/dev/vg00/lvtmp 7340032 44240 6847811 1% /tmp
/dev/vg00/lvol6 8880128 5471504 3382024 62% /opt
/dev/vg00/lvol5 131072 22648 107632 17% /home
DevFS 6 6 0 100% /dev/deviceFileSystem
NFS getattr failed for server prddbci: error 5 (RPC: Timed out)
NFS fsstat failed for server prddbci: error 5 (RPC: Timed out)
bdf: /sapmnt/PRD: Connection timed out

root@erpdb2 [/var/adm/syslog]


Aungshu

[R.K. #]

>> NFS getattr failed for server prddbci: error 5 (RPC: Timed out)
>> NFS fsstat failed for server prddbci: error 5 (RPC: Timed out)

Is there any directory/file system mounted NFS related to package??

Kindly provide:
1. cmviewcl -v #(from any node)
2. Try starting package on any one node and at the same time monitor the that node's pkg.cntl.log and see what does it log.
3. What is OS ver?
4. SG version? (#cmversion)

[Felix Eng]

Can you able to start the cluster in debug mode, and database application starts in that mode?
But the cluster is not starting in normal mode.

Can you mount or able bring all cluster related VG in non cluster mode?
Can you database log file, is there any error in it? Please update...

[Viktor Balogh]

Hi Paul,

The cause of this must be visible in the /etc/cmcluster/PRD/dbciPRD.control.script.log file...

Answer these questions:

- Did you modify this package before the start? What?
- Did you add some VGs to it?
- If yes, did you set the cluster flag on the newly added VGs? (vgchange -c y)

[Viktor Balogh]

oh, and what are these mounts in the bdf-output?

NFS getattr failed for server prddbci: error 5 (RPC: Timed out)
NFS fsstat failed for server prddbci: error 5 (RPC: Timed out)

Is this prddbci your package? Are you using here HA-NFS extension?

[Aungshuman Paul]

My Problem solved. It was due to IP address error. We changed IP Address f the system as well as change the cluster related file . But forget to execute cmapplyconf command for apply the change,


Thanks for everyone's effort.


Aungshu