topic Re: Cluster Error in Operating System - HP-UX

Cluster Error

Henry Manda — Wed, 26 Aug 2009 05:22:53 GMT

I need urgent help!!
Im getting this error.

cmdisklockd[4268]: WARNING: Cluster lock on disk /dev/dsk/c12t0d2 is missing! Until it is fixed, a single failure could cause all nodes in the cluster to crash.

Re: Cluster Error

Michal Kapalka (mikap) — Wed, 26 Aug 2009 05:40:17 GMT

hi,

check this link there are lot of advices :

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=793216

mikap

Re: Cluster Error

avizen9 — Wed, 26 Aug 2009 05:41:59 GMT

Hello Henry,
you should check ioscan -fnC disk
and compare the out put with your cluster configuration,
there should be disk lock the path is /dev/dsk/c12t0d2, this disk must be persent

other commands to work around are

mstm
cstm
xstm

Re: Cluster Error

Lijeesh N G_1 — Wed, 26 Aug 2009 05:43:42 GMT

Hi,

==>What the following command outputs saying,

#ioscan -fnC disk from both the nodes,
#diskinfo /dev/rdsk/c12t0d2 from both the nodes
#cmviewcl -v

==>Check syslog for any errors

==>Did you performed any activity on the disk /dev/dsk/c12t0d2.??

Regards,
LIJEESH N G

Re: Cluster Error

Henry Manda — Wed, 26 Aug 2009 07:13:25 GMT

Here is the output.

# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
========================================================================
disk 17 0/0/4/0/0/0/0/4/0/0/0.1.1.0.0.0.1 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c7t0d1 /dev/rdsk/c7t0d1
/dev/dsk/c7t0d1s1 /dev/rdsk/c7t0d1s1
/dev/dsk/c7t0d1s2 /dev/rdsk/c7t0d1s2
/dev/dsk/c7t0d1s3 /dev/rdsk/c7t0d1s3
disk 18 0/0/4/0/0/0/0/4/0/0/0.1.1.0.0.0.2 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c7t0d2 /dev/rdsk/c7t0d2
disk 19 0/0/4/0/0/0/0/4/0/0/0.1.1.0.0.0.3 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c7t0d3 /dev/rdsk/c7t0d3
disk 20 0/0/4/0/0/0/0/4/0/0/0.1.1.0.0.0.4 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c7t0d4 /dev/rdsk/c7t0d4
disk 21 0/0/4/0/0/0/0/4/0/0/0.1.1.0.0.0.5 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c7t0d5 /dev/rdsk/c7t0d5
disk 22 0/0/4/0/0/0/0/4/0/0/0.1.2.0.0.0.1 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c9t0d1 /dev/rdsk/c9t0d1
/dev/dsk/c9t0d1s1 /dev/rdsk/c9t0d1s1
/dev/dsk/c9t0d1s2 /dev/rdsk/c9t0d1s2
/dev/dsk/c9t0d1s3 /dev/rdsk/c9t0d1s3
disk 23 0/0/4/0/0/0/0/4/0/0/0.1.2.0.0.0.2 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c9t0d2 /dev/rdsk/c9t0d2
disk 24 0/0/4/0/0/0/0/4/0/0/0.1.2.0.0.0.3 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c9t0d3 /dev/rdsk/c9t0d3
disk 25 0/0/4/0/0/0/0/4/0/0/0.1.2.0.0.0.4 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c9t0d4 /dev/rdsk/c9t0d4
disk 26 0/0/4/0/0/0/0/4/0/0/0.1.2.0.0.0.5 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c9t0d5 /dev/rdsk/c9t0d5
disk 1 0/0/6/0/0/0/0/4/0/0/0.1.1.0.0.0.1 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c12t0d1 /dev/rdsk/c12t0d1
/dev/dsk/c12t0d1s1 /dev/rdsk/c12t0d1s1
/dev/dsk/c12t0d1s2 /dev/rdsk/c12t0d1s2
/dev/dsk/c12t0d1s3 /dev/rdsk/c12t0d1s3
disk 5 0/0/6/0/0/0/0/4/0/0/0.1.1.0.0.0.2 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c12t0d2 /dev/rdsk/c12t0d2
disk 7 0/0/6/0/0/0/0/4/0/0/0.1.1.0.0.0.3 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c12t0d3 /dev/rdsk/c12t0d3
disk 9 0/0/6/0/0/0/0/4/0/0/0.1.1.0.0.0.4 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c12t0d4 /dev/rdsk/c12t0d4
disk 11 0/0/6/0/0/0/0/4/0/0/0.1.1.0.0.0.5 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c12t0d5 /dev/rdsk/c12t0d5
disk 2 0/0/6/0/0/0/0/4/0/0/0.1.2.0.0.0.1 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c14t0d1 /dev/rdsk/c14t0d1
/dev/dsk/c14t0d1s1 /dev/rdsk/c14t0d1s1
/dev/dsk/c14t0d1s2 /dev/rdsk/c14t0d1s2
/dev/dsk/c14t0d1s3 /dev/rdsk/c14t0d1s3
disk 13 0/0/6/0/0/0/0/4/0/0/0.1.2.0.0.0.2 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c14t0d2 /dev/rdsk/c14t0d2
disk 14 0/0/6/0/0/0/0/4/0/0/0.1.2.0.0.0.3 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c14t0d3 /dev/rdsk/c14t0d3
disk 15 0/0/6/0/0/0/0/4/0/0/0.1.2.0.0.0.4 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c14t0d4 /dev/rdsk/c14t0d4
disk 16 0/0/6/0/0/0/0/4/0/0/0.1.2.0.0.0.5 sdisk CLAIMED DEVICE HP HSV300
/dev/dsk/c14t0d5 /dev/rdsk/c14t0d5
disk 29 0/0/10/1/1.2.0 sdisk CLAIMED DEVICE Optiarc DVD RW AD- 5200A
/dev/dsk/c5t2d0 /dev/rdsk/c5t2d0
#
# diskinfo /dev/rdsk/c12t0d2
SCSI describe of /dev/rdsk/c12t0d2:
vendor: HP
product id: HSV300
type: direct access
size: 209715200 Kbytes
bytes per sector: 512
# cmviewcl -v

CLUSTER STATUS
KWFT_CLUSTER up

NODE STATUS STATE
kwftmain up running

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vgt24 /dev/dsk/c12t0d2 down

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up LinkAgg0 lan900
PRIMARY up 0/0/6/0/0/0/0/2/0/ lan2

PACKAGE STATUS STATE AUTO_RUN NODE
t24pkg up running enabled kwftmain

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Subnet up 192.168.0.0

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up enabled kwftmain (current)
Alternate up enabled kwftbackup

NODE STATUS STATE
kwftbackup up running

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vgt24 /dev/dsk/c14t0d2 down

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up LinkAgg0 lan900
PRIMARY up 1/0/6/0/0/0/0/2/0/ lan2

Re: Cluster Error

Lijeesh N G_1 — Thu, 27 Aug 2009 11:08:33 GMT

Hi,

Check with following command, paste the output,

#dd if=/dev/rdsk/c12t0d2 of=/dev/null bs=1024 count=30

#cmquerycl -v -n kwftmain -n kwftbackup -C testascii.conf

#more testascii.conf ==> check for the entry "Possible Cluster Lock Devices:"
and check whether it contains disk /dev/dsk/c12t0d2 ?

Regards,
LIJEESH N G

Re: Cluster Error

Vishu — Thu, 27 Aug 2009 15:46:08 GMT

Hi Henry,

you must have a failed or failing disk at /dev/dsk/c12t0d0. please paste the output of

# dd if=/dev/rdsk/c12t0d0 of=/dev/null bs=1024 count=1000

# pvdisplay /dev/dsk/c12t0d2

this error states that the cluster lock PV(the disk in question) is not accessible.

If it is faulty, then replace it and restore the configuration with vgcfgrestore command only. ELSE you need to re-applying the cluster config again to make that disk cluster lock aware.

Re: Cluster Error

Rita C Workman — Thu, 27 Aug 2009 15:52:00 GMT

Is your lockdisk in it's own volume group?
Is that volume group active?

Rita

Re: Cluster Error

melvyn burnard — Sun, 30 Aug 2009 16:35:17 GMT

"cmdisklockd[4268]: WARNING: Cluster lock on disk /dev/dsk/c12t0d2 is missing! Until it is fixed, a single failure could cause all nodes in the cluster to crash."

This is a warning message.
It is telling you that Serviceguard cannot see the Cluster Lock bit set on this physical device.
Are the other cluster nodes seeing the same issue?
Has anything been changed recently?
What version of SG and which SG patch have you got installed?

It may be that someone has run a command that caused the cluster lock bit to become unset.
The diskinfo command certainly shows that the device appears to be there.

Re: Cluster Error

Henry Manda — Mon, 31 Aug 2009 03:03:51 GMT

I managed to sort this way:

# cmdeleteconf -f
Checking current status
Completed the cluster deletion
# vgchange -a y /dev/vgt24
Activated volume group
Volume group "/dev/vgt24" has been successfully changed.
# cmapplyconf -C kwftconfig.ascii -P /etc/cmcluster/t24pkg/t24pkg.conf
cmapplyconf: Nonexistent file: kwftconfig.ascii.
# cmapplyconf -C /etc/cmclusterkwftconfig.ascii -P /etc/cmcluster/t24pkg
cmapplyconf: Nonexistent file: /etc/cmclusterkwftconfig.ascii.
# ig.ascii -P /etc/cmcluster/t24pkg/t24pkg.conf <
Note : a NODE_TIMEOUT value of 2000000 was found in line 136. This value
is recommended if the top priority is to reform the cluster as fast
as possible in case of failure. If the top priority is to minimize
reformations, consider using a higher setting. For more information see
the cluster configuration ASCII file or the Managing Serviceguard manual.
WARNING: Incorrect permissions for /etc/cmcluster/t24pkg (40777). Directory must be executable for owner, and not writable by group and others on node kwftbackup.
Minimum network configuration requirements for the cluster have
not been met. Minimum network configuration requirements are:
- 2 or more heartbeat networks OR
- 1 heartbeat network with local switch (HP-UX Only) OR
- 1 heartbeat network using APA with 2 trunk members (HP-UX Only) OR
- 1 heartbeat network using bonding (mode 1) with 2 slaves (Linux Only)
Adding node kwftbackup to cluster KWFT_CLUSTER
Adding node kwftmain to cluster KWFT_CLUSTER
Completed the cluster creation

#cmviewcl -v

CLUSTER STATUS
KWFT_CLUSTER up

NODE STATUS STATE
kwftmain up running

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vgt24 /dev/dsk/c12t0d2 up

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/0/6/0/0/0/0/2/0/ lan2
PRIMARY up LinkAgg0 lan900

PACKAGE STATUS STATE AUTO_RUN NODE
t24pkg up running enabled kwftmain

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Subnet up 192.168

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up enabled kwftmain (c
Alternate up enabled kwftbackup

NODE STATUS STATE
kwftbackup up running

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vgt24 /dev/dsk/c14t0d2 up

Network_Parameters:
INTERFACE STATUS PATH NA
PRIMARY up 1/0/6/0/0/0/0/2/0/ la
PRIMARY up LinkAgg0 la

Re: Cluster Error

melvyn burnard — Mon, 31 Aug 2009 08:02:24 GMT

So what is the point of people trying to help you, but you don't bother to respond to their threads asking for details so they can help?