topic Re: Cluster lock disk problem in Operating System - Linux

Cluster lock disk problem

Dan Sargis — Thu, 21 Jun 2007 18:05:46 GMT

I think that I may have shot myself in the foot.

Awhile back I had to remove some SG packages from a cluster. One of the packages I removed was associated with the Cluster Lock Disk. I issued the following commands to remove each package:

cmdeleteconf -p
lvremove /dev//
vgremove /dev/

I notice now that in syslog.log, the following messages repeat hourly:

Jun 21 15:24:38 atlas02 cmcld: WARNING: Cluster lock disk /dev/dsk/c13t0d0 has f
ailed: I/O error
Jun 21 15:24:38 atlas02 cmcld: Until it is fixed, a single failure could
Jun 21 15:24:38 atlas02 cmcld: cause all nodes in the cluster to crash

The reported Cluster lock disk is one for which the volume group was removed.

I attempted to run cmquerycl to recreate the cluster but here is what I get:

atlas02:/var/adm/syslog# cmquerycl -v -C tibco.cluster -n atlas02 -n titan02

Begin checking the nodes...
Looking for other clusters ... Done
Gathering configuration information ........... Done
Warning: The disk at /dev/dsk/c13t0d0 on node atlas02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c13t0d3 on node atlas02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c13t0d4 on node atlas02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c13t0d5 on node atlas02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c13t0d6 on node atlas02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c13t0d7 on node atlas02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c13t1d0 on node atlas02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c13t1d1 on node atlas02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c13t1d3 on node atlas02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c12t0d0 on node titan02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c12t0d3 on node titan02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c12t0d4 on node titan02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c12t0d5 on node titan02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c12t0d6 on node titan02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c12t0d7 on node titan02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c12t1d0 on node titan02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c12t1d1 on node titan02 does not have an ID, or a
disk label.
Warning: The disk at /dev/dsk/c12t1d4 on node titan02 does not have an ID, or a
disk label.
Warning: Disks which do not have IDs cannot be included in the topology descript
ion.
Use pvcreate(1M) to initialize a disk for LVM or,
use vxdiskadm(1M) to initialize a disk for VxVM.
Error: Unable to determine a unique identifier for physical volume /dev/dsk/c12t
1d5 on node titan02. Use pvcreate to give the disk an identifier.
Warning: Network interface lan0 on node atlas02 couldn't talk to itself.
Warning: Network interface lan2 on node atlas02 couldn't talk to itself.
Warning: Network interface lan0 on node titan02 couldn't talk to itself.
Warning: Network interface lan3 on node titan02 couldn't talk to itself.
Failed to gather configuration information.
atlas02:/var/adm/syslog#

Obviously, these are the vg's that I removed.

How do I clean up after myself and reassign the Cluster lock disk to an existing volume?

Re: Cluster lock disk problem

Wouter Jagers — Fri, 22 Jun 2007 04:38:39 GMT

Hi,

In your ascii cluster configuration file, you should change the values for FIRST_CLUSTER_LOCK_VG and FIRST_CLUSTER_LOCK_PV to an existing cluster volume group and one of its physical devices respectively.

Keep in mind that applying this change requires the cluster to be down.

Cheers,
Wout

Re: Cluster lock disk problem

Steven E. Protter — Fri, 22 Jun 2007 10:51:20 GMT

Shalom,

Well, you need another cluster lock disk. Thats normally set in the cmquerycl stage of cluster building. It need not have any data on it or can have data on it, it matters not.

SEP

Re: Cluster lock disk problem

Dan Sargis — Fri, 22 Jun 2007 11:25:24 GMT

Gentlemen,

Thank you for both replies.

Since cmquerycl seems to fail (above), I would assume that simply modifying the cluster.ascii file on both cluster nodes such that FIRST_CLUSTER_LOCK_PV and FIRST_CLUSTER_LOCK_VG are valid, then bouncing the cluster would be sufficient?

That would be great if that is all that's required.

Please advise if there is more to consider than that.

Thanks again.

Dan

Re: Cluster lock disk problem

Wouter Jagers — Fri, 22 Jun 2007 14:21:52 GMT

Yep, that should do the trick.

Just make sure you have it defined for all your nodes and it's the same actual disk.

Make sure the cluster is down when you apply the configuration: I've seen the cmapplyconf complete without complaining, but the lockdisk change would not be applied. And without a valid lockdisk, shutting down a node would make the other node(s) go down as well, which is not what we'd expect from a high-availability solution :-)

Cheers,
Wout

Re: Cluster lock disk problem

skt_skt — Sat, 23 Jun 2007 22:41:40 GMT

Try using the same OLD cluster lock VG/same disk and so the system should detect the disk/VG is accessible now.

Re: Cluster lock disk problem

Serviceguard for Linux — Mon, 25 Jun 2007 00:48:14 GMT

Dan,

Since this is the Serviceguard for Linux forum, you may want to repost in the normal Serviceguard forum.

http://forums1.itrc.hp.com/service/forums/categoryhome.do?categoryId=210