topic Yet another Cluster lock missing - Full info provided in Operating System - HP-UX

Yet another Cluster lock missing - Full info provided

rventura — Fri, 15 Jul 2011 19:47:26 GMT

Ok, here is the error i am constantly getting:

4e2058e61c0140WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
4e2058e61c0140Until it is fixed, a single failure could
4e2058e61c0140cause all nodes in the cluster to crash

We are running a single node cluster for reason beyond my control, here is the output:

# cmviewcl -v

CLUSTER      STATUS
clsigma1     up

  NODE         STATUS       STATE
  unix-1       up           running

    Network_Parameters:
    INTERFACE    STATUS       PATH                NAME
    PRIMARY      up           0/1/2/0             lan0
    PRIMARY      up           0/4/1/0/6/0         lan1
    STANDBY      up           0/4/1/0/6/1         lan2

    PACKAGE      STATUS       STATE        AUTO_RUN     NODE
    pkg_roi      up           running      enabled      unix-1

      Policy_Parameters:
      POLICY_NAME     CONFIGURED_VALUE
      Failover        configured_node
      Failback        manual

      Node_Switching_Parameters:
      NODE_TYPE    STATUS       SWITCHING    NAME
      Primary      up           enabled      unix-1       (current)
      Alternate    down                      unix-2

  NODE         STATUS       STATE
  unix-2       down         unknown

    Network_Parameters:
    INTERFACE    STATUS       PATH                NAME
    PRIMARY      unknown      0/1/2/0             lan0
    STANDBY      unknown      0/4/1/0/6/1         lan2
    PRIMARY      unknown      0/4/1/0/6/0         lan1

Here is the ioscan:

# ioscan -fnC disk
Class     I  H/W Path       Driver  S/W State   H/W Type     Description
=========================================================================
disk      0  0/0/2/0.0.0.0  sdisk   CLAIMED     DEVICE       TEAC    DV-28E-N
                           /dev/dsk/c0t0d0   /dev/rdsk/c0t0d0
disk      1  0/1/1/0.0.0    sdisk   CLAIMED     DEVICE       HP 73.4GST373454LC
                           /dev/dsk/c2t0d0   /dev/rdsk/c2t0d0
disk      3  0/1/1/1.2.0    sdisk   CLAIMED     DEVICE       HP 73.4GST373454LC
                           /dev/dsk/c3t2d0   /dev/rdsk/c3t2d0
disk      2  0/3/1/0.8.0    sdisk   CLAIMED     DEVICE       HP 73.4GST373207LC
                           /dev/dsk/c4t8d0   /dev/rdsk/c4t8d0
disk      5  0/3/1/1.8.0    sdisk   CLAIMED     DEVICE       HP 73.4GMAT3073NC
                           /dev/dsk/c5t8d0   /dev/rdsk/c5t8d0

Can anyone provide any insight on this...?

Thanks!

Re: Yet another Cluster lock missing - Full info provided

donna hofmeister — Fri, 15 Jul 2011 19:52:10 GMT

anything else in syslog related to this disk?

what's diskinfo have to say about it? or pvdisplay?

Re: Yet another Cluster lock missing - Full info provided

rventura — Fri, 15 Jul 2011 21:07:19 GMT

# pvdisplay /dev/dsk/c4t8d0
--- Physical volumes ---
PV Name                     /dev/dsk/c4t8d0
VG Name                     /dev/vg01
PV Status                   available
Allocatable                 yes
VGDA                        2
Cur LV                      4
PE Size (Mbytes)            4
Total PE                    17499
Free PE                     7874
Allocated PE                9625
Stale PE                    0
IO Timeout (Seconds)        default
Autoswitch                  On

# vgdisplay -v vg01
--- Volume groups ---
VG Name                     /dev/vg01
VG Write Access             read/write
VG Status                   available, exclusive
Max LV                      255
Cur LV                      5
Open LV                     4
Max PV                      16
Cur PV                      3
Act PV                      2
Max PE per PV               17501
VGDA                        4
PE Size (Mbytes)            4
Total PE                    34998
Alloc PE                    19275
Free PE                     15723
Total PVG                   0
Total Spare PVs             0
Total Spare PVs in use      0

   --- Logical volumes ---
   LV Name                     /dev/vg01/lvol1
   LV Status                   available/syncd
   LV Size (Mbytes)            25000
   Current LE                  6250
   Allocated PE                12500
   Used PV                     2

   LV Name                     /dev/vg01/lvol2
   LV Status                   available/syncd
   LV Size (Mbytes)            1000
   Current LE                  250
   Allocated PE                500
   Used PV                     2

   LV Name                     /dev/vg01/lvol3
   LV Status                   available/syncd
   LV Size (Mbytes)            2000
   Current LE                  500
   Allocated PE                1000
   Used PV                     2

   LV Name                     /dev/vg01/lvol4
   LV Status                   available/syncd
   LV Size (Mbytes)            10500
   Current LE                  2625
   Allocated PE                5250
   Used PV                     2


   --- Physical volumes ---
   PV Name                     /dev/dsk/c5t8d0
   PV Status                   available
   Total PE                    17499
   Free PE                     7849
   Autoswitch                  On

   PV Name                     /dev/dsk/c4t8d0
   PV Status                   available
   Total PE                    17499
   Free PE                     7874
   Autoswitch                  On

# grep '/dev/dsk/c4t8d0' syslog.log
Jul  1 15:26:15 unix-1 cmcld: Cluster lock disk: /dev/dsk/c4t8d0.
Jul  1 15:26:18 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul  1 15:27:33 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul  1 16:28:48 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
.
.
.
.
.
Jul 15 09:11:23 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 10:12:38 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 11:13:53 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 12:15:08 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 13:16:23 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 14:17:38 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 15:18:53 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!

All it mentions in syslog.log is the same error as in my first post:

Jul 15 15:18:53 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 15:18:53 unix-1 cmcld: Until it is fixed, a single failure could
Jul 15 15:18:53 unix-1 cmcld: cause all nodes in the cluster to crash

Thanks

Re: Yet another Cluster lock missing - Full info provided

donna hofmeister — Sat, 16 Jul 2011 14:47:49 GMT

in syslog...i was really more interested in knowing if you were getting any disk hardware errors. You'll not be able to grep for cXtYdZ to find that out.

however since pvdisplay came back that seems to imply that the disk is still alive. (again, diskinfo output would be nice see)

Let's determine if a disk has a cluster lock structure loaded.

Perform the command on a cluster lock disk. An example of this command follows:

# echo '0x2084?4D' | adb /dev/dsk/c0t2d0
2084:           1               4194145         1010590466      256
                 ^                  ^^^             ^^^          ^^^
            (lock flag)      (lock location)    (clusterID)  (activation/
          1=initialized,                                      configured mode)
          0=uninitialized

The 1 in the second column indicates that this disk is a cluster lock disk.

Re: Yet another Cluster lock missing - Full info provided

rventura — Mon, 18 Jul 2011 14:06:26 GMT

Diskinfo

# diskinfo -v /dev/rdsk/c4t8d0
SCSI describe of /dev/rdsk/c4t8d0:
             vendor: HP 73.4G
         product id: ST373207LC
               type: direct access
               size: 71687369 Kbytes
   bytes per sector: 512
          rev level: HPC3
    blocks per disk: 143374738
        ISO version: 0
       ECMA version: 0
       ANSI version: 3
    removable media: no
    response format: 2
   (Additional inquiry bytes: (32)4b (33)54 (34)33 (35)45 (36)43 (37)4e (38)5a (39)20 (40)20 (41)20 (42)20 (43)20 (44)20 (45)20 (46)20 (47)20 (48)20 (49)20 (50)20 (51)f (52)0 (53)0 (54)60 (55)a (56)dc (57)b (58)59 (59)3 (60)1 (61)3 (62)22 (63)0 (64)0 (65)0 (66)0 (67)0 (68)0 (69)0 (70)0 (71)0 (72)0 (73)0 (74)0 (75)0 (76)0 (77)0 (78)0 (79)0 (80)0 (81)0 (82)0 (83)0 (84)0 (85)0 (86)0 (87)0 (88)0 (89)0 (90)0 (91)0 (92)43 (93)6f (94)70 (95)79 (96)72 (97)69 (98)67 (99)68 (100)74 (101)20 (102)28 (103)63 (104)29 (105)20 (106)32 (107)30 (108)30 (109)37 (110)20 (111)53 (112)65 (113)61 (114)67 (115)61 (116)74 (117)65 (118)20 (119)41 (120)6c (121)6c (122)20 (123)8 (124)8b (125)b9 (126)92 (127)0 (128)0 (129)2 (130)0 (131)7b (132)4 (133)20 (134)40 (135)7f (136)7f (137)6 (138)7c )

Here is the second command

echo '0x2084?4D' | adb /dev/dsk/c4t8d0
2084:           0               0               1706243227      256

I also tried it on a different disk and

# echo '0x2084?4D' | adb /dev/dsk/c5t8d0
2084:           0               0               1706243227      256

Does this mean there is no lock at all..?

Will this change if i start the other node..? Currently we are just running the cluster is a single node because of the ERP software, but if starting the other node fixes this, i can go ahead and do it, however, i dont think it will..

Any other ideas..?

Thanks

Re: Yet another Cluster lock missing - Full info provided

rventura — Mon, 18 Jul 2011 14:07:39 GMT

Also, there are no errors in syslog about any disk failing...

Re: Yet another Cluster lock missing - Full info provided

donna hofmeister — Mon, 18 Jul 2011 14:23:24 GMT

so -- we clearly see that the lock flag is not set -- and now the question becomes why? (and i wouldn't expect c5t8d0 to have the flag set)

backing up to to vgdisplay, i see:

Cur LV                      5
Open LV                     4


Cur PV                      3
Act PV                      2

why is that? what happened to this volume? if you do an "ls -l" of /etc/lvmconf/, what are the dates for the vg01 file(s)?

Re: Yet another Cluster lock missing - Full info provided

rventura — Mon, 18 Jul 2011 14:53:39 GMT

# ls -l lvmconf
total 5984
----------   1 root       root             0 Dec 16  2005 lvm_lock
-rw-------   1 root       sys         330752 Sep 11  2008 vg00.conf
-rw-------   1 root       sys         330752 Apr 12  2007 vg00.conf.old
-rw-r--r--   1 root       sys            104 Sep 11  2008 vg00.mapfile
-rw-------   1 root       sys        1182720 Sep 11  2008 vg01.conf
-rw-------   1 root       sys        1182720 Apr 12  2007 vg01.conf.old
-rw-r--r--   1 root       sys             54 Sep 11  2008 vg01.mapfile

This happened: http://h30499.www3.hp.com/t5/LVM-and-VxVM/Mirrored-disk-replacement-refresher/td-p/4727551

Basically the disk went bad and it was replaced.

Re: Yet another Cluster lock missing - Full info provided

donna hofmeister — Mon, 18 Jul 2011 15:15:28 GMT

according to your old thread and the lvm data you've shown in this thread -- this volume group is still is in an undesireable state.

any chance you could start over from scratch?

Re: Yet another Cluster lock missing - Full info provided

rventura — Mon, 18 Jul 2011 15:56:44 GMT

yes, i can do that. But where do i start?

Re: Yet another Cluster lock missing - Full info provided

donna hofmeister — Mon, 18 Jul 2011 16:09:29 GMT

ami correct in thinking that 'vgchange -a' does not work on your system? what about the other side of the cluster -- does vgchange know about '-a'?

Re: Yet another Cluster lock missing - Full info provided

rventura — Mon, 18 Jul 2011 18:55:55 GMT

Well, i believe this command will deactivate and activate the vg, therefore, i cant execute this command since the ERP data resides in that VG and people is using the ERP... Unless this is a non dirructive operation..?

Thanks

Re: Yet another Cluster lock missing - Full info provided

donna hofmeister — Mon, 18 Jul 2011 19:48:27 GMT

a 'man vgchange' should tell you if '-a' is a valid option

Re: Yet another Cluster lock missing - Full info provided

rventura — Mon, 18 Jul 2011 20:17:56 GMT

Yeah, the man page show the '-a' option.

I also read about the '-c' option, maybe i should had used that one instead of '-a'...?

Re: Yet another Cluster lock missing - Full info provided

donna hofmeister — Mon, 18 Jul 2011 21:15:21 GMT

to remove the volume group, you'll need to do N lvremoves (one for each lvol), N vgreduces (one for each disk minus one) and one vgremove (to remove the last physical volume).

when you go to make the replacement vg, you'll want to start with a "pvcreate -f" to write fresh/untainted lvm headers onto each disk.

------------

that's the lovely theoretical part!

since these disks are visible to both nodes of your cluster -- if you touch these disks, the application WILL BE impacted. so you'll need to arrange for application downtime!!

double and triple check that you've got backups (plural!!) of whatever is currently on this volume before you start anything!!

use this command to record the PVRA information of the disk that will be the cluster lock disk:

# echo "0x2008?4X" | adb /dev/dsk/cXtYdZ

following the vgimport of the map file on the 2nd node, issue this command (on the 2nd node) to make sure cXtYdZ is the same on both systems. (the values returned must match!).

cmapplyconf will distribute the SG information but also modify the LVM headers (the PVRA).

as a reminder -- mirroring does NOT replicate lvm headers. when you have to replace a disk -- especially one that is part of a SG cluster you need to do a vgcfgrestore as well.

Re: Yet another Cluster lock missing - Full info provided

rventura — Mon, 18 Jul 2011 21:27:01 GMT

Wont the PVCREATE destroy all my data in the drives..?

Re: Yet another Cluster lock missing - Full info provided

donna hofmeister — Tue, 19 Jul 2011 13:07:41 GMT

it sure will....hence the recommendation to do multiple backups and plan for an application outtage.

Re: Yet another Cluster lock missing - Full info provided

Stephen Doud — Wed, 20 Jul 2011 20:00:10 GMT

You stated "We are running a single node cluster for reason beyond my control" hence the cluster lock functionality is not being used.

However, if you cannot afford the down-time but need to reinstall the cluster lock structure on disk, open a support case with the HP Support Center and request the cminitlock tool. It has the ability to restore the cluster lock structure on the disk designated in the cluster binary file.