Yet another Cluster lock missing - Full info provided

[rventura]

Ok, here is the error i am constantly getting:

4e2058e61c0140WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
4e2058e61c0140Until it is fixed, a single failure could
4e2058e61c0140cause all nodes in the cluster to crash

We are running a single node cluster for reason beyond my control, here is the output:

# cmviewcl -v

CLUSTER      STATUS
clsigma1     up

  NODE         STATUS       STATE
  unix-1       up           running

    Network_Parameters:
    INTERFACE    STATUS       PATH                NAME
    PRIMARY      up           0/1/2/0             lan0
    PRIMARY      up           0/4/1/0/6/0         lan1
    STANDBY      up           0/4/1/0/6/1         lan2

    PACKAGE      STATUS       STATE        AUTO_RUN     NODE
    pkg_roi      up           running      enabled      unix-1

      Policy_Parameters:
      POLICY_NAME     CONFIGURED_VALUE
      Failover        configured_node
      Failback        manual

      Node_Switching_Parameters:
      NODE_TYPE    STATUS       SWITCHING    NAME
      Primary      up           enabled      unix-1       (current)
      Alternate    down                      unix-2

  NODE         STATUS       STATE
  unix-2       down         unknown

    Network_Parameters:
    INTERFACE    STATUS       PATH                NAME
    PRIMARY      unknown      0/1/2/0             lan0
    STANDBY      unknown      0/4/1/0/6/1         lan2
    PRIMARY      unknown      0/4/1/0/6/0         lan1

Here is the ioscan:

# ioscan -fnC disk
Class     I  H/W Path       Driver  S/W State   H/W Type     Description
=========================================================================
disk      0  0/0/2/0.0.0.0  sdisk   CLAIMED     DEVICE       TEAC    DV-28E-N
                           /dev/dsk/c0t0d0   /dev/rdsk/c0t0d0
disk      1  0/1/1/0.0.0    sdisk   CLAIMED     DEVICE       HP 73.4GST373454LC
                           /dev/dsk/c2t0d0   /dev/rdsk/c2t0d0
disk      3  0/1/1/1.2.0    sdisk   CLAIMED     DEVICE       HP 73.4GST373454LC
                           /dev/dsk/c3t2d0   /dev/rdsk/c3t2d0
disk      2  0/3/1/0.8.0    sdisk   CLAIMED     DEVICE       HP 73.4GST373207LC
                           /dev/dsk/c4t8d0   /dev/rdsk/c4t8d0
disk      5  0/3/1/1.8.0    sdisk   CLAIMED     DEVICE       HP 73.4GMAT3073NC
                           /dev/dsk/c5t8d0   /dev/rdsk/c5t8d0

Can anyone provide any insight on this...?

Thanks!

[donna hofmeister]

anything else in syslog related to this disk?

what's diskinfo have to say about it?  or pvdisplay?

[rventura]

# pvdisplay /dev/dsk/c4t8d0
--- Physical volumes ---
PV Name                     /dev/dsk/c4t8d0
VG Name                     /dev/vg01
PV Status                   available
Allocatable                 yes
VGDA                        2
Cur LV                      4
PE Size (Mbytes)            4
Total PE                    17499
Free PE                     7874
Allocated PE                9625
Stale PE                    0
IO Timeout (Seconds)        default
Autoswitch                  On

# vgdisplay -v vg01
--- Volume groups ---
VG Name                     /dev/vg01
VG Write Access             read/write
VG Status                   available, exclusive
Max LV                      255
Cur LV                      5
Open LV                     4
Max PV                      16
Cur PV                      3
Act PV                      2
Max PE per PV               17501
VGDA                        4
PE Size (Mbytes)            4
Total PE                    34998
Alloc PE                    19275
Free PE                     15723
Total PVG                   0
Total Spare PVs             0
Total Spare PVs in use      0

   --- Logical volumes ---
   LV Name                     /dev/vg01/lvol1
   LV Status                   available/syncd
   LV Size (Mbytes)            25000
   Current LE                  6250
   Allocated PE                12500
   Used PV                     2

   LV Name                     /dev/vg01/lvol2
   LV Status                   available/syncd
   LV Size (Mbytes)            1000
   Current LE                  250
   Allocated PE                500
   Used PV                     2

   LV Name                     /dev/vg01/lvol3
   LV Status                   available/syncd
   LV Size (Mbytes)            2000
   Current LE                  500
   Allocated PE                1000
   Used PV                     2

   LV Name                     /dev/vg01/lvol4
   LV Status                   available/syncd
   LV Size (Mbytes)            10500
   Current LE                  2625
   Allocated PE                5250
   Used PV                     2


   --- Physical volumes ---
   PV Name                     /dev/dsk/c5t8d0
   PV Status                   available
   Total PE                    17499
   Free PE                     7849
   Autoswitch                  On

   PV Name                     /dev/dsk/c4t8d0
   PV Status                   available
   Total PE                    17499
   Free PE                     7874
   Autoswitch                  On

# grep '/dev/dsk/c4t8d0' syslog.log
Jul  1 15:26:15 unix-1 cmcld: Cluster lock disk: /dev/dsk/c4t8d0.
Jul  1 15:26:18 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul  1 15:27:33 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul  1 16:28:48 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
.
.
.
.
.
Jul 15 09:11:23 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 10:12:38 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 11:13:53 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 12:15:08 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 13:16:23 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 14:17:38 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 15:18:53 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!

All it mentions in syslog.log is the same error as in my first post:

Jul 15 15:18:53 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 15:18:53 unix-1 cmcld: Until it is fixed, a single failure could
Jul 15 15:18:53 unix-1 cmcld: cause all nodes in the cluster to crash

Thanks

[donna hofmeister]

in syslog...i was really more interested in knowing if you were getting any disk hardware errors.  You'll not be able to grep for cXtYdZ to find that out.

however since pvdisplay came back that seems to imply that the disk is still alive.  (again, diskinfo output would be nice see)

Let's determine if a disk has a cluster lock structure loaded.

Perform the command on a cluster lock disk. An example of this command follows:

 # echo '0x2084?4D' | adb /dev/dsk/c0t2d0
 2084:           1               4194145         1010590466      256
                 ^                  ^^^             ^^^          ^^^
            (lock flag)      (lock location)    (clusterID)  (activation/
          1=initialized,                                      configured mode)
          0=uninitialized  

The 1 in the second column indicates that this disk is a cluster lock disk.

[rventura]

Diskinfo

# diskinfo -v /dev/rdsk/c4t8d0
SCSI describe of /dev/rdsk/c4t8d0:
             vendor: HP 73.4G
         product id: ST373207LC
               type: direct access
               size: 71687369 Kbytes
   bytes per sector: 512
          rev level: HPC3
    blocks per disk: 143374738
        ISO version: 0
       ECMA version: 0
       ANSI version: 3
    removable media: no
    response format: 2
   (Additional inquiry bytes: (32)4b (33)54 (34)33 (35)45 (36)43 (37)4e (38)5a (39)20 (40)20 (41)20 (42)20 (43)20 (44)20 (45)20 (46)20 (47)20 (48)20 (49)20 (50)20 (51)f (52)0 (53)0 (54)60 (55)a (56)dc (57)b (58)59 (59)3 (60)1 (61)3 (62)22 (63)0 (64)0 (65)0 (66)0 (67)0 (68)0 (69)0 (70)0 (71)0 (72)0 (73)0 (74)0 (75)0 (76)0 (77)0 (78)0 (79)0 (80)0 (81)0 (82)0 (83)0 (84)0 (85)0 (86)0 (87)0 (88)0 (89)0 (90)0 (91)0 (92)43 (93)6f (94)70 (95)79 (96)72 (97)69 (98)67 (99)68 (100)74 (101)20 (102)28 (103)63 (104)29 (105)20 (106)32 (107)30 (108)30 (109)37 (110)20 (111)53 (112)65 (113)61 (114)67 (115)61 (116)74 (117)65 (118)20 (119)41 (120)6c (121)6c (122)20 (123)8 (124)8b (125)b9 (126)92 (127)0 (128)0 (129)2 (130)0 (131)7b (132)4 (133)20 (134)40 (135)7f (136)7f (137)6 (138)7c )

 Here is the second command 

echo '0x2084?4D' | adb /dev/dsk/c4t8d0
2084:           0               0               1706243227      256

I also tried it on a different disk and

# echo '0x2084?4D' | adb /dev/dsk/c5t8d0
2084:           0               0               1706243227      256

Does this mean there is no lock at all..?

Will this change if i start the other node..? Currently we are just running the cluster is a single node because of the ERP software, but if starting the other node fixes this, i can go ahead and do it, however, i dont think it will..

Any other ideas..?

Thanks

[rventura]

Also, there are no errors in syslog about any disk failing...

[donna hofmeister]

so -- we clearly see that the lock flag is not set -- and now the question becomes why?  (and i wouldn't expect c5t8d0 to have the flag set)

backing up to to vgdisplay, i see:

Cur LV                      5
Open LV                     4

Cur PV                      3
Act PV                      2

why is that?  what happened to this volume?  if you do an "ls -l" of /etc/lvmconf/, what are the dates for the vg01 file(s)?

[rventura]

# ls -l lvmconf
total 5984
----------   1 root       root             0 Dec 16  2005 lvm_lock
-rw-------   1 root       sys         330752 Sep 11  2008 vg00.conf
-rw-------   1 root       sys         330752 Apr 12  2007 vg00.conf.old
-rw-r--r--   1 root       sys            104 Sep 11  2008 vg00.mapfile
-rw-------   1 root       sys        1182720 Sep 11  2008 vg01.conf
-rw-------   1 root       sys        1182720 Apr 12  2007 vg01.conf.old
-rw-r--r--   1 root       sys             54 Sep 11  2008 vg01.mapfile

This happened: http://h30499.www3.hp.com/t5/LVM-and-VxVM/Mirrored-disk-replacement-refresher/td-p/4727551

Basically the disk went bad and it was replaced.

[donna hofmeister]

according to your old thread and the lvm data you've shown in this thread -- this volume group is still is in an undesireable state.

any chance you could start over from scratch?

[rventura]

yes, i can do that. But where do i start?