Operating System - HP-UX
1823415 Members
2658 Online
109655 Solutions
New Discussion

Re: Yet another Cluster lock missing - Full info provided

 
rventura
Frequent Advisor

Yet another Cluster lock missing - Full info provided

Ok, here is the error i am constantly getting:

 

4e2058e61c0140WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
4e2058e61c0140Until it is fixed, a single failure could
4e2058e61c0140cause all nodes in the cluster to crash

 

 

We are running a single node cluster for reason beyond my control, here is the output:

# cmviewcl -v

CLUSTER      STATUS
clsigma1     up

  NODE         STATUS       STATE
  unix-1       up           running

    Network_Parameters:
    INTERFACE    STATUS       PATH                NAME
    PRIMARY      up           0/1/2/0             lan0
    PRIMARY      up           0/4/1/0/6/0         lan1
    STANDBY      up           0/4/1/0/6/1         lan2

    PACKAGE      STATUS       STATE        AUTO_RUN     NODE
    pkg_roi      up           running      enabled      unix-1

      Policy_Parameters:
      POLICY_NAME     CONFIGURED_VALUE
      Failover        configured_node
      Failback        manual

      Node_Switching_Parameters:
      NODE_TYPE    STATUS       SWITCHING    NAME
      Primary      up           enabled      unix-1       (current)
      Alternate    down                      unix-2

  NODE         STATUS       STATE
  unix-2       down         unknown

    Network_Parameters:
    INTERFACE    STATUS       PATH                NAME
    PRIMARY      unknown      0/1/2/0             lan0
    STANDBY      unknown      0/4/1/0/6/1         lan2
    PRIMARY      unknown      0/4/1/0/6/0         lan1

 

Here is the ioscan:

# ioscan -fnC disk
Class     I  H/W Path       Driver  S/W State   H/W Type     Description
=========================================================================
disk      0  0/0/2/0.0.0.0  sdisk   CLAIMED     DEVICE       TEAC    DV-28E-N
                           /dev/dsk/c0t0d0   /dev/rdsk/c0t0d0
disk      1  0/1/1/0.0.0    sdisk   CLAIMED     DEVICE       HP 73.4GST373454LC
                           /dev/dsk/c2t0d0   /dev/rdsk/c2t0d0
disk      3  0/1/1/1.2.0    sdisk   CLAIMED     DEVICE       HP 73.4GST373454LC
                           /dev/dsk/c3t2d0   /dev/rdsk/c3t2d0
disk      2  0/3/1/0.8.0    sdisk   CLAIMED     DEVICE       HP 73.4GST373207LC
                           /dev/dsk/c4t8d0   /dev/rdsk/c4t8d0
disk      5  0/3/1/1.8.0    sdisk   CLAIMED     DEVICE       HP 73.4GMAT3073NC
                           /dev/dsk/c5t8d0   /dev/rdsk/c5t8d0

 

 

Can anyone provide any insight on this...?

 

Thanks!

 

17 REPLIES 17
donna hofmeister
Trusted Contributor

Re: Yet another Cluster lock missing - Full info provided

anything else in syslog related to this disk?

what's diskinfo have to say about it?  or pvdisplay?

rventura
Frequent Advisor

Re: Yet another Cluster lock missing - Full info provided

# pvdisplay /dev/dsk/c4t8d0
--- Physical volumes ---
PV Name                     /dev/dsk/c4t8d0
VG Name                     /dev/vg01
PV Status                   available
Allocatable                 yes
VGDA                        2
Cur LV                      4
PE Size (Mbytes)            4
Total PE                    17499
Free PE                     7874
Allocated PE                9625
Stale PE                    0
IO Timeout (Seconds)        default
Autoswitch                  On

 

# vgdisplay -v vg01
--- Volume groups ---
VG Name                     /dev/vg01
VG Write Access             read/write
VG Status                   available, exclusive
Max LV                      255
Cur LV                      5
Open LV                     4
Max PV                      16
Cur PV                      3
Act PV                      2
Max PE per PV               17501
VGDA                        4
PE Size (Mbytes)            4
Total PE                    34998
Alloc PE                    19275
Free PE                     15723
Total PVG                   0
Total Spare PVs             0
Total Spare PVs in use      0

   --- Logical volumes ---
   LV Name                     /dev/vg01/lvol1
   LV Status                   available/syncd
   LV Size (Mbytes)            25000
   Current LE                  6250
   Allocated PE                12500
   Used PV                     2

   LV Name                     /dev/vg01/lvol2
   LV Status                   available/syncd
   LV Size (Mbytes)            1000
   Current LE                  250
   Allocated PE                500
   Used PV                     2

   LV Name                     /dev/vg01/lvol3
   LV Status                   available/syncd
   LV Size (Mbytes)            2000
   Current LE                  500
   Allocated PE                1000
   Used PV                     2

   LV Name                     /dev/vg01/lvol4
   LV Status                   available/syncd
   LV Size (Mbytes)            10500
   Current LE                  2625
   Allocated PE                5250
   Used PV                     2


   --- Physical volumes ---
   PV Name                     /dev/dsk/c5t8d0
   PV Status                   available
   Total PE                    17499
   Free PE                     7849
   Autoswitch                  On

   PV Name                     /dev/dsk/c4t8d0
   PV Status                   available
   Total PE                    17499
   Free PE                     7874
   Autoswitch                  On

 

# grep '/dev/dsk/c4t8d0' syslog.log
Jul  1 15:26:15 unix-1 cmcld: Cluster lock disk: /dev/dsk/c4t8d0.
Jul  1 15:26:18 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul  1 15:27:33 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul  1 16:28:48 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
.
.
.
.
.
Jul 15 09:11:23 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 10:12:38 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 11:13:53 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 12:15:08 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 13:16:23 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 14:17:38 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 15:18:53 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!

 

All it mentions in syslog.log is the same error as in my first post:

Jul 15 15:18:53 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Jul 15 15:18:53 unix-1 cmcld: Until it is fixed, a single failure could
Jul 15 15:18:53 unix-1 cmcld: cause all nodes in the cluster to crash

 

 

Thanks

donna hofmeister
Trusted Contributor

Re: Yet another Cluster lock missing - Full info provided

in syslog...i was really more interested in knowing if you were getting any disk hardware errors.  You'll not be able to grep for cXtYdZ to find that out.

 

however since pvdisplay came back that seems to imply that the disk is still alive.  (again, diskinfo output would be nice see)

 

Let's determine if a disk has a cluster lock structure loaded.

 

Perform the command on a cluster lock disk. An example of this command follows:

 # echo '0x2084?4D' | adb /dev/dsk/c0t2d0
 2084:           1               4194145         1010590466      256
                 ^                  ^^^             ^^^          ^^^
            (lock flag)      (lock location)    (clusterID)  (activation/
          1=initialized,                                      configured mode)
          0=uninitialized  

The 1 in the second column indicates that this disk is a cluster lock disk.

rventura
Frequent Advisor

Re: Yet another Cluster lock missing - Full info provided

Diskinfo

# diskinfo -v /dev/rdsk/c4t8d0
SCSI describe of /dev/rdsk/c4t8d0:
             vendor: HP 73.4G
         product id: ST373207LC
               type: direct access
               size: 71687369 Kbytes
   bytes per sector: 512
          rev level: HPC3
    blocks per disk: 143374738
        ISO version: 0
       ECMA version: 0
       ANSI version: 3
    removable media: no
    response format: 2
   (Additional inquiry bytes: (32)4b (33)54 (34)33 (35)45 (36)43 (37)4e (38)5a (39)20 (40)20 (41)20 (42)20 (43)20 (44)20 (45)20 (46)20 (47)20 (48)20 (49)20 (50)20 (51)f (52)0 (53)0 (54)60 (55)a (56)dc (57)b (58)59 (59)3 (60)1 (61)3 (62)22 (63)0 (64)0 (65)0 (66)0 (67)0 (68)0 (69)0 (70)0 (71)0 (72)0 (73)0 (74)0 (75)0 (76)0 (77)0 (78)0 (79)0 (80)0 (81)0 (82)0 (83)0 (84)0 (85)0 (86)0 (87)0 (88)0 (89)0 (90)0 (91)0 (92)43 (93)6f (94)70 (95)79 (96)72 (97)69 (98)67 (99)68 (100)74 (101)20 (102)28 (103)63 (104)29 (105)20 (106)32 (107)30 (108)30 (109)37 (110)20 (111)53 (112)65 (113)61 (114)67 (115)61 (116)74 (117)65 (118)20 (119)41 (120)6c (121)6c (122)20 (123)8 (124)8b (125)b9 (126)92 (127)0 (128)0 (129)2 (130)0 (131)7b (132)4 (133)20 (134)40 (135)7f (136)7f (137)6 (138)7c )

 Here is the second command 

echo '0x2084?4D' | adb /dev/dsk/c4t8d0
2084:           0               0               1706243227      256

 

I also tried it on a different disk and

# echo '0x2084?4D' | adb /dev/dsk/c5t8d0
2084:           0               0               1706243227      256

 

Does this mean there is no lock at all..?

 

Will this change if i start the other node..? Currently we are just running the cluster is a single node because of the ERP software, but if starting the other node fixes this, i can go ahead and do it, however, i dont think it will..

 

Any other ideas..?

 

Thanks

rventura
Frequent Advisor

Re: Yet another Cluster lock missing - Full info provided

Also, there are no errors in syslog about any disk failing...
donna hofmeister
Trusted Contributor

Re: Yet another Cluster lock missing - Full info provided

so -- we clearly see that the lock flag is not set -- and now the question becomes why?  (and i wouldn't expect c5t8d0 to have the flag set)

 

backing up to to vgdisplay, i see:

Cur LV                      5
Open LV                     4

Cur PV 3 Act PV 2

why is that?  what happened to this volume?  if you do an "ls -l" of /etc/lvmconf/, what are the dates for the vg01 file(s)?

rventura
Frequent Advisor

Re: Yet another Cluster lock missing - Full info provided

# ls -l lvmconf
total 5984
----------   1 root       root             0 Dec 16  2005 lvm_lock
-rw-------   1 root       sys         330752 Sep 11  2008 vg00.conf
-rw-------   1 root       sys         330752 Apr 12  2007 vg00.conf.old
-rw-r--r--   1 root       sys            104 Sep 11  2008 vg00.mapfile
-rw-------   1 root       sys        1182720 Sep 11  2008 vg01.conf
-rw-------   1 root       sys        1182720 Apr 12  2007 vg01.conf.old
-rw-r--r--   1 root       sys             54 Sep 11  2008 vg01.mapfile

 

This happened: http://h30499.www3.hp.com/t5/LVM-and-VxVM/Mirrored-disk-replacement-refresher/td-p/4727551

 

Basically the disk went bad and it was replaced.

donna hofmeister
Trusted Contributor

Re: Yet another Cluster lock missing - Full info provided

according to your old thread and the lvm data you've shown in this thread -- this volume group is still is in an undesireable state.

 

any chance you could start over from scratch?

rventura
Frequent Advisor

Re: Yet another Cluster lock missing - Full info provided

yes, i can do that. But where do i start?
donna hofmeister
Trusted Contributor

Re: Yet another Cluster lock missing - Full info provided

ami correct in thinking that 'vgchange -a' does not work on your system?  what about the other side of the cluster -- does vgchange know about '-a'?

rventura
Frequent Advisor

Re: Yet another Cluster lock missing - Full info provided

Well, i believe this command will deactivate and activate the vg, therefore, i cant execute this command since the ERP data resides in that VG and people is using the ERP... Unless this is a non dirructive operation..?

 

Thanks

donna hofmeister
Trusted Contributor

Re: Yet another Cluster lock missing - Full info provided

a 'man vgchange' should tell you if '-a' is a valid option

rventura
Frequent Advisor

Re: Yet another Cluster lock missing - Full info provided

Yeah, the man page show the '-a' option.

I also read about the '-c' option, maybe i should had used that one instead of '-a'...?
donna hofmeister
Trusted Contributor

Re: Yet another Cluster lock missing - Full info provided

to remove the volume group, you'll need to do N lvremoves (one for each lvol), N vgreduces (one for each disk minus one) and one vgremove (to remove the last physical volume).

 

when you go to make the replacement vg, you'll want to start with a "pvcreate -f" to write fresh/untainted lvm headers onto each disk.

------------

that's the lovely theoretical part!

 

since these disks are visible to both nodes of your cluster -- if you touch these disks, the application WILL BE impacted.  so you'll need to arrange for application downtime!!

 

double and triple check that you've got backups (plural!!) of whatever is currently on this volume before you start anything!!

 

use this command to record the PVRA information of the disk that will be the cluster lock disk:

# echo "0x2008?4X" | adb /dev/dsk/cXtYdZ

following the vgimport of the map file on the 2nd node, issue this command (on the 2nd node) to make sure cXtYdZ is the same on both systems. (the values returned must match!).

 

cmapplyconf will distribute the SG information but also modify the LVM headers (the PVRA).

 

as a reminder -- mirroring does NOT replicate lvm headers.  when you have to replace a disk -- especially one that is part of a SG cluster you need to do a vgcfgrestore as well.

 

 

rventura
Frequent Advisor

Re: Yet another Cluster lock missing - Full info provided

Wont the PVCREATE destroy all my data in the drives..?
donna hofmeister
Trusted Contributor

Re: Yet another Cluster lock missing - Full info provided

it sure will....hence the recommendation to do multiple backups and plan for an application outtage.

Stephen Doud
Honored Contributor

Re: Yet another Cluster lock missing - Full info provided

You stated "We are running a single node cluster for reason beyond my control" hence the cluster lock functionality is not being used.


However, if you cannot afford the down-time but need to reinstall the cluster lock structure on disk, open a support case with the HP Support Center and request the cminitlock tool.  It has the ability to restore the cluster lock structure on the disk designated in the cluster binary file.