- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- Re: Yet another Cluster lock missing - Full info p...
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Forums
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-15-2011 12:47 PM
07-15-2011 12:47 PM
Yet another Cluster lock missing - Full info provided
Ok, here is the error i am constantly getting:
4e2058e61c0140WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing! 4e2058e61c0140Until it is fixed, a single failure could 4e2058e61c0140cause all nodes in the cluster to crash
We are running a single node cluster for reason beyond my control, here is the output:
# cmviewcl -v CLUSTER STATUS clsigma1 up NODE STATUS STATE unix-1 up running Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY up 0/1/2/0 lan0 PRIMARY up 0/4/1/0/6/0 lan1 STANDBY up 0/4/1/0/6/1 lan2 PACKAGE STATUS STATE AUTO_RUN NODE pkg_roi up running enabled unix-1 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING NAME Primary up enabled unix-1 (current) Alternate down unix-2 NODE STATUS STATE unix-2 down unknown Network_Parameters: INTERFACE STATUS PATH NAME PRIMARY unknown 0/1/2/0 lan0 STANDBY unknown 0/4/1/0/6/1 lan2 PRIMARY unknown 0/4/1/0/6/0 lan1
Here is the ioscan:
# ioscan -fnC disk Class I H/W Path Driver S/W State H/W Type Description ========================================================================= disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N /dev/dsk/c0t0d0 /dev/rdsk/c0t0d0 disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC /dev/dsk/c2t0d0 /dev/rdsk/c2t0d0 disk 3 0/1/1/1.2.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC /dev/dsk/c3t2d0 /dev/rdsk/c3t2d0 disk 2 0/3/1/0.8.0 sdisk CLAIMED DEVICE HP 73.4GST373207LC /dev/dsk/c4t8d0 /dev/rdsk/c4t8d0 disk 5 0/3/1/1.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC /dev/dsk/c5t8d0 /dev/rdsk/c5t8d0
Can anyone provide any insight on this...?
Thanks!
- Tags:
- lock disk
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-15-2011 12:52 PM
07-15-2011 12:52 PM
Re: Yet another Cluster lock missing - Full info provided
anything else in syslog related to this disk?
what's diskinfo have to say about it? or pvdisplay?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-15-2011 02:07 PM
07-15-2011 02:07 PM
Re: Yet another Cluster lock missing - Full info provided
# pvdisplay /dev/dsk/c4t8d0 --- Physical volumes --- PV Name /dev/dsk/c4t8d0 VG Name /dev/vg01 PV Status available Allocatable yes VGDA 2 Cur LV 4 PE Size (Mbytes) 4 Total PE 17499 Free PE 7874 Allocated PE 9625 Stale PE 0 IO Timeout (Seconds) default Autoswitch On
# vgdisplay -v vg01 --- Volume groups --- VG Name /dev/vg01 VG Write Access read/write VG Status available, exclusive Max LV 255 Cur LV 5 Open LV 4 Max PV 16 Cur PV 3 Act PV 2 Max PE per PV 17501 VGDA 4 PE Size (Mbytes) 4 Total PE 34998 Alloc PE 19275 Free PE 15723 Total PVG 0 Total Spare PVs 0 Total Spare PVs in use 0 --- Logical volumes --- LV Name /dev/vg01/lvol1 LV Status available/syncd LV Size (Mbytes) 25000 Current LE 6250 Allocated PE 12500 Used PV 2 LV Name /dev/vg01/lvol2 LV Status available/syncd LV Size (Mbytes) 1000 Current LE 250 Allocated PE 500 Used PV 2 LV Name /dev/vg01/lvol3 LV Status available/syncd LV Size (Mbytes) 2000 Current LE 500 Allocated PE 1000 Used PV 2 LV Name /dev/vg01/lvol4 LV Status available/syncd LV Size (Mbytes) 10500 Current LE 2625 Allocated PE 5250 Used PV 2 --- Physical volumes --- PV Name /dev/dsk/c5t8d0 PV Status available Total PE 17499 Free PE 7849 Autoswitch On PV Name /dev/dsk/c4t8d0 PV Status available Total PE 17499 Free PE 7874 Autoswitch On
# grep '/dev/dsk/c4t8d0' syslog.log Jul 1 15:26:15 unix-1 cmcld: Cluster lock disk: /dev/dsk/c4t8d0. Jul 1 15:26:18 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing! Jul 1 15:27:33 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing! Jul 1 16:28:48 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing! . . . . . Jul 15 09:11:23 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing! Jul 15 10:12:38 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing! Jul 15 11:13:53 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing! Jul 15 12:15:08 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing! Jul 15 13:16:23 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing! Jul 15 14:17:38 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing! Jul 15 15:18:53 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
All it mentions in syslog.log is the same error as in my first post:
Jul 15 15:18:53 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing! Jul 15 15:18:53 unix-1 cmcld: Until it is fixed, a single failure could Jul 15 15:18:53 unix-1 cmcld: cause all nodes in the cluster to crash
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-16-2011 07:47 AM
07-16-2011 07:47 AM
Re: Yet another Cluster lock missing - Full info provided
in syslog...i was really more interested in knowing if you were getting any disk hardware errors. You'll not be able to grep for cXtYdZ to find that out.
however since pvdisplay came back that seems to imply that the disk is still alive. (again, diskinfo output would be nice see)
Let's determine if a disk has a cluster lock structure loaded.
Perform the command on a cluster lock disk. An example of this command follows:
# echo '0x2084?4D' | adb /dev/dsk/c0t2d0
2084: 1 4194145 1010590466 256
^ ^^^ ^^^ ^^^
(lock flag) (lock location) (clusterID) (activation/
1=initialized, configured mode)
0=uninitialized
The 1 in the second column indicates that this disk is a cluster lock disk.
- Tags:
- adb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2011 07:06 AM
07-18-2011 07:06 AM
Re: Yet another Cluster lock missing - Full info provided
Diskinfo
# diskinfo -v /dev/rdsk/c4t8d0 SCSI describe of /dev/rdsk/c4t8d0: vendor: HP 73.4G product id: ST373207LC type: direct access size: 71687369 Kbytes bytes per sector: 512 rev level: HPC3 blocks per disk: 143374738 ISO version: 0 ECMA version: 0 ANSI version: 3 removable media: no response format: 2 (Additional inquiry bytes: (32)4b (33)54 (34)33 (35)45 (36)43 (37)4e (38)5a (39)20 (40)20 (41)20 (42)20 (43)20 (44)20 (45)20 (46)20 (47)20 (48)20 (49)20 (50)20 (51)f (52)0 (53)0 (54)60 (55)a (56)dc (57)b (58)59 (59)3 (60)1 (61)3 (62)22 (63)0 (64)0 (65)0 (66)0 (67)0 (68)0 (69)0 (70)0 (71)0 (72)0 (73)0 (74)0 (75)0 (76)0 (77)0 (78)0 (79)0 (80)0 (81)0 (82)0 (83)0 (84)0 (85)0 (86)0 (87)0 (88)0 (89)0 (90)0 (91)0 (92)43 (93)6f (94)70 (95)79 (96)72 (97)69 (98)67 (99)68 (100)74 (101)20 (102)28 (103)63 (104)29 (105)20 (106)32 (107)30 (108)30 (109)37 (110)20 (111)53 (112)65 (113)61 (114)67 (115)61 (116)74 (117)65 (118)20 (119)41 (120)6c (121)6c (122)20 (123)8 (124)8b (125)b9 (126)92 (127)0 (128)0 (129)2 (130)0 (131)7b (132)4 (133)20 (134)40 (135)7f (136)7f (137)6 (138)7c )
Here is the second command
echo '0x2084?4D' | adb /dev/dsk/c4t8d0 2084: 0 0 1706243227 256
I also tried it on a different disk and
# echo '0x2084?4D' | adb /dev/dsk/c5t8d0 2084: 0 0 1706243227 256
Does this mean there is no lock at all..?
Will this change if i start the other node..? Currently we are just running the cluster is a single node because of the ERP software, but if starting the other node fixes this, i can go ahead and do it, however, i dont think it will..
Any other ideas..?
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2011 07:07 AM
07-18-2011 07:07 AM
Re: Yet another Cluster lock missing - Full info provided
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2011 07:23 AM
07-18-2011 07:23 AM
Re: Yet another Cluster lock missing - Full info provided
so -- we clearly see that the lock flag is not set -- and now the question becomes why? (and i wouldn't expect c5t8d0 to have the flag set)
backing up to to vgdisplay, i see:
Cur LV 5 Open LV 4
Cur PV 3 Act PV 2
why is that? what happened to this volume? if you do an "ls -l" of /etc/lvmconf/, what are the dates for the vg01 file(s)?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2011 07:53 AM
07-18-2011 07:53 AM
Re: Yet another Cluster lock missing - Full info provided
# ls -l lvmconf total 5984 ---------- 1 root root 0 Dec 16 2005 lvm_lock -rw------- 1 root sys 330752 Sep 11 2008 vg00.conf -rw------- 1 root sys 330752 Apr 12 2007 vg00.conf.old -rw-r--r-- 1 root sys 104 Sep 11 2008 vg00.mapfile -rw------- 1 root sys 1182720 Sep 11 2008 vg01.conf -rw------- 1 root sys 1182720 Apr 12 2007 vg01.conf.old -rw-r--r-- 1 root sys 54 Sep 11 2008 vg01.mapfile
This happened: http://h30499.www3.hp.com/t5/LVM-and-VxVM/Mirrored-disk-replacement-refresher/td-p/4727551
Basically the disk went bad and it was replaced.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2011 08:15 AM
07-18-2011 08:15 AM
Re: Yet another Cluster lock missing - Full info provided
according to your old thread and the lvm data you've shown in this thread -- this volume group is still is in an undesireable state.
any chance you could start over from scratch?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2011 08:56 AM
07-18-2011 08:56 AM
Re: Yet another Cluster lock missing - Full info provided
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2011 09:09 AM
07-18-2011 09:09 AM
Re: Yet another Cluster lock missing - Full info provided
ami correct in thinking that 'vgchange -a' does not work on your system? what about the other side of the cluster -- does vgchange know about '-a'?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2011 11:55 AM
07-18-2011 11:55 AM
Re: Yet another Cluster lock missing - Full info provided
Well, i believe this command will deactivate and activate the vg, therefore, i cant execute this command since the ERP data resides in that VG and people is using the ERP... Unless this is a non dirructive operation..?
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2011 12:48 PM
07-18-2011 12:48 PM
Re: Yet another Cluster lock missing - Full info provided
a 'man vgchange' should tell you if '-a' is a valid option
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2011 01:17 PM
07-18-2011 01:17 PM
Re: Yet another Cluster lock missing - Full info provided
I also read about the '-c' option, maybe i should had used that one instead of '-a'...?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2011 02:15 PM
07-18-2011 02:15 PM
Re: Yet another Cluster lock missing - Full info provided
to remove the volume group, you'll need to do N lvremoves (one for each lvol), N vgreduces (one for each disk minus one) and one vgremove (to remove the last physical volume).
when you go to make the replacement vg, you'll want to start with a "pvcreate -f" to write fresh/untainted lvm headers onto each disk.
------------
that's the lovely theoretical part!
since these disks are visible to both nodes of your cluster -- if you touch these disks, the application WILL BE impacted. so you'll need to arrange for application downtime!!
double and triple check that you've got backups (plural!!) of whatever is currently on this volume before you start anything!!
use this command to record the PVRA information of the disk that will be the cluster lock disk:
# echo "0x2008?4X" | adb /dev/dsk/cXtYdZ
following the vgimport of the map file on the 2nd node, issue this command (on the 2nd node) to make sure cXtYdZ is the same on both systems. (the values returned must match!).
cmapplyconf will distribute the SG information but also modify the LVM headers (the PVRA).
as a reminder -- mirroring does NOT replicate lvm headers. when you have to replace a disk -- especially one that is part of a SG cluster you need to do a vgcfgrestore as well.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2011 02:27 PM
07-18-2011 02:27 PM
Re: Yet another Cluster lock missing - Full info provided
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-19-2011 06:07 AM
07-19-2011 06:07 AM
Re: Yet another Cluster lock missing - Full info provided
it sure will....hence the recommendation to do multiple backups and plan for an application outtage.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-20-2011 01:00 PM
07-20-2011 01:00 PM
Re: Yet another Cluster lock missing - Full info provided
You stated "We are running a single node cluster for reason beyond my control" hence the cluster lock functionality is not being used.
However, if you cannot afford the down-time but need to reinstall the cluster lock structure on disk, open a support case with the HP Support Center and request the cminitlock tool. It has the ability to restore the cluster lock structure on the disk designated in the cluster binary file.