LVM mirrored disk stopped syncing.

[rvent]

Hello,

A few months back i had a HDD crash (http://h30499.www3.hp.com/t5/LVM-and-VxVM/Mirrored-disk-replacement-refresher/m-p/4727551#M38388) everything was working fine, but all of the sudden i am getting messages about some PEs not syncing. Here is the lvdisplay:

# lvdisplay -v /dev/vg01/lvol2 | pg
--- Logical volumes ---
LV Name /dev/vg01/lvol2
VG Name /dev/vg01
LV Permission read/write
LV Status available/stale
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 1000
Current LE 250
Allocated PE 500
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c5t8d0 250 250
/dev/dsk/c4t8d0 250 250

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c5t8d0 06250 current /dev/dsk/c4t8d0 06250 stale
00001 /dev/dsk/c5t8d0 06251 current /dev/dsk/c4t8d0 06251 current
00002 /dev/dsk/c5t8d0 06252 current /dev/dsk/c4t8d0 06252 current
00003 /dev/dsk/c5t8d0 06253 current /dev/dsk/c4t8d0 06253 stale
00004 /dev/dsk/c5t8d0 06254 current /dev/dsk/c4t8d0 06254 current
00005 /dev/dsk/c5t8d0 06255 current /dev/dsk/c4t8d0 06255 current
00006 /dev/dsk/c5t8d0 06256 current /dev/dsk/c4t8d0 06256 current
00007 /dev/dsk/c5t8d0 06257 current /dev/dsk/c4t8d0 06257 current
00008 /dev/dsk/c5t8d0 06258 current /dev/dsk/c4t8d0 06258 current
00009 /dev/dsk/c5t8d0 06259 current /dev/dsk/c4t8d0 06259 current
00010 /dev/dsk/c5t8d0 06260 current /dev/dsk/c4t8d0 06260 stale
00011 /dev/dsk/c5t8d0 06261 current /dev/dsk/c4t8d0 06261 stale
00012 /dev/dsk/c5t8d0 06262 current /dev/dsk/c4t8d0 06262 stale
00013 /dev/dsk/c5t8d0 06263 current /dev/dsk/c4t8d0 06263 current


c4t8d0 is the disk that was replaced.


syslog.log shows this relevant message:

Mar 23 10:06:57 unix-1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
Mar 23 10:06:57 unix-1 cmcld: Until it is fixed, a single failure could
Mar 23 10:06:57 unix-1 cmcld: cause all nodes in the cluster to crash
Mar 23 10:06:58 unix-1 cmclconfd[3441]: Updated file /var/adm/cmcluster/frdump.cmcld.5 for node unix-1 (length = 512096).

I check that dump file and i see:
4d8a0c911c0133QUERY Cluster lock[0] ioctl succeeded.
4d8a0c911c0103Cluster lock[0] query operation failed: No such file or directory
4d8a0c911c0b31Automatic dump request is issued.
4d8a0c911c0b32fr_auto_dump_cmcld() queued an event
4d8a0c911c0140WARNING: Cluster lock on disk /dev/dsk/c4t8d0 is missing!
4d8a0c911c0140Until it is fixed, a single failure could
4d8a0c911c0140cause all nodes in the cluster to crash
4d8a0c911c0133After cluster lock query ioctl: lock[0] status=4
4d8a0c911a0b32fr_dump_cmcld_event_handler() get request
4d8a0c911a0b31Automatic dump is scheduled after 1000000 usec.
4d8a0c911d0151HB from node unix-2
4d8a0c911d0151Updated safety time to 854291761
4d8a0c911d0151HB to node unix-2, 14706
4d8a0c921c0b33Adding a worker thread with func=400230ca, arg=00000000
4d8a0c92120b33About to call worker func=400230ca, arg=00000000
4d8a0c92120b32cl_cf_thread() start
4d8a0c92120b32cl_cf_thread() get request
4d8a0c92120b32FR: start dump
4d8a0c92120b33Automatic dump is not scheduled now.

Weird thing is that there are no activity lights on the disk itself. I am running a "dd if=/dev/dsk/c4t8d0 of=/dev/null bs=64k" and no lights flash on the disk enclosure.

Any ideas...? Could the errors shown above be the result of those "stale" PEs...? would re-syncing all the Lvols get rid of the "stale" and perhaps fix the messages about the "cluster lock"

Thanks

[rvent]

Well, i think the above is the result of:

# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 3 0/1/1/1.2.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 2 0/3/1/0.8.0 sdisk NO_HW DEVICE HP 73.4GST373207LC
/dev/dsk/c4t8d0 /dev/rdsk/c4t8d0
disk 5 0/3/1/1.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t8d0 /dev/rdsk/c5t8d0


What the hell is going on...? I just replaced this disk...

The 2 servers connected to this array see the same thing...

[Torsten.]

Did you do all the steps after replacement (vgcfgrestore, vgchange -a e ... etc)?

Is the disk available in vgdisplay?

[Patrick Wallek]

Yes, that would do it. It appears as if that disk has failed again.

I would have it replaced asap.

[rvent]

Torsten:
Yes, i did all that once i replaced it. All this was done during December 2010.


I am on hold with HP Storage team, 4 hours response time should get me a replacement before 5PM CST.

Those drives have been good for the last 6 years, then 1 dies, it gets replaced then the same disk dies within 4 months.. They probably sent me a refurbished disk..

Thanks

[INH]

<
It means the disk /dev/rdsk/c4t8d0 is either dead or in verge of failing. I would get the disk replaced and perfrom steps accordingly to sync it.