Validating LVM mirrors

[David Hixson]

Forum,

Does anyone know of any tools or commands to determine the validity of a mirror'd lvol?

The problem is this: When a disk that is part of a mirrored lvol fails, it usualy isn't detected until you do something silly, like split the mirror. The problem is that once you split the mirror you have no way to recover the data.... Current SCSI patches are helping to detect these failures, but are not yet fool-proof.

Checking VG display to see if the extents are stale or current is all well and good, but what I'd like is a way to say that they are actually 'the same', not just that they are flagged that they should be the same.

A related question is: Does anyone know of any tools to map Logical extents to files? (I realize this would be technicaly VERY complex and computationaly intensive) There are times when it would be nice to know what files got corrupted after a disk starts to fail.

[Pedro Sousa]

That appears to be very dificult, but I think the Online Diagnostics is the best tool to check the all system status.
If you don't have it installed, I think it would be a great idea to try it.
good luck.

[Patrick Wallek]

To answer your last question first: I don't know of anything that will map LEs to files.

I also don't know of any way to make sure your mirrors are synced and everything is the same other than to perform a periodic vgsync or lvsync, but there is really not much point in doing that becuase LVM will keep your mirrors in sync automatically.

The way I verify that all my mirrors are synced and that I don't have any bad disks is by running a script every morning at 6AM that checks the status of everything. I do a 'vgdisplay -v' and grep for stale. If nothing shows up, then I am OK. I also send a tail of my syslog.log file and look at that. If I have a disk that has gone completely bad, then I will see a message in the syslog saying something to the effect that the disk has been POWERFAILED.

Checking and making sure that your mirrors are synced with vgdisplay is probably the easiest and most effective thing to do. LVM does a good job of keeping everything synced, which is what it is supposed to do.

Have a look at this thread to and look for my responses to see the script that I run every morning.

http://forums.itrc.hp.com/cm/QuestionAnswer/1,1150,0x04c86af52b04d5118fef0090279cd0f9,00.html

[David Hixson]

Pedro,

I've tried using the Online Diagnostics several times with mixed success. The HPUX systems I'm most concerned with have problems running some of the tools in OnlineDiag, and I'm not sure they have anything specific to validating mirror status.


Patrick,

The vgdisplay or lvdisplay status of 'stale' or 'sync' is just a flag set somewhere in LVM. It indicates that LVM put the same stuff in each block, but doesn't have a way to check that they are still the same (for example, if the disk is corrupting data).


The tool I'm hoping for would be something along the lines of a 'dd' from each side of the lvol running through 'diff' and hoping for an identical comparision. The problem is that there isn't a particularly direct way to do this because of how information is filtered through LVM.

[Pedro Sousa]

I had some problems with mirrored pvs, and since I use OnlineDiag I can detect it very easily, just by checking the syslog!

The vgdisplay/lvdisplay will work, but you can use the following "dd if=/dev/rdsk/cxtxdx of=/dev/null bs=xxx" and check the final output.
The output should be something like:
1855546+0 records in
1855546+0 records out
good luck.

[Thierry Poels_1]

Hi David,

Predictive should warn you in advance of any disk problems.
lvdisplay -v should also show some stale extents.

regards,
Thierry.

[Ajitkumar Rane]

I dont know if you like this solution or not.Basicaly I use to note down the H/W address for the original and and mirror copy. Just unplug the original copy and you system stays online on the mirror this confims the mirroing is valid. If you thinks this is a stunt you can shutdown the system unplug the disk you want to test and boot the system in lq mode if the system comes up fine your mirroring is working and then you can reconnect the disk and vgsync the VG.I use to do this task when I was working as a CE. But do this stunt with a full backup. Also I had never faced any problems testing this way actuall its a real life scenario.

Good Luck

[Mark Mitchell]

I have a scrip that basically does this,
vgdisplay -v vg00
and you get these 2 disks back with the first being the live disk and the second being the mirror.
/dev/dsk/c0t5d0
/dev/dsk/c0t8d0
Now if you have different size disks you will have to do a better comparison.
So then I do a
pvdisplay -v /dev/dsk/c0t5d0 |tee /tmp/disk1
pvdisplay -v /dev/dsk/c0t8d0 |tee /tmp/disk2
This outlines all of the extents and the vols that they belong to, then I do a diff
diff disk1 disk2 | tee /tmp/difference
the only difference you should see here is that there are 2 disk addresses

[David Hixson]

Thanks Mark,

Another interesting way to check and see that both sides are marked current under LVM. However the core problem is finding out when the actual data on the drives differ and the state on LVM is still showing 'current'. I expect that I'll eventualy have to hack together a 'reproduceable test case' and take it to WTEC.

[Thierry Poels_1]

Hi,
an other option you have is "vgdisplay -v VGxx" and check for stale logical volumes.
(or lvdisplay -v)
regards,
Thierry.

[Thierry Poels_1]

even have an example (crashed disk on a 9.04 system :( )

LV Name /dev/vg01/lvol8
LV Status available/syncd
LV Size (Mbytes) 200
Current LE 50
Allocated PE 100
Used PV 2

LV Name /dev/vg01/lvol7
LV Status available/stale <---
LV Size (Mbytes) 300
Current LE 75
Allocated PE 150
Used PV 1