PVG disk replace problem

[Help 9]

I have a problem with a group of disks I've been working with.
I have an L-class server with two FC10 arrays attached. For this explanation I'll call them FC10 A & FC10 B. All the disks are part of VG01.
There is one logical volume (lvol1) spanning 10 of the disks using PVG. FC10 A has all ten grouped in one PVG. FC10 B has all ten disks in another PVG. The two PVG's are mirrored I.E. FC10 A is mirrored with FC10 B.
The problem we had was a server powerdown due to an air conditioning problem in the datacentre. When the server was powered back up, there was a faulty disk. The disk was replaced but the vgcfgrestore wouldn't work (it turned out we had a ghost disk). Using the normal procedure to remove a ghost disk, we reduced the mirror, reduced the disk from the volume group, moved lvmtab, vgscan etc...
We added the disk back to the volume group and all looked ok apart from the /etc/lvmpvg. Vgdisplay and lvmtab looked fine.
PVG1
c6t0d0
c4t1d0
c4t2d0
c4t3d0
...

PVG2
c4t0d0
c6t1d0
c6t2d0
c6t3d0
...

So the first disk in the PVG was from the wrong array. I edited the lvmpvg file so they were in the correct order but it wouldn't let me lvextend due to lack of space. After trying various different settings, I decided to change the /etc/lvmpvg to the state it was in (above). I ran lvextend -m 1 /dev/vg01/lvol1 PGV2 and the logical volume synchronised.
So the current state of the system is the disks are mirrored but instead of FC10 A being a mirror image of FC10 B, we have nine disks in FC10 A + 1 disk in FC10 B mirrored to nine disks in FC10 B + 1 disk in FC10 A.
So if we have a failed disk all is ok, but if we lose a disk shelf, we lose our data. So our resilience has shortened. Other than that, it looks messy in the lvmtab.
Can any of you tell me a way I can transfer c6t0d0 with c4t0d0 so they are on appropriate sides of the mirror? Remember they are in a PVG, so not standard LVM mirroring.
My thoughts are to "fail" disk c6t0d0, but will the c4t0d0 disk flip over to the correct side of the mirror if I do this?
regards

[Ivan Krastev]

We have the same issue with mirrors over 2 array. The sollution is to create a new -m 2 mirror over new PVG3, reduce 1st two and re-create it again with lvextend.

regards,
ivan

[Steven E. Protter]

Shalom,

Lots of good stuff in here:
http://www.docs.hp.com/en/7161/LVM_OLR_whitepaper.pdf

I would suggest a disk to disk backup using dd to insure no data loss

I think you plan is worth a try but hope you have a backup.

SEP

[Torsten.]

Consider this:

vgexport the volume group

# vgexport -p -v -s -m mapxx vgxx

swap the both disks, adjust the PVG file and import them again.

You'll need to allow the swapped disks to log on again using fcmsutil replace_dsk or just reboot the server.

IMHO this should work.

Make sure you have a backup.

;-)

[Torsten.]

Correction:

You need to do a real vgexport (without -p):

# vgexport -v -s -m mapxx vgxx

[Ganesan R]

Hi,

You can try this. Add one new LUN to both PVG. Just move the data from first disk to new LUN
PVG1 ->pvmove c6t0d0 --> new lun
PVG2 ->pvmore c4t0d0 --> new lun

Now both c6t0d0 and c4t0d0 will be free.
Take vgcfgbackup

edit lvmpvg and interchange the disks c6t0d0 and c4t0d0 to make it correct order.

Now again move back the data from new luns to these disks.

PVG1 ->pvmove new lun -->c4t0d0
PVG2 ->pvmore new lun -->c6t0d0

Later you can remove both the new luns from lvmpvg file

[Help 9]

Thanks for the ideas... I'm carrying out the work tomorrow morning, so I'll let you know how it goes.

[Torsten.]

Any results so far?

If I think about this - you should not have a problem even if a whole chassis fails, because you still have one side of the mirror available.

(for 1 disk this is just the "other side of the mirror").

[Help 9]

Hi Torsten,
We've had to delay the operation until they have a full backup sorted. They have a secondary server building in event worst case scenario loss of data.
We have thought about a different approach, I was wondering what you thought I.E. would it work?
# lvchange -s n /dev/vg01/lvol1
Would this remove the PVG allocation policy?
If so, would this allow us to edit the lvmpvg table to the correct order?
Then if we did the above, re-allocating strict policy would be all that's needed?
# lvchange -s g /dev/vg01/lvol1

We have a test system building to re-create the fault and try the various options.

Please give any comments on the process above to say why it would/wouldn't work.

Thanks
Ewan

[Torsten.]

"... Then if we did the above ..."

Not sure what you want to do after disabling the strict policy.


As said before, as long as both disks in question are mirrored 1:1 (consider to check this with lvdisplay); you can "safely" loose one of the chassis.

You have a problem if the disk in question is mirrored to a disk in the same chassis.