Cannot sync mirror because replacement disc too small

[Keith Jahn]

All discs are in HP 9000 SC10 cabinets. Disc /dev/dsk/c10t8d0 is part of a 3-disc mirrored set which are the only discs in /dev/vg02. This disc failed untidily and was replaced with a HP spare but fails to sync leaving 489 sectors unsynced. The mirror set contains one partition occupying the whole disc.
After much delving it turns out that the replacement disc is not 36.4 Gb but 32 Gb though all part numbers match!
See the size field for /dev/dsk/c10t8d0 "33554432 Kbytes" not "35566480 Kbytes" :

diskinfo /dev/rdsk/c6t8d0
SCSI describe of /dev/rdsk/c6t8d0:
vendor: FUJITSU
product id: MAJ3364MC
type: direct access
size: 35566480 Kbytes
bytes per sector: 512

diskinfo /dev/rdsk/c3t8d0
SCSI describe of /dev/rdsk/c3t8d0:
vendor: HP 36.4G
product id: ST336706LC
type: direct access
size: 35566480 Kbytes
bytes per sector: 512

diskinfo /dev/rdsk/c10t8d0
SCSI describe of /dev/rdsk/c10t8d0:
vendor: HP 36.4G
product id: ST336706LC
type: direct access
size: 33554432 Kbytes
bytes per sector: 512


Now! Trying to split the mirror fails. Running the VG with one disc missing fails a quorum check (there are only 3 discs in the VG).

I have spare discs available which are hopfully the right size. I am able to quiesce the filesystems and normally would do this before changing a disc in a so-called "hot pull" SC10.

Question: What is the recommended safest way to replace the bad disc bearing in mind that it is sound in "ioscan" but unsound in LVM?

[Steven E. Protter]

Shalom,

IMO you need to start over with the proper disk. The spare was probably mis-labeled or otherwise tainted.

I would get this current spare out of the volume group with vgreduce.

SEP

[Jayakrishnan G Naik · Accepted solution]

Hi

You can try the lvreduce-->vgreduce-->pvchange-->replacedisk-->vgcfgrestore-->mkboot-->pvchange-->vgsync-->lvlnboot -R method to replace disk


Hope the attached document will be very handy in similar situations.

Thanks & Regards
Jayakrishnan G Naik

[Keith Jahn]

Thanks.
If it will take the "lvreduce" while the mirrors are not synced and it will let me "vgreduce" below the VG quorum value I'm home and dry. I'll try it as soon as I can get downtime.

[Steven E. Protter]

Shalom again,

Once you are sure the disk has no data on it: pvdisplay -v you can use vgreduce to temporarily remove the disk from the volume group.

Doing this, and then replacing the disk followed by vgextend is a reliable way to insure that the new disk is recognized at the size you need.

SEP

[Anoop Sivan]

Hi,

Use pvdisplay -v /dev/dsk/cxtxdx (failed disk) to find the list of logical volumes using this disk.

Then remove LE extents from these pv's
use lvreduce

Next remove the failed pv from vg01
vgreduce -f vg01

Note: If pvdisplay fails, use the pvkey to reduce the failed disk from vg01.

[Keith Jahn]

Thanks Anoop Sivan.
I have replaced numerous failed discs on these systems (including this disc). The original replacement was complicated because the whole 3-disc mirror set failed with "quorum" errors because they are the only discs in Volume Group vg02. I don't have a spare slot to fit a fourth disc so I expect issues when trying to reduce vg02 from 3 discs to 2 discs.
This is the first time where I have a disc which is essentially sound but will not sync and where the quorum issue gave me so much pain. I will umount the single filesystem before starting the exercise in the knowledge that it will not mount until the disc is replaced.

[Anoop Sivan]

Hello Jahn,

Disable quorum for vg01 and try to remove the failed disks.

#vgchange -q n /dev/vg01

[Keith Jahn]

Hmm the "vgchange" command above gave a syntax error.

The final sequence was:

umount /partition

vgchange -a y -q n vg02

lvreduce -m 1 /dev/vg02/lvol1 /dev/dsk/c10t8d0

Remove old disc, ioscan, replace disc, ioscan.
Capacity still wrong in "diskinfo". Reboot server, capacity now correct in "diskinfo".
This also remounted partiton (without quorum issue) on a synced single mirror.

vgcfgrestore -n vg02 /dev/rdsk/c10t8d0

vgchange -a y vg02

lvextend -m 2 /dev/vg02/lvol1 /dev/dsk/c10t8d0


Thanks to all who helped.