replaced incorrect root disk

[Vanja]

Got a critical issue.

I have a v2500 with 2 jamaicas attached and a bunch of other ldevs from xp arrays. vg00 is mirrored from c2 to c3. c2t1d0, c2t2d0, c2t3d0, and c2t4d0 come from one jamaica. c3t3d0, c3t4d0, c3t5d0, and c3t6d0 come from another jamaica. Here's the output of vgdisplay -v vg00:

VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 9
Open LV 9
Max PV 16
Cur PV 8
Act PV 8
Max PE per PV 2500
VGDA 16
PE Size (Mbytes) 4
Total PE 8184
Alloc PE 5699
Free PE 2485
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/stale
LV Size (Mbytes) 240
Current LE 60
Allocated PE 120
Used PV 2

LV Name /dev/vg00/lvol2
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 512
Allocated PE 1024
Used PV 2

LV Name /dev/vg00/lvol3
LV Status available/stale
LV Size (Mbytes) 512
Current LE 128
Allocated PE 256
Used PV 2

LV Name /dev/vg00/lvol4
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 256
Allocated PE 512
Used PV 2

LV Name /dev/vg00/lvol5
LV Status available/stale
LV Size (Mbytes) 1432
Current LE 358
Allocated PE 716
Used PV 4

LV Name /dev/vg00/lvol6
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 256
Allocated PE 512
Used PV 2

LV Name /dev/vg00/lvol7
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 256
Allocated PE 512
Used PV 2

LV Name /dev/vg00/lvol8
LV Status available/stale
LV Size (Mbytes) 2048
Current LE 512
Allocated PE 1024
Used PV 4

LV Name /dev/vg00/dup_cpm
LV Status available/syncd
LV Size (Mbytes) 4092
Current LE 1023
Allocated PE 1023
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c3t6d0
PV Status available
Total PE 1023
Free PE 0
Autoswitch On

PV Name /dev/dsk/c3t5d0
PV Status available
Total PE 1023
Free PE 0
Autoswitch On

PV Name /dev/dsk/c3t3d0
PV Status available
Total PE 1023
Free PE 0
Autoswitch On

PV Name /dev/dsk/c2t2d0
PV Status available
Total PE 1023
Free PE 731
Autoswitch On

PV Name /dev/dsk/c2t1d0
PV Status available
Total PE 1023
Free PE 1023
Autoswitch On

PV Name /dev/dsk/c2t4d0
PV Status unavailable
Total PE 1023
Free PE 0
Autoswitch On

PV Name /dev/dsk/c2t3d0
PV Status available
Total PE 1023
Free PE 0
Autoswitch On

PV Name /dev/dsk/c3t4d0
PV Status available
Total PE 1023
Free PE 731
Autoswitch On


I have a stale extent on c3t4d0. I called in to get this replaced and the engineer replaced c2t4d0 instead of c3t4d0. I'm getting the lvmrec error now on c2t4d0. I have receded c2t4d0 now. What should I do at this point???
c3t6d0 and c2t4d0 are the mirrored boot disks. I know I can't readd the old c2t4d0 disk because then the lv info wouldn't be in sync. Should I reduce c2t4d0, and then remirror? I'm not sure what the best option would be in order not to cause any corruption.

Here'e the output of lvlnboot -Rv after receding c2t4d0:

goliath# lvlnboot -Rv
lvlnboot: Logical Volume has no extents.
lvlnboot: Logical Volume has no extents.
lvlnboot: Logical Volume has no extents.

Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c3t6d0 (1/0/0.6.0) -- Boot Disk
/dev/dsk/c3t5d0 (1/0/0.5.0)
/dev/dsk/c3t3d0 (1/0/0.3.0)
/dev/dsk/c2t2d0 (0/3/0.2.0)
/dev/dsk/c2t1d0 (0/3/0.1.0)
/dev/dsk/c2t4d0 (0/3/0.4.0)
/dev/dsk/c2t3d0 (0/3/0.3.0)
/dev/dsk/c3t4d0 (1/0/0.4.0)
Boot: lvol1 on: /dev/dsk/c3t6d0
Root: lvol3 on: /dev/dsk/c3t6d0
Swap: lvol2 on: /dev/dsk/c3t6d0
Dump: lvol2 on: /dev/dsk/c3t6d0, 0

vgcfgbackup: Unable to read the physical volume.: No such device or address
vgcfgbackup: Invalid LVMREC on Physical Volume /dev/rdsk/c2t4d0

Here's the output of the setboot command:

goliath# setboot
Primary bootpath : 0/3/0.4.0
Alternate bootpath : 1/0/0.6.0

Autoboot is ON (enabled)
Autosearch is OFF (disabled)


Any pointers and help would be greatly appreciated!
Thanks!
Vanja

[Florian Heigl (new acc)]

I can't really give You good hints any more, it's friday evening and my brain is quite too wiped out already for giving a 100% sure hint.

- how do You want to go about the stale PE???


- vgcfgrestore will fix the LVMREC issue, but in my opinion will interfere with the LIF area. OTOH You already are down to one bootable disk anyway without a working c2t4d0, so You could try.

I don't know how well LVM will manage consistency if You put the old disk back in.

I'd put it aside for now, count the stale pe, do an ignite *backup*, and then assess if/what data is missing at the moment.
possibly one could recover it from the old c2t4d0, i.e. vgimporting it in low-quorum mode as a new vg + read-only.

I just can't help You fix it today :)
If You've got a mission critical contract, call in HP and let them handle it, just to be sure. I'll try to check in ITRC tomorrow, hopefully You'll have some more promising answers from others by then!

[John Dvorchak · Accepted solution]

I would treat it like a failed mirror drive and replace the disk.


Do a vgcfgrestore -n /dev/vg00 /dev/rdsk/c2t4d0

then a vgsync on vg00.

Then do a lvlnboot -Rv to see that it shows up as a boot disk if it was a boot disk before.

[Ammar_4]

Replace a Non-Mirrored Disk
#vgcfgrestore -n /dev/vgXX /dev/rdsk/cxtxdx ; volume group configuration restore

# vgchange -a y /dev/vgXX ; change volume group to available (-a y)

# newfs -F fstype /dev/vgXX/rlvolx ; create filesystem for every lvol on physical volume

# mount /mountpointname ; mount every new filesystem


Notes:

Confirm you have /etc/lvmconf/vgXX.conf

[Devender Khatana]

Hi,

Reputting in the c2t4d0 and issing a vgsync command will sync the disk with other copy. Once the syncing is over you should be able to replace the actual disk. Follow following procedure for replacing it.

http://docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad.pdf

HTH,
Devender