Procedure for vg02 mirrored disk replacement. !!?

[rveri]

Hi All ,

Can any one give me best procedure for doing this task.

Details:
----------

1. I have a L2000-44 server hpux 11.00 .

# strings /etc/lvmtab

# strings /etc/lvmtab
/dev/vg00
/dev/dsk/c1t2d0
/dev/dsk/c2t2d0
/dev/vg01
/dev/dsk/c6t8d0
/dev/dsk/c6t9d0
/dev/dsk/c6t10d0
/dev/dsk/c6t11d0
/dev/dsk/c6t12d0
/dev/dsk/c7t8d0
/dev/dsk/c7t9d0
/dev/dsk/c7t10d0
/dev/dsk/c7t11d0
/dev/dsk/c7t12d0
/dev/vg02
/dev/dsk/c8t0d0
/dev/dsk/c8t1d0
/dev/dsk/c8t2d0
/dev/dsk/c8t3d0 ** (Bad disk, need to replace)
/dev/dsk/c8t4d0 ** (Bad disk, need to replace)
/dev/dsk/c8t5d0
-------------------------------

# vgdisplay -v /dev/vg02
--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 3
Open LV 3
Max PV 16
Cur PV 6
Act PV 6
Max PE per PV 8683
VGDA 12
PE Size (Mbytes) 4
Total PE 52086
Alloc PE 52086
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg02/lvol06
LV Status available/syncd
LV Size (Mbytes) 34724
Current LE 8681
Allocated PE 17362
Used PV 2

LV Name /dev/vg02/lvol07
LV Status available/syncd
LV Size (Mbytes) 34724
Current LE 8681
Allocated PE 17362
Used PV 2

LV Name /dev/vg02/lvol08
LV Status available/syncd
LV Size (Mbytes) 34724
Current LE 8681
Allocated PE 17362
Used PV 2


--- Physical volumes ---
PV Name /dev/dsk/c8t0d0
PV Status available
Total PE 8681
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t1d0
PV Status available
Total PE 8681
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t2d0
PV Status available
Total PE 8681
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t3d0
PV Status available
Total PE 8681
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t4d0
PV Status available
Total PE 8681
Free PE 0
Autoswitch On

PV Name /dev/dsk/c8t5d0
PV Status available
Total PE 8681
Free PE 0
Autoswitch On


#
----------------------------------------
# lvdisplay -v /dev/vg02/lvol06 | more
--- Logical volumes ---
LV Name /dev/vg02/lvol06
VG Name /dev/vg02
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 34724
Current LE 8681
Allocated PE 17362
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c8t0d0 8681 8681
/dev/dsk/c8t3d0 8681 8681

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c8t0d0 00000 current /dev/dsk/c8t3d0 00000 current
00001 /dev/dsk/c8t0d0 00001 current /dev/dsk/c8t3d0 00001 current
00002 /dev/dsk/c8t0d0 00002 current /dev/dsk/c8t3d0 00002 current
00003 /dev/dsk/c8t0d0 00003 current /dev/dsk/c8t3d0 00003 current
00004 /dev/dsk/c8t0d0 00004 current /dev/dsk/c8t3d0 00004 current
00005 /dev/dsk/c8t0d0 00005 current /dev/dsk/c8t3d0 00005 current
00006 /dev/dsk/c8t0d0 00006 current /dev/dsk/c8t3d0 00006 current
00007 /dev/dsk/c8t0d0 00007 current /dev/dsk/c8t3d0 00007 current
============================================

# lvdisplay -v /dev/vg02/lvol07 | more
--- Logical volumes ---
LV Name /dev/vg02/lvol07
VG Name /dev/vg02
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 34724
Current LE 8681
Allocated PE 17362
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c8t1d0 8681 8681
/dev/dsk/c8t4d0 8681 8681

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c8t1d0 00000 current /dev/dsk/c8t4d0 00000 current
00001 /dev/dsk/c8t1d0 00001 current /dev/dsk/c8t4d0 00001 current
00002 /dev/dsk/c8t1d0 00002 current /dev/dsk/c8t4d0 00002 current
00003 /dev/dsk/c8t1d0 00003 current /dev/dsk/c8t4d0 00003 current
00004 /dev/dsk/c8t1d0 00004 current /dev/dsk/c8t4d0 00004 current

=========================================



Both the disks are having one filesystem , and mirrored to other disk.

I want to replace the disk marked ** , ie c8t3d0 and c3t4d0,

What would be the best procedure and steps to replace the disk, and bring back the system as it is. Its disk of DS2300.

Question:
1.Can I shutdown the system, and change disk and boot the system back, in single user mode and do the activity.
2. whats the role of vgcfgbackup.
3. Do I need to de-activate the vg02 , before shutdown and replace disk.
4. what are the steps after h/w replacement.


Please suggest,

Thanks ,
R.Veri.
--------

[Siddhesh]

Hi,

How did you conclude that these PVs are gone bad? I don't see any problem with vgdisplay and pvdisplay output for these PVs.

Regards,
Siddhesh.

[rveri]

Well Good Question:

here is the message , frequently comes in the syslog.log and making the system hang: and need to reset thru GSP.

----------------------------
Jul 20 04:25:18 gudvcdw1 vmunix: SCSI: Unexpected Disconnect -- lbolt: 5324208, dev: 1f083000, io_id: 80a9176
Jul 20 04:25:18 gudvcdw1 vmunix: SCSI Gross Error on 0/2/0/0:
Jul 20 04:25:18 gudvcdw1 vmunix: shadowed SIST0 10 shadowed SIST1 04.
Jul 20 04:25:18 gudvcdw1 vmunix: SCSI: isrEscape Controller at 0/2/0/0.
Jul 20 04:25:18 gudvcdw1 vmunix:
Jul 20 04:25:18 gudvcdw1 vmunix: SCSI: -- lbolt: 5324240, bus: 8
Jul 20 04:25:18 gudvcdw1 vmunix: lbp->state: 30008
Jul 20 04:25:18 gudvcdw1 vmunix: lbp->offset: ffffffff
Jul 20 04:25:18 gudvcdw1 vmunix: lbp->nominalOffset: 360
Jul 20 04:25:18 gudvcdw1 vmunix: lbp->Cmdindex: a
Jul 20 04:25:18 gudvcdw1 vmunix: lbp->last_nexus_index: 70
Jul 20 04:25:18 gudvcdw1 vmunix: lbp->nexus_index: 71
Jul 20 04:25:18 gudvcdw1 vmunix: uCmdSent: c00dac0 uNexus_offset: 3d7c4
Jul 20 04:25:18 gudvcdw1 vmunix: last lbp->puStatus [00000000401757a4]:
--------------
Jul 20 04:25:25 gudvcdw1 EMS [9267]: ------ EMS Event Notification ------ Value: "SERIOUS (4)" for Resource: "/storage
/events/disks/default/0_2_0_0.3.0" (Threshold: >= " 3") Execute the following command to obtain event details:
/opt/resmon/bin/resdata -R 607322138 -r /storage/events/disks/default/0_2_0_0.3.0 -n 607322119 -a
Jul 20 04:25:46 gudvcdw1 vmunix:
Jul 20 04:25:46 gudvcdw1 vmunix: SCSI: Read error -- dev: b 31 0x083000, errno: 126, resid: 1024,
Jul 20 04:25:46 gudvcdw1 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
Jul 20 04:25:46 gudvcdw1 vmunix: LVM: VG 64 0x020000: PVLink 31 0x083000 Failed! The PV is not accessible.
Jul 20 04:25:51 gudvcdw1 vmunix: LVM: VG 64 0x020000: PVLink 31 0x083000 Recovered.

--------------------
We have replaced one BCC card on DS2300, already.


Thanks ,
R.Veri.
----------


------------------

[Rajeev Shukla]

One method i can suggest is if the disks are hot-swap then replace the disk and do a vgcfgrestore on the replaced disks.
Or shutdown the system and then replace the disks and bring the system in single user mode and do a vgcfgrestore on the replaced disks.
Other option i see is, since the disks still show the LV attached to them you could probably remove the mirroring by doing
pvdisplay -v /dev/dsk/c8t4d0 |grep ^" \/dev\/"|awk '{print $1}' > /tmp/list1

pvdisplay -v /dev/dsk/c8t3d0 |grep ^" \/dev\/"|awk '{print $1}' > /tmp/list2

while read i
do
lvreduce -m 0 $i /dev/dsk/c8t4d0
done
do the same for /tmp/list2

then remove both the disks from vg02
vgreduce /dev/vg02 /dev/dsk/c8t4d0 /dev/dsk/c8t3d0

Then replace the disks and then add the new replaced disks to vg02 and establish the mirrors as per /tmp/list1 and /tmp/list2

[morganelan]

Hi,
If you mirror a logical volume on a separate disk, the mirror copy will be online and available while recovering from a disk failure. With hot-swappable disks, your users will have no indication that a disk was lost.

To remove the mirror copy from a specific disk, use lvreduce, and specify the disk from which to remove the mirror copy:
# lvreduce -m 0 -A n /dev/vgname/lvname pvname (if you have a single mirror copy) or
# lvreduce -m 1 -A n /dev/vgname/lvname pvname (if you have two mirror copies)

The â A n option is used to prevent the lvreduce command from performing an automatic vgcfgbackup operation, which could hang while accessing a defective disk.

your disks are hot-swappable, all the logical volumes are mirrored, and the LVM OLR patches are installed. In this case, you can detach the disk with pvchange, replace it, reattach it, and let LVMâ s mirroring synchronize the logical volumesâ all while the system remains booted.And then

# pvchange -a N /dev/dsk/c8t3d0
#
# vgcfgrestore â n vg02 /dev/rdsk/c8t3d0
# vgchange â a y vg02

[Rajeev Shukla]

Yes sometimes the disks dont die permanently, they fail intermetantly and kernel reports them being failed and recovered. It is best practice to replace them coz they are going to die soon

[D Block 2]

question: have you run the suggested command for EMS (from your syslog):
opt/resmon/bin/resdata -R 607322138 -r /storage/events/disks/default/0_2_0_0.3.0 -n 607322119 -a

[melvyn burnard]

You may wish to take a read of this document:
http://docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad.pdf

[rveri]

Hi all ,
Thanks for the reply.

I have replaced the disk and all went well,

1. Shutdown the system.
2. The exact disk replaced.[ c8t3d0 ] Hp Engg found problem with this disk. As it was not reading , in # dd if = /dev/dsk/c8t3d0 of=/dev/null bs=1024 command.
3. Restarted the server in normal mode.
4. # vgcfgrestore -n /dev/vg02 /dev/rdsk/c8t3d0
5. # vgchange -a y /dev/vg02
6. # vgsync vg02.

Resynchronized logical volume "/dev/vg02/lvol06".
Resynchronized logical volume "/dev/vg02/lvol07".
Resynchronized logical volume "/dev/vg02/lvol08".
Resynchronized volume group "vg02".
----------

Everything is looks fine now.

Many thanks for all the helps and response.