file system corrupted and VG crashed with LV during non-root mirrored disk replacement

[Amit_Mohabanshi]

Hi All,

We were performing non-root mirrored disk replacement. As it was a hot-swappable disk configured LVM on it. we performed below step to temporary halt LVM access to disk before replacement. but after replacement disk was not come online.

pvchange -a N /dev/dsk/c4t12d0 //Detach disk from LVM                                                                            

I am suspecting that either we could have to do unmount the file system before performing pvchange or we have replaced another disk which was mirrored with faulty one. Please help me to find out actual reason behind this error message we have get in syslog.

VXFS:vx_disable - /dev/vg_db2/lv_dbf12 file system disabled

What could be the reason behind it??

[Torsten.]

>> we have replaced another disk which was mirrored with faulty one.

This could be a reason. Hard to tell without more details.

[Amit_Mohabanshi]

Hi Torsten,

Please find below information contain df, VG  info. and system logs recorded during investigation. Please let me know if it requires any more details.

------

#df -k /dev/vg_db2/lv_dbf12
df: cannot determine file system statistics for /dev/vg_db2/lv_dbf12

syslog:

Sep 22 08:48:39 slbsprod above message repeats 16 times
Sep 22 08:48:56 slbsprod vmunix: SCSI Ultra320 0/2/1/0 instance 4: IO Type : SCSI IO has timed-out. Target ID: 12, LUN ID: 0. Read Command - CDB: 28 00 00 00 00 00 00 00 01 00
Sep 22 08:49:30 slbsprod vmunix: SCSI Ultra320 0/2/1/0 instance 4: IO Type : SCSI IO has timed-out. Target ID: 12, LUN ID: 0. Read Command - CDB: 28 00 00 00 00 00 00 02 00 00
Sep 22 08:50:12 slbsprod vmunix: msgcnt 2 vxfs: mesg 025: vx_wsuper - /dev/vg_db2/lv_dbf12 file system super-block update failed
Sep 22 08:50:12 slbsprod vmunix: msgcnt 3 vxfs: mesg 031: vx_disable - /dev/vg_db2/lv_dbf12 file system disabled

VG information:

# vgdisplay -v /dev/vg_db2
--- Volume groups ---
VG Name /dev/vg_db2
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 2
Act PV 1
Max PE per PV 17881
VGDA 4
PE Size (Mbytes) 16
Total PE 35758
Alloc PE 35758
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg_db2/lv_dbf12
LV Status available/stale
LV Size (Mbytes) 286064
Current LE 17879
Allocated PE 35758
Used PV 2


--- Physical volumes ---
PV Name /dev/dsk/c4t12d0
PV Status unavailable
Total PE 17879
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t12d0
PV Status unavailable
Total PE 17879
Free PE 0
Autoswitch On

--

Thanks

Amit

[Torsten.]

Both disks are unavailable.

Please post

# ioscan -fn

[Amit_Mohabanshi]

ioscan showing below output after replacement.

slbsprod[/] -->ioscan -fnC disk
Class     I  H/W Path       Driver  S/W State   H/W Type     Description
=========================================================================
disk      0  0/0/3/0.0.0.0  sdisk   CLAIMED     DEVICE       TEAC    DV-28E-N
                           /dev/dsk/c0t0d0   /dev/rdsk/c0t0d0
disk      1  0/1/1/0.1.0    sdisk   CLAIMED     DEVICE       HP 73.4GST373454LC
                           /dev/dsk/c2t1d0   /dev/rdsk/c2t1d0
disk      2  0/1/1/1.0.0    sdisk   CLAIMED     DEVICE       HP 73.4GST373454LC
                           /dev/dsk/c3t0d0   /dev/rdsk/c3t0d0
disk      3  0/2/1/0.8.0    sdisk   CLAIMED     DEVICE       HP 300 GST3300007LC
                           /dev/dsk/c4t8d0   /dev/rdsk/c4t8d0
disk      4  0/2/1/0.10.0   sdisk   CLAIMED     DEVICE       HP 300 GMAW3300NC
                           /dev/dsk/c4t10d0   /dev/rdsk/c4t10d0
disk      5  0/2/1/0.12.0   sdisk   NO_HW       DEVICE       HP 300 GMAW3300NC
                           /dev/dsk/c4t12d0   /dev/rdsk/c4t12d0
disk      6  0/2/1/0.14.0   sdisk   CLAIMED     DEVICE       HP 300 GMAW3300NC
                           /dev/dsk/c4t14d0   /dev/rdsk/c4t14d0
disk      7  0/6/1/0.8.0    sdisk   CLAIMED     DEVICE       HP 300 GST3300007LC
                           /dev/dsk/c6t8d0   /dev/rdsk/c6t8d0
disk      8  0/6/1/0.10.0   sdisk   CLAIMED     DEVICE       HP 300 GST3300007LC
                           /dev/dsk/c6t10d0   /dev/rdsk/c6t10d0
disk      9  0/6/1/0.12.0   sdisk   CLAIMED     DEVICE       HP 300 GST3300007LC
                           /dev/dsk/c6t12d0   /dev/rdsk/c6t12d0
disk     10  0/6/1/0.14.0   sdisk   CLAIMED     DEVICE       HP 300 GST3300007LC
                           /dev/dsk/c6t14d0   /dev/rdsk/c6t14d0

----

Thanks

Amit

[Torsten.]

c4t12d0 is dead. Looks like you have 2 DS21x0 chassis.
Did you pull and re-insert the wrong disk?

Try to activate the VG (vgchange -a y ... in quorum mode).

[Amit_Mohabanshi]

Yeah.. you are right. c4t12d0 is dead and we have 2 DS21x0 chassis.
Actually disk c4t12d0 was mirrored with c6t12d0. but after replacement we found that both PVs were unavailable.

At initial stage when we had found that c4t12d0 is dead then we opened a case with HP. As we are working from remote support and were not able to identify faulty drive located at datacenter so we provided some logs to HP tech. and they help us  to identify faulty drive. As per discussion with HP  we identified that  faulty drive is 3rd one from left to right.  so we went ahead and detached the PV for same with ( pvchange -a N /dev/dsk/c4t12d0 ) to halt LVM access to disk before replacement.  I am not sure if they pulled the same drive in DC as we had told them or another one but  unfortunately VG is corrupted and now  needs to be rebuilt as per our HP Admin.

Still we are in the confusion that either HP was wrong in identification of drive or we have replaced working mirrored c6t12d0 instead of faulty c412d0 which has made VG corrupted. I don't think so vgchange will work on corrupted VG. Please analyze the scenario and let me know what exactly was happened at that time per your guess. All kind of suggestions and guessing will be appreiated. Thanks for replying Torsten.

--

Best Regards

Amit M.

[Amit_Mohabanshi]

is there any one help me on this? i just want someone to help me to analyze the issue and tell me probable cause for the same.

If anyone could make any guessings as per his knowledge it would be so much thankful for me.

---
Thanks
Amit

[Torsten. · Accepted solution]

c412d0 is still dead, so you probably replaced the wrong disk.

Hence the data is gone.

To identify the disk, I often use a "dd" read to the disk next to the dead one (since this is a healthy disk the process will not get hung, so you don't have a problem.

Example:

# dd if=/dev/rdsk/c410d0 of=/dev/null

then the disk left to the bad disk c412d0 will lit.