LV Status available/syncd but PV unavailable

[YA Unix Admin]

--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 2
Open LV 2
Max PV 16
Cur PV 2
Act PV 2
Max PE per PV 8683
VGDA 4
PE Size (Mbytes) 4
Total PE 17362
Alloc PE 2774
Free PE 14588
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg02/lvol1
LV Status available/syncd
LV Size (Mbytes) 4096
Current LE 1024
Allocated PE 1024
Used PV 1

LV Name /dev/vg02/lvol02
LV Status available/syncd
LV Size (Mbytes) 7000
Current LE 1750
Allocated PE 1750
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c1t0d0
PV Status unavailable
Total PE 8681
Free PE 5907

PV Name /dev/dsk/c2t0d0
PV Status available
Total PE 8681
Free PE 8681

fstab says
/dev/vg02/lvol1 /finance/u10 vxfs delaylog,datainlog,largefiles,rw,suid 0 2
/dev/vg02/lvol02 /finance/u09 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2

bdf used to say
/dev/vg02/lvol1 4096000 7112 3838324 0% 5 1022219 0% /finance/u10
/dev/vg02/lvol02 7168000 3591311 3353722 52% 33539 894169 4% /finance/u09

today it gave I/O error for u09
I unmounted both u09 and u10 and now if I try
#mount /finance/u10
mount - read of super-block on /dev/vg02/lvol1 failed: I/O error

fsck -F vxfs /dev/vg02/rlvol02
cannot open device /dev/vg02/rlvol02
file system check failure, aborting ...

ioscan is quite happy
H/W Path Class Description
=========================================================================
root
0 ioa System Bus Adapter (582)
0/0 ba Local PCI Bus Adapter (782)
0/0/0/0 lan HP PCI 10/100Base-TX Core
0/0/1/0 ext_bus SCSI C896 Fast Wide LVD
0/0/1/0.7 target
0/0/1/0.7.0 ctl Initiator
0/0/1/1 ext_bus SCSI C896 Ultra Wide Single-Ended
0/0/1/1.0 target
0/0/1/1.0.0 disk FUJITSU MAJ3364MC
0/0/1/1.2 target
0/0/1/1.2.0 disk SEAGATE ST318404LC
0/0/1/1.7 target
0/0/1/1.7.0 ctl Initiator
0/0/2/0 ext_bus SCSI C875 Ultra Wide Single-Ended
0/0/2/0.0 target
0/0/2/0.0.0 disk SEAGATE ST336704LC
0/0/2/0.2 target
0/0/2/0.2.0 disk SEAGATE ST318404LC
0/0/2/0.7 target
0/0/2/0.7.0 ctl Initiator
0/0/2/1 ext_bus SCSI C875 Ultra Wide Single-Ended
0/0/2/1.2 target
0/0/2/1.2.0 disk HP DVD-ROM 304
0/0/2/1.7 target
0/0/2/1.7.0 ctl Initiator
0/0/4/0 tty PCI Serial (103c1048)
0/0/5/0 tty PCI Serial (103c1048)
0/1 ba Local PCI Bus Adapter (782)

.. can somebody point me in a direction!?!

[Torsten.]

An

"ioscan -fn"

would be more useful.

The available disk /dev/dsk/c2t0d0 is totally unused, so I think all data is on the unavailable disk.

The problem with the status could be caused by missing patches. What is your OS version?

I think you need to check and probably replace the disk and restore the data, because the lvols are not mirrored.

[YA Unix Admin]

++#ioscan -fn
Class I H/W Path Driver S/W State H/W Type Description
======================================================================================
root 0 root CLAIMED BUS_NEXUS
ioa 0 0 sba CLAIMED BUS_NEXUS System Bus Adapter (582)
ba 0 0/0 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
lan 0 0/0/0/0 btlan3 CLAIMED INTERFACE HP PCI 10/100Base-TX Core
/dev/diag/lan0 /dev/ether0
ext_bus 0 0/0/1/0 c720 CLAIMED INTERFACE SCSI C896 Fast Wide LVD
target 0 0/0/1/0.7 tgt CLAIMED DEVICE
ctl 0 0/0/1/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c0t7d0
ext_bus 1 0/0/1/1 c720 CLAIMED INTERFACE SCSI C896 Ultra Wide Single-Ended
target 1 0/0/1/1.0 tgt CLAIMED DEVICE
disk 0 0/0/1/1.0.0 sdisk CLAIMED DEVICE FUJITSU MAJ3364MC
/dev/dsk/c1t0d0 /dev/rdsk/c1t0d0
target 2 0/0/1/1.2 tgt CLAIMED DEVICE
disk 1 0/0/1/1.2.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
target 3 0/0/1/1.7 tgt CLAIMED DEVICE
ctl 1 0/0/1/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c1t7d0
ext_bus 2 0/0/2/0 c720 CLAIMED INTERFACE SCSI C875 Ultra Wide Single-Ended
target 4 0/0/2/0.0 tgt CLAIMED DEVICE
disk 2 0/0/2/0.0.0 sdisk CLAIMED DEVICE SEAGATE ST336704LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
target 5 0/0/2/0.2 tgt CLAIMED DEVICE
disk 3 0/0/2/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0
target 6 0/0/2/0.7 tgt CLAIMED DEVICE
ctl 2 0/0/2/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c2t7d0
ext_bus 3 0/0/2/1 c720 CLAIMED INTERFACE SCSI C875 Ultra Wide Single-Ended
target 7 0/0/2/1.2 tgt CLAIMED DEVICE
disk 4 0/0/2/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 304
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
target 8 0/0/2/1.7 tgt CLAIMED DEVICE
ctl 3 0/0/2/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c3t7d0
tty 0 0/0/4/0 asio0 CLAIMED INTERFACE PCI Serial (103c1048)
/dev/GSPdiag1 /dev/diag/mux0 /dev/mux0 /dev/tty0p0 /dev/tty0p1 /dev/tty0p2
tty 1 0/0/5/0 asio0 CLAIMED INTERFACE PCI Serial (103c1048)
/dev/GSPdiag2 /dev/diag/mux1 /dev/mux1 /dev/tty1p1

[Torsten.]

Do a

dd if=dev/rdsk/c1t0d0 of=/dev/null bs=1024k

and see if this runs without errors.

[YA Unix Admin]

how long is it suppose to take to run that ... my nerves !!!

[YA Unix Admin]

dd if=/dev/rdsk/c1t0d0 of=/dev/null bs=1024k

still just sitting there

started it before 10:47
now 10:56

last time I tried this I freaked out and killed it - took a mighty long time to get the kill signal ..

[Torsten.]

This command will read the whole disk and write the contents to "nowhere".

If there is a read failure you can be sure to have a bad disk.

Be patient, this takes some time (~30min, more or less)

[YA Unix Admin]

huge sigh ... all righty then .. I'll pick my heart up of the floor

[Torsten.]

Anyway, I assume a bad disk because of this:

#mount /finance/u10
mount - read of super-block on /dev/vg02/lvol1 failed: I/O error

You may look for your last backup meanwhile ...

[YA Unix Admin]

I actually already restored to an unused mount point. But I still want to "find" the error. Why does everything look so darn happy but *NOT* be .. :(

[Torsten.]

Please read my first comment again ... and answer the question about your OS.

Consider to attach a "swlist" output.

[YA Unix Admin]

the dd is still running
time now is
Tue Jul 15 13:14:54 EDT 2008
time to kill it I suppose...??.. give up on it

about the patch level ... I inherited these old(ish) servers and we never touch them unless they break. So this server has been running un- harassed for long time .. and I'm sure the patches are out of date .. but the only thing that could have "happened" is the hardware died - no soft changes were made. I expected an obvious disk error /hardware error .. *somewhere* - but no, was not what I got.

[Torsten.]

Based on the ioscan I assume this is an L-class server, probably running an early 11.11 or even 11.00.

Years ago there were problems like your with failed disks and LVM - fixed by several LVM patches.

If your system is not patched, I guess you have a failed disk and LVM is not able to detect this correctly.

You should wait for dd to finish or stop, search the root mailbox and syslog for related errors.

[YA Unix Admin]

I see what you mean about patches ... I found these bits and pieces

from syslog

Jul 15 08:29:08 nickel vmunix: vxfs: mesg 055: vx_metaioerr - /dev/vg02/lvol1 file system meta data write error
Jul 15 08:29:23 nickel vmunix: vxfs: mesg 031: vx_disable - /dev/vg02/lvol1 file system disabled

from dmesg

vxfs: mesg 056: vx_dataioerr - /dev/vg02/lvol02 file system file data read error
vxfs: mesg 055: vx_metaioerr - /dev/vg02/lvol02 file system meta data write error
vxfs: mesg 020: vx_logerr - /dev/vg02/lvol02 file system log error 5
vxfs: mesg 031: vx_disable - /dev/vg02/lvol02 file system disabled
vxfs: mesg 055: vx_metaioerr - /dev/vg02/lvol1 file system meta data write error
vxfs: mesg 031: vx_disable - /dev/vg02/lvol1 file system disabled
~

so the prob reason i did not get an obvious error .. is because we are so far out of patches. I'll buy that ...

This is an old box and I will make management aware of the unpatched situation. I ain't inclined to open that pandora's box at this point ...


In the mean time the dd is still sitting there ... fwiw sar does not show the disk doing anything. Is there anyway to see if it's doing anything ..

(Thanks for all you help btw )

[Torsten. · Accepted solution]

The dd may hang because of the bad disk. Disks can die in several ways - fast or just slow ... but I'm sure this disk is bad and needs to be replaced because of all the I/O errors.

[YA Unix Admin]

Now that found something in the logs ... I'll settle for that as my error.


Thanks for all the fish :)

[YA Unix Admin]

Because of the not current patches .. the os prob can't give pretty errors ... but dmesg does show the disk having problems.