Issues with lvol's not able to mount or run fsck

[Venkat_11]

Hi,

We are having issue with some logical volumes under one of the volume group on server which is running HPUX 11i version 1. We unmounted it and we started recieving errors when we tried to run fsck against the lvol's in that volume group. Please see the below error messages and advice accordingly.

Thanks and Regards
Venkat

Feb 24 16:33:10 usxsl003 vmunix: msgcnt 1 vxfs: mesg 037: vx_metaioerr - /dev/vg08a/lvol3 file system meta data write error
Feb 24 16:33:10 usxsl003 vmunix: msgcnt 2 vxfs: mesg 038: vx_dataioerr - /dev/vg08a/lvol3 file system file data write error
Feb 24 16:33:10 usxsl003 vmunix: msgcnt 3 vxfs: mesg 020: vx_logerr - /dev/vg08a/lvol3 file system log error 5
Feb 24 16:33:10 usxsl003 vmunix: msgcnt 4 vxfs: mesg 031: vx_disable - /dev/vg08a/lvol3 file system disabled
Feb 24 16:33:14 usxsl003 vmunix: msgcnt 5 vxfs: mesg 057: vx_esum_bad - /dev/vg08a/lvol3 file system extent allocation unit summary number 19 marked bad
Feb 24 16:34:15 usxsl003 vmunix: msgcnt 6 vxfs: mesg 057: vx_esum_bad - /dev/vg08a/lvol3 file system extent allocation unit summary number 7 marked bad
Feb 24 16:34:49 usxsl003 vmunix: msgcnt 7 vxfs: mesg 037: vx_metaioerr - /dev/vg08a/lvol1 file system meta data read error
Feb 24 16:35:15 usxsl003 vmunix: msgcnt 9 vxfs: mesg 057: vx_esum_bad - /dev/vg08a/lvol3 file system extent allocation unit summary number 20 marked bad
Feb 24 16:35:57 usxsl003 vmunix: msgcnt 10 vxfs: mesg 037: vx_metaioerr - /dev/vg08a/lvol2 file system meta data read error
Feb 24 16:35:57 usxsl003 vmunix: msgcnt 12 vxfs: mesg 038: vx_dataioerr - /dev/vg08a/lvol2 file system file data read error
Feb 24 16:36:05 usxsl003 vmunix: msgcnt 13 vxfs: mesg 020: vx_logerr - /dev/vg08a/lvol2 file system log error 5
Feb 24 16:36:06 usxsl003 vmunix: msgcnt 14 vxfs: mesg 031: vx_disable - /dev/vg08a/lvol2 file system disabled


usxsl003:root# fsck -F vxfs -o full /dev/vg08a/lvol2
vxfs fsck: file system had I/O error(s) on meta-data.
vxfs fsck: file system had I/O error(s) on user data.
vxfs fsck: cannot read OLT extent 0
read of primary OLT failed
vxfs fsck: cannot read OLT extent 1
read of OLT copy failed
vxfs fsck: no valid OLT, cannot continue
file system check failure, aborting ...

[James R. Ferguson]

Hi:

I hope you have a good backup. I would 'newfs' the filesystem and reload it.

What symptoms or what were you doing that caused you to unmount it an 'fsck' it?

Were you seeing underlying physical disk errors? If so, you should check the physical disk for any errors:

# dd if=/dev/rdsk/cXtYdZ of=/dev/null bs=1024k

Regards!

...JRF...

[T G Manikandan]

mostly its a bad disk, use
cstm/mstm/xstm to find and replace it.

[Venkat_11]

I noticed that one of the disk on vg08a is unavailable and ran vgcfgrestore -n command to restore data on physical volume and after disk showing available but we are seeing the following errors when we tried to mount and fsck the lv's.

Please advice.

Thanks and Regards
Venkat

usxsl003:root# mount /home
vxfs mount: /dev/vg08a/lvol3 is corrupted. needs checking
usxsl003:root# mount /data04/archive
vxfs mount: /dev/vg08a/lvol1 is corrupted. needs checking
usxsl003:root# fsck -F vxfs -o full /dev/vg08a/lvol3
vxfs fsck: file system had I/O error(s) on meta-data.
vxfs fsck: file system had I/O error(s) on user data.
vxfs fsck: OLT extent 0 has bad magic
read of primary OLT failed
vxfs fsck: OLT extent 1 has bad magic
read of OLT copy failed
vxfs fsck: no valid OLT, cannot continue
file system check failure, aborting ...
usxsl003:root# fsck -F vxfs -o full /dev/vg08a/lvol1
vxfs fsck: file system had I/O error(s) on meta-data.
vxfs fsck: OLT extent 0 has bad magic
read of primary OLT failed
vxfs fsck: OLT extent 1 has bad magic
read of OLT copy failed
vxfs fsck: no valid OLT, cannot continue
file system check failure, aborting ...

[sujit kumar singh]

hi


the metadata area read and wrire errors as well as the errors pertaining to data area read/write that is a serious error; might be the FS is entireley bad. Can be due to a bad disk. Perhaps you do not have an LV mirror of those LVs.
As JRF said you might need to do an new creation of the LV, newfs and resyore data to that LV using the Backups.

Rule out the disk failure / unavailability that you have found by replacing the disk and do newfs and restore the data as you do not have the LV Mirroring from the LVM that seems.you can further do some more tests on that disk like dd or diskinfo to confirm that as bad.

#dd if=/dev/dsk/cXtYdZ of=/dev/null count=100
#diskinfo /dev/rdsk/cXtYdZ


To replace the disk, correctly, you need to put a new disk and do a vgcfgrestore on that disk. And keeping in fact that you do not have the LV as mirrored, you shall need doing a newfs on that LV(s) and restore from the backups.
Replacing a disk you can have a look as belows:
When_Good_Disks_Go_Bad

http://docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad_WP.pdf
regards
sujit