VXFS , lvol corrupt every couple days

zs_hzh — Sun, 02 Nov 2008 03:28:32 GMT

hi, experts

i'm facing a critical problem , my backup logical volume was corrupted and need fsck every couple days , and that caused database to hang after archives are full ..

my OS 11i, my vxfs Version 4
fileset 999 is deleted files ?

any idea of why it happend and where i should be looking at ?

thanks!
br/ricky

here is log of fsck
root% fsck -y /dev/vgbackup/lvol1
fsck: /etc/default/fs is used for determining the file system type
log replay in progress
file system is not clean, full fsck required
pass0 - checking structural files
pass1 - checking inode sanity and blocks
pass2 - checking directory linkage
pass3 - checking reference counts
pass4 - checking resource maps
fileset 999 au 13 imap incorrect - fix (ynq)y
fileset 999 au 16 imap incorrect - fix (ynq)y
fileset 999 au 33 imap incorrect - fix (ynq)y
fileset 999 au 36 imap incorrect - fix (ynq)y
fileset 999 au 37 imap incorrect - fix (ynq)y
fileset 999 au 39 imap incorrect - fix (ynq)y
fileset 999 au 40 imap incorrect - fix (ynq)y
au 0 emap incorrect - fix? (ynq)y
au 0 summary incorrect - fix? (ynq)y
au 1 emap incorrect - fix? (ynq)y
au 1 summary incorrect - fix? (ynq)y
au 2 summary incorrect - fix? (ynq)y
au 3 summary incorrect - fix? (ynq)y
au 7 summary incorrect - fix? (ynq)y
au 8 summary incorrect - fix? (ynq)y
au 9 emap incorrect - fix? (ynq)y
au 9 summary incorrect - fix? (ynq)y
au 10 summary incorrect - fix? (ynq)y
au 11 summary incorrect - fix? (ynq)y
au 12 summary incorrect - fix? (ynq)y
au 14 emap incorrect - fix? (ynq)y
au 14 summary incorrect - fix? (ynq)y
au 15 summary incorrect - fix? (ynq)y
au 16 summary incorrect - fix? (ynq)y
au 17 summary incorrect - fix? (ynq)y
au 18 summary incorrect - fix? (ynq)y
au 19 summary incorrect - fix? (ynq)y
au 20 summary incorrect - fix? (ynq)y
au 21 summary incorrect - fix? (ynq)y
au 22 summary incorrect - fix? (ynq)y
au 23 summary incorrect - fix? (ynq)y
au 24 emap incorrect - fix? (ynq)y
au 24 summary incorrect - fix? (ynq)y
au 25 emap incorrect - fix? (ynq)y
au 25 summary incorrect - fix? (ynq)y
au 26 emap incorrect - fix? (ynq)y
au 26 summary incorrect - fix? (ynq)y
au 27 emap incorrect - fix? (ynq)y
au 27 summary incorrect - fix? (ynq)y
au 30 summary incorrect - fix? (ynq)y
au 31 summary incorrect - fix? (ynq)y
au 32 emap incorrect - fix? (ynq)y
au 32 summary incorrect - fix? (ynq)y
au 33 summary incorrect - fix? (ynq)y
au 34 summary incorrect - fix? (ynq)y
au 35 summary incorrect - fix? (ynq)y
au 36 summary incorrect - fix? (ynq)y
au 48 summary incorrect - fix? (ynq)y
au 49 summary incorrect - fix? (ynq)y
au 50 summary incorrect - fix? (ynq)y
au 51 summary incorrect - fix? (ynq)y
au 52 summary incorrect - fix? (ynq)y
au 53 summary incorrect - fix? (ynq)y
au 54 emap incorrect - fix? (ynq)y
au 54 summary incorrect - fix? (ynq)y
au 55 emap incorrect - fix? (ynq)y
au 55 summary incorrect - fix? (ynq)y
au 56 emap incorrect - fix? (ynq)y
au 56 summary incorrect - fix? (ynq)y
au 57 emap incorrect - fix? (ynq)y
au 57 summary incorrect - fix? (ynq)y
au 58 summary incorrect - fix? (ynq)y
au 59 summary incorrect - fix? (ynq)y
au 60 emap incorrect - fix? (ynq)y
au 60 summary incorrect - fix? (ynq)y
au 61 summary incorrect - fix? (ynq)y
au 62 summary incorrect - fix? (ynq)y
au 64 emap incorrect - fix? (ynq)y
au 64 summary incorrect - fix? (ynq)y
au 65 emap incorrect - fix? (ynq)y
au 65 summary incorrect - fix? (ynq)y
au 66 emap incorrect - fix? (ynq)y
au 66 summary incorrect - fix? (ynq)y
au 67 emap incorrect - fix? (ynq)y
au 67 summary incorrect - fix? (ynq)y
au 68 emap incorrect - fix? (ynq)y
au 68 summary incorrect - fix? (ynq)y
au 69 emap incorrect - fix? (ynq)y
au 69 summary incorrect - fix? (ynq)y
au 70 summary incorrect - fix? (ynq)y
au 71 summary incorrect - fix? (ynq)y
au 72 summary incorrect - fix? (ynq)y
au 73 summary incorrect - fix? (ynq)y
au 74 summary incorrect - fix? (ynq)y
au 75 emap incorrect - fix? (ynq)y
au 75 summary incorrect - fix? (ynq)y
au 76 summary incorrect - fix? (ynq)y
au 77 summary incorrect - fix? (ynq)y
au 78 emap incorrect - fix? (ynq)y
au 78 summary incorrect - fix? (ynq)y
au 79 summary incorrect - fix? (ynq)y
au 80 summary incorrect - fix? (ynq)y
au 81 summary incorrect - fix? (ynq)y
au 83 emap incorrect - fix? (ynq)y
au 83 summary incorrect - fix? (ynq)y
au 85 summary incorrect - fix? (ynq)y
au 86 summary incorrect - fix? (ynq)y
au 87 summary incorrect - fix? (ynq)y
au 88 summary incorrect - fix? (ynq)y
au 89 summary incorrect - fix? (ynq)y
au 92 summary incorrect - fix? (ynq)y
au 93 summary incorrect - fix? (ynq)y
au 94 summary incorrect - fix? (ynq)y
au 95 summary incorrect - fix? (ynq)y
au 97 emap incorrect - fix? (ynq)y
au 97 summary incorrect - fix? (ynq)y
au 112 emap incorrect - fix? (ynq)y
au 112 summary incorrect - fix? (ynq)y
au 113 summary incorrect - fix? (ynq)y
au 114 emap incorrect - fix? (ynq)y
au 114 summary incorrect - fix? (ynq)y
au 115 emap incorrect - fix? (ynq)y
au 115 summary incorrect - fix? (ynq)y
au 120 summary incorrect - fix? (ynq)y
au 121 emap incorrect - fix? (ynq)y
au 121 summary incorrect - fix? (ynq)y
au 123 summary incorrect - fix? (ynq)y
au 124 emap incorrect - fix? (ynq)y
au 124 summary incorrect - fix? (ynq)y
au 125 emap incorrect - fix? (ynq)y
au 125 summary incorrect - fix? (ynq)y
au 126 emap incorrect - fix? (ynq)y
au 126 summary incorrect - fix? (ynq)y
au 127 emap incorrect - fix? (ynq)y
au 127 summary incorrect - fix? (ynq)y
au 128 summary incorrect - fix? (ynq)y
au 130 summary incorrect - fix? (ynq)y
au 131 emap incorrect - fix? (ynq)y
au 131 summary incorrect - fix? (ynq)y
au 132 emap incorrect - fix? (ynq)y
au 132 summary incorrect - fix? (ynq)y
au 133 emap incorrect - fix? (ynq)y
au 133 summary incorrect - fix? (ynq)y
au 134 emap incorrect - fix? (ynq)y
au 134 summary incorrect - fix? (ynq)y
au 136 emap incorrect - fix? (ynq)y
au 136 summary incorrect - fix? (ynq)y
au 137 summary incorrect - fix? (ynq)y
au 138 emap incorrect - fix? (ynq)y
au 138 summary incorrect - fix? (ynq)y
au 139 summary incorrect - fix? (ynq)y
au 140 emap incorrect - fix? (ynq)y
au 140 summary incorrect - fix? (ynq)y
au 141 emap incorrect - fix? (ynq)y
au 141 summary incorrect - fix? (ynq)y
au 142 emap incorrect - fix? (ynq)y
au 142 summary incorrect - fix? (ynq)y
au 146 summary incorrect - fix? (ynq)y
au 147 summary incorrect - fix? (ynq)y
au 149 emap incorrect - fix? (ynq)y
au 149 summary incorrect - fix? (ynq)y
au 152 summary incorrect - fix? (ynq)y
au 153 emap incorrect - fix? (ynq)y
au 153 summary incorrect - fix? (ynq)y
au 154 emap incorrect - fix? (ynq)y
au 154 summary incorrect - fix? (ynq)y
au 159 emap incorrect - fix? (ynq)y
au 159 summary incorrect - fix? (ynq)y
au 161 emap incorrect - fix? (ynq)y
au 161 summary incorrect - fix? (ynq)y
au 171 emap incorrect - fix? (ynq)y
au 171 summary incorrect - fix? (ynq)y
au 173 emap incorrect - fix? (ynq)y
au 173 summary incorrect - fix? (ynq)y
au 178 emap incorrect - fix? (ynq)y
au 178 summary incorrect - fix? (ynq)y
au 180 summary incorrect - fix? (ynq)y
au 185 emap incorrect - fix? (ynq)y
au 185 summary incorrect - fix? (ynq)y
au 188 summary incorrect - fix? (ynq)y
au 191 emap incorrect - fix? (ynq)y
au 191 summary incorrect - fix? (ynq)y
au 193 emap incorrect - fix? (ynq)y
au 193 summary incorrect - fix? (ynq)y
au 195 emap incorrect - fix? (ynq)y
au 195 summary incorrect - fix? (ynq)y
au 196 emap incorrect - fix? (ynq)y
au 196 summary incorrect - fix? (ynq)y
au 197 summary incorrect - fix? (ynq)y
au 198 summary incorrect - fix? (ynq)y
au 200 summary incorrect - fix? (ynq)y
au 202 emap incorrect - fix? (ynq)y
au 202 summary incorrect - fix? (ynq)y
au 203 emap incorrect - fix? (ynq)y
au 203 summary incorrect - fix? (ynq)y
au 208 summary incorrect - fix? (ynq)y
au 210 summary incorrect - fix? (ynq)y
au 211 summary incorrect - fix? (ynq)y
au 212 summary incorrect - fix? (ynq)y
au 215 summary incorrect - fix? (ynq)y
au 216 summary incorrect - fix? (ynq)y
au 219 summary incorrect - fix? (ynq)y
au 220 summary incorrect - fix? (ynq)y
au 221 summary incorrect - fix? (ynq)y
au 222 emap incorrect - fix? (ynq)y
au 222 summary incorrect - fix? (ynq)y
au 225 emap incorrect - fix? (ynq)y
au 225 summary incorrect - fix? (ynq)y
au 226 summary incorrect - fix? (ynq)y
au 229 summary incorrect - fix? (ynq)y
au 231 summary incorrect - fix? (ynq)y
au 232 summary incorrect - fix? (ynq)y
au 236 summary incorrect - fix? (ynq)y
au 237 summary incorrect - fix? (ynq)y
au 238 summary incorrect - fix? (ynq)y
au 241 emap incorrect - fix? (ynq)y
au 241 summary incorrect - fix? (ynq)y
au 243 emap incorrect - fix? (ynq)y
au 243 summary incorrect - fix? (ynq)y
au 245 summary incorrect - fix? (ynq)y
au 246 emap incorrect - fix? (ynq)y
au 246 summary incorrect - fix? (ynq)y
au 247 emap incorrect - fix? (ynq)y
au 247 summary incorrect - fix? (ynq)y
au 248 emap incorrect - fix? (ynq)y
au 248 summary incorrect - fix? (ynq)y
au 249 emap incorrect - fix? (ynq)y
au 249 summary incorrect - fix? (ynq)y
au 253 emap incorrect - fix? (ynq)y
au 253 summary incorrect - fix? (ynq)y
au 255 summary incorrect - fix? (ynq)y
au 257 summary incorrect - fix? (ynq)y
au 260 summary incorrect - fix? (ynq)y
au 264 emap incorrect - fix? (ynq)y
au 264 summary incorrect - fix? (ynq)y
au 265 summary incorrect - fix? (ynq)y
au 267 summary incorrect - fix? (ynq)y
au 274 emap incorrect - fix? (ynq)y
au 274 summary incorrect - fix? (ynq)y
au 279 summary incorrect - fix? (ynq)y
au 298 emap incorrect - fix? (ynq)y
au 298 summary incorrect - fix? (ynq)y
au 310 summary incorrect - fix? (ynq)y
au 322 summary incorrect - fix? (ynq)y
au 327 emap incorrect - fix? (ynq)y
au 327 summary incorrect - fix? (ynq)y
au 329 emap incorrect - fix? (ynq)y
au 329 summary incorrect - fix? (ynq)y
au 343 emap incorrect - fix? (ynq)y
au 343 summary incorrect - fix? (ynq)y
au 346 emap incorrect - fix? (ynq)y
au 346 summary incorrect - fix? (ynq)y
au 353 emap incorrect - fix? (ynq)y
au 353 summary incorrect - fix? (ynq)y
au 354 summary incorrect - fix? (ynq)y
au 356 summary incorrect - fix? (ynq)y
au 361 emap incorrect - fix? (ynq)y
au 361 summary incorrect - fix? (ynq)y
au 400 emap incorrect - fix? (ynq)y
au 400 summary incorrect - fix? (ynq)y
au 484 emap incorrect - fix? (ynq)y
au 484 summary incorrect - fix? (ynq)y
au 498 emap incorrect - fix? (ynq)y
au 498 summary incorrect - fix? (ynq)y
au 512 summary incorrect - fix? (ynq)y
au 513 summary incorrect - fix? (ynq)y
au 514 summary incorrect - fix? (ynq)y
au 515 summary incorrect - fix? (ynq)y
au 516 summary incorrect - fix? (ynq)y
au 517 summary incorrect - fix? (ynq)y
au 523 summary incorrect - fix? (ynq)y
au 524 summary incorrect - fix? (ynq)y
au 530 emap incorrect - fix? (ynq)y
au 530 summary incorrect - fix? (ynq)y
au 534 summary incorrect - fix? (ynq)y
au 545 summary incorrect - fix? (ynq)y
au 547 summary incorrect - fix? (ynq)y
au 549 summary incorrect - fix? (ynq)y
au 580 summary incorrect - fix? (ynq)y
au 581 emap incorrect - fix? (ynq)y
au 581 summary incorrect - fix? (ynq)y
au 584 emap incorrect - fix? (ynq)y
au 584 summary incorrect - fix? (ynq)y
au 585 emap incorrect - fix? (ynq)y
au 585 summary incorrect - fix? (ynq)y
au 586 emap incorrect - fix? (ynq)y
au 586 summary incorrect - fix? (ynq)y
au 587 summary incorrect - fix? (ynq)y
au 590 emap incorrect - fix? (ynq)y
au 590 summary incorrect - fix? (ynq)y
au 591 summary incorrect - fix? (ynq)y
au 593 emap incorrect - fix? (ynq)y
au 593 summary incorrect - fix? (ynq)y
au 594 emap incorrect - fix? (ynq)y
au 594 summary incorrect - fix? (ynq)y
au 595 summary incorrect - fix? (ynq)y
au 596 summary incorrect - fix? (ynq)y
au 597 summary incorrect - fix? (ynq)y
au 608 emap incorrect - fix? (ynq)y
au 608 summary incorrect - fix? (ynq)y
au 609 emap incorrect - fix? (ynq)y
au 609 summary incorrect - fix? (ynq)y
au 611 summary incorrect - fix? (ynq)y
au 612 emap incorrect - fix? (ynq)y
au 612 summary incorrect - fix? (ynq)y
au 616 emap incorrect - fix? (ynq)y
au 616 summary incorrect - fix? (ynq)y
au 642 summary incorrect - fix? (ynq)y
au 644 summary incorrect - fix? (ynq)y
au 645 summary incorrect - fix? (ynq)y
au 646 summary incorrect - fix? (ynq)y
au 653 summary incorrect - fix? (ynq)y
au 654 summary incorrect - fix? (ynq)y
au 655 summary incorrect - fix? (ynq)y
au 691 summary incorrect - fix? (ynq)y
au 809 emap incorrect - fix? (ynq)y
au 809 summary incorrect - fix? (ynq)y
au 1199 emap incorrect - fix? (ynq)y
au 1199 summary incorrect - fix? (ynq)y
au 1206 emap incorrect - fix? (ynq)y
au 1206 summary incorrect - fix? (ynq)y
au 1247 summary incorrect - fix? (ynq)y
au 1286 summary incorrect - fix? (ynq)y
au 1306 emap incorrect - fix? (ynq)y
au 1306 summary incorrect - fix? (ynq)y
au 1318 emap incorrect - fix? (ynq)y
au 1318 summary incorrect - fix? (ynq)y
au 1328 emap incorrect - fix? (ynq)y
au 1328 summary incorrect - fix? (ynq)y
au 1335 summary incorrect - fix? (ynq)y
au 1341 summary incorrect - fix? (ynq)y
au 1342 emap incorrect - fix? (ynq)y
au 1342 summary incorrect - fix? (ynq)y
au 1343 emap incorrect - fix? (ynq)y
au 1343 summary incorrect - fix? (ynq)y
au 1344 emap incorrect - fix? (ynq)y
au 1344 summary incorrect - fix? (ynq)y
au 1353 emap incorrect - fix? (ynq)y
au 1353 summary incorrect - fix? (ynq)y
au 1357 emap incorrect - fix? (ynq)y
au 1357 summary incorrect - fix? (ynq)y
au 1384 emap incorrect - fix? (ynq)y
au 1384 summary incorrect - fix? (ynq)y
au 1394 summary incorrect - fix? (ynq)y
au 1397 emap incorrect - fix? (ynq)y
au 1397 summary incorrect - fix? (ynq)y
au 1398 summary incorrect - fix? (ynq)y
au 1399 summary incorrect - fix? (ynq)y
au 1401 summary incorrect - fix? (ynq)y
au 1563 summary incorrect - fix? (ynq)y
au 1876 emap incorrect - fix? (ynq)y
au 1876 summary incorrect - fix? (ynq)y
au 1882 emap incorrect - fix? (ynq)y
au 1882 summary incorrect - fix? (ynq)y
au 1975 emap incorrect - fix? (ynq)y
au 1975 summary incorrect - fix? (ynq)y
au 2331 emap incorrect - fix? (ynq)y
au 2331 summary incorrect - fix? (ynq)y
fileset 999 iau 13 summary incorrect - fix? (ynq)y
fileset 999 iau 16 summary incorrect - fix? (ynq)y
fileset 999 iau 33 summary incorrect - fix? (ynq)y
fileset 999 iau 36 summary incorrect - fix? (ynq)y
fileset 999 iau 37 summary incorrect - fix? (ynq)y
fileset 999 iau 39 summary incorrect - fix? (ynq)y
fileset 999 iau 40 summary incorrect - fix? (ynq)y
fileset 999 iau 41 summary incorrect - fix? (ynq)y
free block count incorrect 1182339877 expected 108109359 fix? (ynq)y
free extent vector incorrect fix? (ynq)y
OK to clear log? (ynq)y
set state to CLEAN? (ynq)y

Re: VXFS , lvol corrupt every couple days

Dennis Handly — Sun, 02 Nov 2008 05:44:55 GMT

>my OS 11i

There are 5 versions of 11i, which?

Re: VXFS , lvol corrupt every couple days

zs_hzh — Sun, 02 Nov 2008 09:08:08 GMT

actually , it's 11.11

Server model : RP4440

4 CPU/16G Memory

with Oracle 9.2.0.8 running on top

the corrupted volume is for backing up of DB and file system

Re: VXFS , lvol corrupt every couple days

Victor Fridyev — Sun, 02 Nov 2008 16:12:46 GMT

Hi

Try to check the partition by
dd if=/dev/vgbackup/rlvol1 of=/dev/null bs=32768k

I'm afraid, you'll get I/O errors, so the problem is in the HW

Re: VXFS , lvol corrupt every couple days

zs_hzh — Mon, 03 Nov 2008 01:13:24 GMT

hi , vitor

do i have to check the disk by dd before i execute fsck or after ?

for HW problem ,which part ? Server or storage system ? which is a HP EVA4K

Re: VXFS , lvol corrupt every couple days

zs_hzh — Mon, 03 Nov 2008 01:17:25 GMT

at least now ,it's working

(101) root% dd if=/dev/vgbackup/rlvol1 of=/dev/null bs=32768k
12+0 records in
12+0 records out

Re: VXFS , lvol corrupt every couple days

chinadns — Mon, 03 Nov 2008 03:15:26 GMT

if hw is ok,and only this one filesystem bad.
i think you should backup the data ,recreate the filesystem and restore the data.

Re: VXFS , lvol corrupt every couple days

Bill Hassell — Mon, 03 Nov 2008 04:32:00 GMT

The dd command can be run at anytime. It does not know anything about filesystems -- it just reads the disk tracks. If it finishes with the words: I/O error, then the disk is defective. If you do not have disk mirroring, you'll have to restore your data from backup tapes after replacing the disk.

If you do not see any dd errors, then some script or process is likely destroying the directory.

Re: VXFS , lvol corrupt every couple days

zs_hzh — Mon, 03 Nov 2008 08:44:12 GMT

the vgbackup is for backing up of rootvg and DB and other VG ,

it has a script that do this and delete old backups ,

i have recreated the entire VG before , but it's easily reoccured

destroied the directory ? you mean lost+found ?

fileset 999 is for files that are deleted ?

can you tell me more of how directory destroying affect the system ,

thanks

topic Re: VXFS , lvol corrupt every couple days in Operating System - HP-UX

VXFS , lvol corrupt every couple days

Re: VXFS , lvol corrupt every couple days

Re: VXFS , lvol corrupt every couple days

Re: VXFS , lvol corrupt every couple days

Re: VXFS , lvol corrupt every couple days

Re: VXFS , lvol corrupt every couple days

Re: VXFS , lvol corrupt every couple days

Re: VXFS , lvol corrupt every couple days

Re: VXFS , lvol corrupt every couple days