pvmove issue

[P_F]

Let me briefly summarize:

LVM2, Kernel 2.6.9, RH AS 4.5

I noticed a series of bad disks in my LVM VG. Since all the VG was full, I requested new disks. Once presented I was then to perform a pvmove to the unallocated PEs on the new disks.

Stupidly, I allocated the space to the LV. Now I can't pvmove since all the extents are used.

1) I'm using LVM2 and there is no online LV reducing command, so I have to use resize2fs to eventually unallocate the new disk, so I can pvmove the stuff off the old bad disk.

A) is resize2fs safe?

B) unmounting - which is required with resize2fs - will disrupt production and cause delays, I'd rather avoid this option.

2) I guess I could wait til the morning, and request new storage and do it right.


Any options. Am I missing something?

[Ivan Ferreira]

Here you have the procedure to reduce the file system:
http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1233622132384+28353475&threadId=1160556

1) I'm using LVM2 and there is no online LV reducing command

You have to umount the file system to reduce the size


A) is resize2fs safe?

If you don't make any mistakes. Anyway, I will always have a good backup.

B) unmounting - which is required with resize2fs - will disrupt production and cause delays, I'd rather avoid this option.

Then no other option, maybe copy/rsync?

2) I guess I could wait til the morning, and request new storage and do it right.

Good news ;)

[Matti_Kurkela]

Have you already extended the _filesystem_ to use the new disks?

If you've used lvextend but not yet extended the filesystem (using resize2fs/ext2online/whatever), just lvreduce back to the original value *but not any smaller than that*.

Use "tune2fs -l" to verify the current size of the filesystem if you're unsure: block count * block size = filesystem size in bytes. Then see if it matches what lvdisplay says about the LV size.

Remember:
filesystem size > LV size: filesystem is corrupted, some data may be lost
filesystem size < LV size: either a resize operation is half-done, or someone has made a mistake
filesystem size = LV size: OK.

If you've already extended the filesystem, the first priority is to take an extra backup to make sure you don't lose data, whatever happens.

Shrinking the filesystem should be safe when the filesystem is unmounted and error-free. However, shrinking operations are much rarer than extensions, so you should be prepared.

In addition, you're working with known bad disks. That is definitely a risk.

I would take a fresh backup ASAP and "do it right" in the morning.

If the disk fails during the night, I'd try to salvage the very latest data if reasonable, then nuke & recreate the LV and the filesystem using good disks only, then restore. Of course, my situation may be different from yours.

MK

[skt_skt]

you could do FS extend online on RHEL for if ext2online is present. But the shrink ops is not the one we are practically done with.

[P_F]

Thanks for the responses.

I have a follow-up question, more on LVM.

1)
If an LVM disk develops bad sectors. Does LVM identify those sectors and will attempt not to write to bad disk sectors, or even the disk itself?

2)
Suppose I complete on pvmove, and then extend the new data into the existing VG. Suppose I can't vgreduce the old bad disk, to remove from the VG. Could that data migrate back onto the old drive? Or could new data be put on the bad drive?

[P_F]

> Have you already extended the _filesystem_ to use the new disks?

Yes. Fortunately, the SAN guys have presented new disks and I am pvmove'ing the data that is readable from the bad disks to the new disks.

Of course, I won't be able to vgreduce those LUNS. So, I'm wondering what will happen to the partially corrupt luns, will new data somehow be written to them again?

[skt_skt]

if the disk has RD/WR error LINUX has a tendency to put the file system READ only. LVM itself does not fix it. But an fsck can help if the error is correctable. FSCK mean u have to have a backup

[P_F]

> if the disk has RD/WR error LINUX has a tendency to put the file system READ only. LVM itself does not fix it.

The type of error we are seeing on the disk is:

Kernel: SCSI error : <0 0 4 10> return code = 0x20000

Kernel: end request: I/O error dev sdk , sector 432582893

It would seem the Linux' FS table would make those bad sectors off limits ?

[skt_skt]

did u confirm if the reported error is coming from the suspected failing disk?

[P_F]

> did u confirm if the reported error is coming from the suspected failing disk?

I used
# dd if=/dev/sdk of=/dev/null bs=1024k

To look for i/o errors and, yes, I found some. Additionally, I can't run fsck on the disk since I can't umount them now.