cancel
Showing results for 
Search instead for 
Did you mean: 

pvmove issue

 
Highlighted
Regular Advisor

pvmove issue

Let me briefly summarize:

LVM2, Kernel 2.6.9, RH AS 4.5

I noticed a series of bad disks in my LVM VG. Since all the VG was full, I requested new disks. Once presented I was then to perform a pvmove to the unallocated PEs on the new disks.

Stupidly, I allocated the space to the LV. Now I can't pvmove since all the extents are used.

1) I'm using LVM2 and there is no online LV reducing command, so I have to use resize2fs to eventually unallocate the new disk, so I can pvmove the stuff off the old bad disk.

A) is resize2fs safe?

B) unmounting - which is required with resize2fs - will disrupt production and cause delays, I'd rather avoid this option.

2) I guess I could wait til the morning, and request new storage and do it right.


Any options. Am I missing something?

13 REPLIES 13
Highlighted
Honored Contributor

Re: pvmove issue

Here you have the procedure to reduce the file system:
http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1233622132384+28353475&threadId=1160556

1) I'm using LVM2 and there is no online LV reducing command

You have to umount the file system to reduce the size


A) is resize2fs safe?

If you don't make any mistakes. Anyway, I will always have a good backup.

B) unmounting - which is required with resize2fs - will disrupt production and cause delays, I'd rather avoid this option.

Then no other option, maybe copy/rsync?

2) I guess I could wait til the morning, and request new storage and do it right.

Good news ;)
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
Highlighted
Honored Contributor

Re: pvmove issue

Have you already extended the _filesystem_ to use the new disks?

If you've used lvextend but not yet extended the filesystem (using resize2fs/ext2online/whatever), just lvreduce back to the original value *but not any smaller than that*.

Use "tune2fs -l" to verify the current size of the filesystem if you're unsure: block count * block size = filesystem size in bytes. Then see if it matches what lvdisplay says about the LV size.

Remember:
filesystem size > LV size: filesystem is corrupted, some data may be lost
filesystem size < LV size: either a resize operation is half-done, or someone has made a mistake
filesystem size = LV size: OK.

If you've already extended the filesystem, the first priority is to take an extra backup to make sure you don't lose data, whatever happens.

Shrinking the filesystem should be safe when the filesystem is unmounted and error-free. However, shrinking operations are much rarer than extensions, so you should be prepared.

In addition, you're working with known bad disks. That is definitely a risk.

I would take a fresh backup ASAP and "do it right" in the morning.

If the disk fails during the night, I'd try to salvage the very latest data if reasonable, then nuke & recreate the LV and the filesystem using good disks only, then restore. Of course, my situation may be different from yours.

MK
MK
Highlighted
Honored Contributor

Re: pvmove issue

you could do FS extend online on RHEL for if ext2online is present. But the shrink ops is not the one we are practically done with.
Highlighted
Regular Advisor

Re: pvmove issue

Thanks for the responses.

I have a follow-up question, more on LVM.

1)
If an LVM disk develops bad sectors. Does LVM identify those sectors and will attempt not to write to bad disk sectors, or even the disk itself?

2)
Suppose I complete on pvmove, and then extend the new data into the existing VG. Suppose I can't vgreduce the old bad disk, to remove from the VG. Could that data migrate back onto the old drive? Or could new data be put on the bad drive?
Highlighted
Regular Advisor

Re: pvmove issue

> Have you already extended the _filesystem_ to use the new disks?

Yes. Fortunately, the SAN guys have presented new disks and I am pvmove'ing the data that is readable from the bad disks to the new disks.

Of course, I won't be able to vgreduce those LUNS. So, I'm wondering what will happen to the partially corrupt luns, will new data somehow be written to them again?
Highlighted
Honored Contributor

Re: pvmove issue

if the disk has RD/WR error LINUX has a tendency to put the file system READ only. LVM itself does not fix it. But an fsck can help if the error is correctable. FSCK mean u have to have a backup
Highlighted
Regular Advisor

Re: pvmove issue

> if the disk has RD/WR error LINUX has a tendency to put the file system READ only. LVM itself does not fix it.

The type of error we are seeing on the disk is:

Kernel: SCSI error : <0 0 4 10> return code = 0x20000

Kernel: end request: I/O error dev sdk , sector 432582893

It would seem the Linux' FS table would make those bad sectors off limits ?
Highlighted
Honored Contributor

Re: pvmove issue

did u confirm if the reported error is coming from the suspected failing disk?
Highlighted
Regular Advisor

Re: pvmove issue

> did u confirm if the reported error is coming from the suspected failing disk?

I used
# dd if=/dev/sdk of=/dev/null bs=1024k

To look for i/o errors and, yes, I found some. Additionally, I can't run fsck on the disk since I can't umount them now.