topic Re: pvmove issue in Operating System - Linux

pvmove issue

P_F — Mon, 02 Feb 2009 23:17:08 GMT

Let me briefly summarize:

LVM2, Kernel 2.6.9, RH AS 4.5

I noticed a series of bad disks in my LVM VG. Since all the VG was full, I requested new disks. Once presented I was then to perform a pvmove to the unallocated PEs on the new disks.

Stupidly, I allocated the space to the LV. Now I can't pvmove since all the extents are used.

1) I'm using LVM2 and there is no online LV reducing command, so I have to use resize2fs to eventually unallocate the new disk, so I can pvmove the stuff off the old bad disk.

A) is resize2fs safe?

B) unmounting - which is required with resize2fs - will disrupt production and cause delays, I'd rather avoid this option.

2) I guess I could wait til the morning, and request new storage and do it right.

Any options. Am I missing something?

Re: pvmove issue

Ivan Ferreira — Tue, 03 Feb 2009 00:51:57 GMT

Here you have the procedure to reduce the file system:
http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1233622132384+28353475&threadId=1160556

1) I'm using LVM2 and there is no online LV reducing command

You have to umount the file system to reduce the size

A) is resize2fs safe?

If you don't make any mistakes. Anyway, I will always have a good backup.

B) unmounting - which is required with resize2fs - will disrupt production and cause delays, I'd rather avoid this option.

Then no other option, maybe copy/rsync?

2) I guess I could wait til the morning, and request new storage and do it right.

Good news ;)

Re: pvmove issue

Matti_Kurkela — Tue, 03 Feb 2009 01:07:15 GMT

Have you already extended the _filesystem_ to use the new disks?

If you've used lvextend but not yet extended the filesystem (using resize2fs/ext2online/whatever), just lvreduce back to the original value *but not any smaller than that*.

Use "tune2fs -l" to verify the current size of the filesystem if you're unsure: block count * block size = filesystem size in bytes. Then see if it matches what lvdisplay says about the LV size.

Remember:
filesystem size > LV size: filesystem is corrupted, some data may be lost
filesystem size < LV size: either a resize operation is half-done, or someone has made a mistake
filesystem size = LV size: OK.

If you've already extended the filesystem, the first priority is to take an extra backup to make sure you don't lose data, whatever happens.

Shrinking the filesystem should be safe when the filesystem is unmounted and error-free. However, shrinking operations are much rarer than extensions, so you should be prepared.

In addition, you're working with known bad disks. That is definitely a risk.

I would take a fresh backup ASAP and "do it right" in the morning.

If the disk fails during the night, I'd try to salvage the very latest data if reasonable, then nuke & recreate the LV and the filesystem using good disks only, then restore. Of course, my situation may be different from yours.

MK

Re: pvmove issue

skt_skt — Tue, 03 Feb 2009 02:19:22 GMT

you could do FS extend online on RHEL for if ext2online is present. But the shrink ops is not the one we are practically done with.

Re: pvmove issue

P_F — Tue, 03 Feb 2009 05:23:51 GMT

Thanks for the responses.

I have a follow-up question, more on LVM.

1)
If an LVM disk develops bad sectors. Does LVM identify those sectors and will attempt not to write to bad disk sectors, or even the disk itself?

2)
Suppose I complete on pvmove, and then extend the new data into the existing VG. Suppose I can't vgreduce the old bad disk, to remove from the VG. Could that data migrate back onto the old drive? Or could new data be put on the bad drive?

Re: pvmove issue

P_F — Tue, 03 Feb 2009 05:31:58 GMT

> Have you already extended the _filesystem_ to use the new disks?

Yes. Fortunately, the SAN guys have presented new disks and I am pvmove'ing the data that is readable from the bad disks to the new disks.

Of course, I won't be able to vgreduce those LUNS. So, I'm wondering what will happen to the partially corrupt luns, will new data somehow be written to them again?

Re: pvmove issue

skt_skt — Tue, 03 Feb 2009 11:23:31 GMT

if the disk has RD/WR error LINUX has a tendency to put the file system READ only. LVM itself does not fix it. But an fsck can help if the error is correctable. FSCK mean u have to have a backup

Re: pvmove issue

P_F — Tue, 03 Feb 2009 12:31:31 GMT

> if the disk has RD/WR error LINUX has a tendency to put the file system READ only. LVM itself does not fix it.

The type of error we are seeing on the disk is:

Kernel: SCSI error : <0 0 4 10> return code = 0x20000

Kernel: end request: I/O error dev sdk , sector 432582893

It would seem the Linux' FS table would make those bad sectors off limits ?

Re: pvmove issue

skt_skt — Tue, 03 Feb 2009 15:14:19 GMT

did u confirm if the reported error is coming from the suspected failing disk?

Re: pvmove issue

P_F — Tue, 03 Feb 2009 15:29:16 GMT

> did u confirm if the reported error is coming from the suspected failing disk?

I used
# dd if=/dev/sdk of=/dev/null bs=1024k

To look for i/o errors and, yes, I found some. Additionally, I can't run fsck on the disk since I can't umount them now.

Re: pvmove issue

P_F — Tue, 03 Feb 2009 20:06:55 GMT

For those following this thread...

The corruption got so bad that I couldn't extend the new disks with the old disk's date pvmoved onto them - because there was a general filesystem corruption... this was after the 5th disk had been moved.

Anyway, the forced me to reboot and disk so with a -F, after a very lenghty fsck it cleared and is up.

So, the FS table ought to have fixed a lot of the mess.

Thanks for the info to all who responded.

Re: pvmove issue

P_F — Tue, 03 Feb 2009 20:08:20 GMT

General information useful

Re: pvmove issue

skt_skt — Wed, 04 Feb 2009 00:33:11 GMT

what is that -F u r saying?

Re: pvmove issue

P_F — Wed, 04 Feb 2009 01:01:52 GMT

> What is the -F ?

# shutdown -rF now