LVM lvreduce nightmare

[ganaiwali]

Hi,

 I tried to reduce the VG vg0 and this is what it looked like before I tried to reduce it 

Filesystem                       Size  Used Avail   Use% Mounted on
/dev/mapper/vg0-data         2.2T  1.7T  433G  80%  /data

Out of the available 433G I wanted to reduce 100G in vg0 so that I could use it for a new partition and this is what I did ..

e2fsck -f /dev/vg0/data
resize2fs /dev/vg0/data 100G
lvreduce -L -100G -n /dev/vg0/data

lvs

  LV   VG   Attr   LSize Origin Snap%  Move Log Copy%  Convert

  data vg0  -wi-ao 2.08T                                      

after the lvreduce it did what I intended to as you can see in 

vgs
  VG   #PV #LV #SN Attr   VSize VFree  
  vg0    1   1   0 wz--n- 2.18T 100.00G   ( it did allocate the 100.00G as free space )

after I mounted /data back , I could even touch a file , however when we start the app (mysql) and when it tries to write into the existing data in /data , the drive goes into read only mode repeatedly ..

in order to fix it i unmounted /data and ran 

e2fsck -f /dev/vg0/data -n

after the fsck completed it would prompt to fix superblock/inodes which I replied with 'yes' however the problem still persists that if i mount /data it goes into read-only mode .

some info  about the server setup

dmsetup table

vg0-data: 0 4477255680 linear 104:17 384

Red Hat Enterprise Linux Server release 5.7

2.6.18-274.17.1.el5 #1 SMP  x86_64

lvm2-2.02.84-6.el5_7.1

I have reduced lvm's with those  sequence of commands in the past but i just dont understand why it seems to have failed this time although i did get 100G free space in vg0 but the partition /data seems useless ..

 I would greatly appreciate any help/insights .. i notice lvm has created a backup file in /etc/lvm/archive as   vg0_00008-1147866134.vg , does restoring from that file actually work ? will it get the drive into original state ? and how can i actually free some space from the volume group if i want to ?

########

some errors spewed in dmesg

attempt to access beyond end of device

dm-0: rw=0, want=4654102632, limit=4477255680

EXT3-fs error (device dm-0): read_block_bitmap: Cannot read block bitmap - block_group = 17145, block_bitmap = 561807360

Aborting journal on device dm-0.

attempt to access beyond end of device

dm-0: rw=0, want=4565762056, limit=4477255680

EXT3-fs error (device dm-0): read_block_bitmap: <2>ext3_abort called.

EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal

Cannot read block bitmap - block_group = 17417, block_bitmap = 570720256

EXT3-fs error (device dm-0): read_block_bitmap: Cannot read block bitmap - block_group = 17417, block_bitmap = 570720256

dm-0: rw=0, want=4683202568, limit=4477255680

EXT3-fs error (device dm-0): read_block_bitmap: Cannot read block bitmap - block_group = 17865, block_bitmap = 585400320

EXT3-fs error (device dm-0) in ext3_free_blocks_sb: Journal has aborted

EXT3-fs error (device dm-0) in ext3_orphan_del: Journal has aborted

__journal_remove_journal_head: freeing b_committed_data

ext3_abort called.

EXT3-fs error (device dm-0): ext3_put_super: Couldn't clean up the journal

kjournald starting.  Commit interval 5 seconds

EXT3-fs warning (device dm-0): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure

EXT3-fs warning (device dm-0): ext3_clear_journal_err: Marking fs in need of filesystem check.

EXT3-fs warning: mounting fs with errors, running e2fsck is recommended

attempt to access beyond end of device

dm-0: rw=0, want=4654102632, limit=4477255680

EXT3-fs error (device dm-0): read_block_bitmap: Cannot read block bitmap - block_group = 17145, block_bitmap = 561807360

Aborting journal on device dm-0.

[Matti_Kurkela]

Too bad you did not include the output of the commands you ran.

Your resize2fs /dev/vg0/data 100G command does not say "shrink this filesystem by 100G": it says "make this filesystem exactly 100G in size". It should have told you "resizefs: New size smaller than minimum", which means it noticed you're trying to shrink the filesystem to smaller size than the data it contains and stopped without doing anything.

(You should have used something like resize2fs /dev/vg0/data 2.1T instead.)

Unfortunately, your lvreduce -L -100G -n /dev/vg0/data command did exactly what you told it to do: "make this LV 100G smaller than it currently is". The "attempt to access beyond end of device" messages in the syslog confirm that this part of the procedure was completed without shrinking the filesystem successfully first. Now the filesystem metadata says the filesystem is larger than the LV. When the filesysten driver tries to access some parts of the filesystem, the LVM rejects those operations.

Restoring the VG configuration backup is *exactly* what you should do at this point.

First, confirm that you're looking at the correct VG configuration file, by running this command:

vgcfgrestore -l vg0

 It will list all the available VG configuration backups for the vg0 VG, their creation timestamps and a description of the context the backup was created in. You'll want the backup which has a description:

"Created *before* executing 'lvreduce -L -100G -n /dev/vg0/data'".

Once you're sure you have the correct backup file, you can restore the VG configuration with a command like this:

vgcfgrestore -f /etc/lvm/archive/vg0_00008-1147866134.vg vg0

If the filesystem has not been damaged by your e2fsck attempts, it should be exactly as it was before your unfortunate lvreduce operation.

Lesson learned: when doing a critical multi-step operation like shrinking a filesystem, always make sure you understand all the messages displayed by the previous step before proceeding to the next step. If you're uncertain whether the previous step completed successfully or not, check it before going any further.

In this case, you could have mounted the filesystem after the resize2fs command and run "df -h". It would have told you the filesystem size was still 2.2T, instead of the expected 2.1T value.