HPE Community read-only access December 15, 2018
This is a maintenance upgrade. You will be able to read articles and posts, but not post or reply.
Hours:
Dec 15, 4:00 am to 10:00 am UTC
Dec 14, 10:00 pm CST to Dec 15, 4:00 am CST
Dec 14, 8:00 pm PST to Dec 15, 2:00 am PST
cancel
Showing results for 
Search instead for 
Did you mean: 

LVM lvreduce nightmare

 
ganaiwali
Advisor

LVM lvreduce nightmare

Hi,
 I tried to reduce the VG vg0 and this is what it looked like before I tried to reduce it 

Filesystem                       Size  Used Avail   Use% Mounted on
/dev/mapper/vg0-data         2.2T  1.7T  433G  80%  /data
Out of the available 433G I wanted to reduce 100G in vg0 so that I could use it for a new partition and this is what I did ..

e2fsck -f /dev/vg0/data
resize2fs /dev/vg0/data 100G
lvreduce -L -100G -n /dev/vg0/data
lvs
  LV   VG   Attr   LSize Origin Snap%  Move Log Copy%  Convert
  data vg0  -wi-ao 2.08T                                      
after the lvreduce it did what I intended to as you can see in 

vgs
  VG   #PV #LV #SN Attr   VSize VFree  
  vg0    1   1   0 wz--n- 2.18T 100.00G   ( it did allocate the 100.00G as free space )
 
after I mounted /data back , I could even touch a file , however when we start the app (mysql) and when it tries to write into the existing data in /data , the drive goes into read only mode repeatedly ..
in order to fix it i unmounted /data and ran 
e2fsck -f /dev/vg0/data -n

after the fsck completed it would prompt to fix superblock/inodes which I replied with 'yes' however the problem still persists that if i mount /data it goes into read-only mode .
some info  about the server setup
dmsetup table
vg0-data: 0 4477255680 linear 104:17 384
Red Hat Enterprise Linux Server release 5.7
2.6.18-274.17.1.el5 #1 SMP  x86_64
lvm2-2.02.84-6.el5_7.1
I have reduced lvm's with those  sequence of commands in the past but i just dont understand why it seems to have failed this time although i did get 100G free space in vg0 but the partition /data seems useless ..
 I would greatly appreciate any help/insights .. i notice lvm has created a backup file in /etc/lvm/archive as   vg0_00008-1147866134.vg , does restoring from that file actually work ? will it get the drive into original state ? and how can i actually free some space from the volume group if i want to ?
########
some errors spewed in dmesg
attempt to access beyond end of device
dm-0: rw=0, want=4654102632, limit=4477255680
EXT3-fs error (device dm-0): read_block_bitmap: Cannot read block bitmap - block_group = 17145, block_bitmap = 561807360
Aborting journal on device dm-0.
attempt to access beyond end of device
dm-0: rw=0, want=4565762056, limit=4477255680
EXT3-fs error (device dm-0): read_block_bitmap: <2>ext3_abort called.
EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal
Cannot read block bitmap - block_group = 17417, block_bitmap = 570720256
EXT3-fs error (device dm-0): read_block_bitmap: Cannot read block bitmap - block_group = 17417, block_bitmap = 570720256
dm-0: rw=0, want=4683202568, limit=4477255680
EXT3-fs error (device dm-0): read_block_bitmap: Cannot read block bitmap - block_group = 17865, block_bitmap = 585400320
EXT3-fs error (device dm-0) in ext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device dm-0) in ext3_orphan_del: Journal has aborted
__journal_remove_journal_head: freeing b_committed_data
ext3_abort called.
EXT3-fs error (device dm-0): ext3_put_super: Couldn't clean up the journal
kjournald starting.  Commit interval 5 seconds
EXT3-fs warning (device dm-0): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
EXT3-fs warning (device dm-0): ext3_clear_journal_err: Marking fs in need of filesystem check.
EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
attempt to access beyond end of device
dm-0: rw=0, want=4654102632, limit=4477255680
EXT3-fs error (device dm-0): read_block_bitmap: Cannot read block bitmap - block_group = 17145, block_bitmap = 561807360
Aborting journal on device dm-0.
1 REPLY
Matti_Kurkela
Honored Contributor

Re: LVM lvreduce nightmare

Too bad you did not include the output of the commands you ran.

 

Your resize2fs /dev/vg0/data 100G command does not say "shrink this filesystem by 100G": it says "make this filesystem exactly 100G in size". It should have told you "resizefs: New size smaller than minimum", which means it noticed you're trying to shrink the filesystem to smaller size than the data it contains and stopped without doing anything.

(You should have used something like resize2fs /dev/vg0/data 2.1T instead.)

 

Unfortunately, your lvreduce -L -100G -n /dev/vg0/data command did exactly what you told it to do: "make this LV 100G smaller than it currently is". The "attempt to access beyond end of device" messages in the syslog confirm that this part of the procedure was completed without shrinking the filesystem successfully first. Now the filesystem metadata says the filesystem is larger than the LV. When the filesysten driver tries to access some parts of the filesystem, the LVM rejects those operations.

 

Restoring the VG configuration backup is *exactly* what you should do at this point.

First, confirm that you're looking at the correct VG configuration file, by running this command:

vgcfgrestore -l vg0

 It will list all the available VG configuration backups for the vg0 VG, their creation timestamps and a description of the context the backup was created in. You'll want the backup which has a description:

"Created *before* executing 'lvreduce -L -100G -n /dev/vg0/data'".

 

Once you're sure you have the correct backup file, you can restore the VG configuration with a command like this:

vgcfgrestore -f /etc/lvm/archive/vg0_00008-1147866134.vg vg0

 

If the filesystem has not been damaged by your e2fsck attempts, it should be exactly as it was before your unfortunate lvreduce operation.

 

Lesson learned: when doing a critical multi-step operation like shrinking a filesystem, always make sure you understand all the messages displayed by the previous step before proceeding to the next step. If you're uncertain whether the previous step completed successfully or not, check it before going any further.

 

In this case, you could have mounted the filesystem after the resize2fs command and run "df -h". It would have told you the filesystem size was still 2.2T, instead of the expected 2.1T value.

MK