Online resize2fs: Operation not permitted While trying to add group #14080

[Liam Curtis]

Hello All,

I am running Oracle Enterprise Linux x64 5.5 on an HP Proliant DL380 G6 attached to various SANs via fibre channel. I have set up multipath, edited multipath.conf, lvm.conf, and after some trial and error, everything behaves and looks as it should in regard to multipath.

However, when I try to resize a multipath ext3 lvm filesystem, The operation runs for a while, but then dies and I get the above error. I then see this error in /var/log/messages:

EXT3-fs warning (device dm-9): ext3_group_add: No reserved GDT blocks, can't resize

-I am only using the /dev/mapper/ paths to the filesystem (ie, resize2fs /dev/mapper/testvol-lvol1 128G)
-if I unmount the filesystem, resize2fs works fine
-Have tried removing and re-adding journal file via tune2fs with no luck
-The VG has plenty of room in it and is composed of a 100GB and 200GB partition.
-Inode resize is on in the filesystem

I am wondering if online resize2fs is only for small filesystems or not recommended as a best practice or just not generally done??? As I mentioned, the operation works fine offline if I unmount the filesystem, but this won't be an option once system is in production.

Any ideas? Thanks in advance!

[J. Maestre · Accepted solution]

Do you see reserved GDT blocks if you run dumpe2fs on the device?

[Liam Curtis]

Thanks for the reply. When I create the filesystem, the reserved GDT blocks are at 127. After the error, there are none and even though resize_inode is part of system features, there is no longer even an entry for reserved GDT blocks. Is there a way that I can increase the reserved GDT blocks for the filesystem?

[J. Maestre]

127 reserved GDT blocks should allow your filesystem to grow to roughly 127 times it's initial size. Does your resize operation fit there?


I don't think you can change the number of reserved GDT blocks, you would have to recreate the filesystem and provide a "-E resize=" value.

[Liam Curtis]

127 reserved GDT blocks should allow your filesystem to grow to roughly 127 times it's initial size. Does your resize operation fit there?

I don't think you can change the number of reserved GDT blocks, you would have to recreate the filesystem and provide a "-E resize=" value.

I have tried various sizes with same results. In this case I started with a very small size of a few MB and grew it out to 50GB okay, but then it blew up when I tried to resize further. I will try some different initial sizes.

I guess I was certainly spoiled with HP-UX and onlineJFS/vxfs!

[J. Maestre]

Also check the filesystem journal size, just in case. A very small journal will also cause online resize to fail.

If that's the case, you can use tune2fs to delete the journal and recreate it. It should automatically assign a proper size (likely ~128MB).

[Steven E. Protter]

Shalom,

My experience shows resize2fs does work exactly like HP-UX Online JFS

I think the tunefs (you can tune a fs but not a fish), will actually resolve this problem.

SEP

[Liam Curtis]

Shalom,

My experience shows resize2fs does work exactly like HP-UX Online JFS

I think the tunefs (you can tune a fs but not a fish), will actually resolve this problem.

Shalom, Steven ;)

I find that with OnlineJFS, resizing is pretty much instantaneous, whereas with Linux and resize2fs, the resize operation takes quite a while--several minutes or more until it bombs in my case.

I had tried deleting and recreating journal, but that did not help. It seems that once GDT blocks are exhausted, that is that as far as online resizing. I must then do an fsck.

Am I perhaps missing something?

[Matti_Kurkela]

Back in RHEL 4 or so, when there was a specific "ext2online" tool for on-line filesystem extension, there was also a "ext2prepare" tool for increasing the GDT block reserve. Using "ext2prepare" was always an off-line operation, because it needed to move things around inside the filesystem to get more contiguous space for future GDT blocks.

Later the functionality of "ext2online" was integrated to "resize2fs", and currently "ext2prepare" is no longer updated. Perhaps the functionality of "ext2prepare" was also integrated to resize2fs, so that it'll automatically replenish the GDT block reserve whenever you use it to extend the filesystem in off-line mode? (This is just a guess; I haven't tested it. I tried looking at the resize2fs source but it was too complicated for me at the moment (perhaps not enough caffeine?))

When you install a system for production, your original filesystem size choices should be based on informed estimates based on expected usage and data size, not wild guesses. If you originally don't have the information to make those estimates, then that's a good reason to make a complete re-install to make the system as clean and as efficiently laid out as possible before the system goes to production.

Perhaps this would also be a good argument for keeping the testing an production systems completely separate from each other, instead of "promoting" test systems to production as-is.

But sooner or later you *will* need to do some kind of maintenance to the production system too (as nothing human-made is absolutely perfect). If the system is so critical that it absolutely cannot have even pre-scheduled maintenance breaks, then it should also be critical enough to have a totally independent failover system which can instantly take over with a minimum of fuss.

If such a failover cannot be implemented for your application, then the application is clearly unsuitable for continuous mission-critical operation; no excuses.

MK

[chris huys_4]

Hi Liam,

LC> I guess I was certainly spoiled with HP-LC> UX and onlineJFS/vxfs!

Yes, you were.

LC> Am I perhaps missing something?

Install (payable) vxfs instead, for the filesystem part, and (payable) vxvm, for the volumemanager part, on the OEL system, and use that, problem solved. ;)

Greetz,
Chris