file system full and inodes

[Scott_20]

Interesting problem this weekend. We had a LVM file system fill up ( several hundred 4k files) and it was determined to be an inode issue. this f/s was at 97%, we rebooted,and the f/s came up at 1% utilized.
Can someone explain this to me? Do I need to reboot each time? we are running 11.i.

[Pete Randall]

Scott,

Questions, questions, questions:

Who determined this was an inode issue? How did they determine that? What type of file system is it? HFS? VxFS? If it's VxFS (or Journalled) it's not likely to be an inode issue.

As far as why the utilization dropped after reboot, it's probably because there was a process holding a file open. After the reboot, the process is gone and the space is reclaimed.


Pete

[A. Clay Stephenson]

First of all there is no such thing as an LVM filesystem; LVM is the thingy that houses the filesystem. In vxfs filesystems, inodes are dynamically allocated as needed but in hfs filesystems they are fixed. I suppose what happened is that you had a herd of temporary files that had been rm'ed but the processes that were using them still had the files open.

[Patrick Wallek]

Is this filesystem VxFS or HFS? If it is VxFS then you should never have an inode problem.

The reason the file system came up as only 1% used after you rebooted is probably because there were files that had been rm'ed but still had a process or processes accessing them. When you rebooted, those processes went away, thus allowing the file to finally be removed.

When you rm a file, it is not actually removed until any and all processes that are accessing it finish and go away.

You don't need to reboot every time this occurrs, just make sure, via lsof or fuser before you rm the file, that whatever processes are accessing the files you want to remove finish or somehow are made to go away.

[GK_5]

Sometime, when you delete files if any hung process was using that file, then the file system will still count the space utilized that file. It will release the space only when those processes are killed.
After reboot those processes got killed and it released the space. That is why you are seeing space after reboot.

[Scott_20]

It is a vxfs file system. are inodes dynamic in this case? can someone explain.
thanks,
s

[Pete Randall]

Scott,

Yes, since it's a VxFS file system, the inodes are allocated dynamically. That means that there is no hard set limit, as is the case with HFS file systems, and it is highly improbably that there was any "inode issue". As has been stated, the process which had all those "rm'd" files open has to terminate before the space occupied by those files will be released. Re-booting was the anwer in this case, but finding and terminating the offending process would be just as effective in the future.


Pete

[A. Clay Stephenson]

Unless you specifically overrode the default behavior when the vxfs filesystem was created, the inodes are automatically added "on the fly" -- as long as extents are available and in the rare case of largefiles not being enabled it is possible to limit the number of inodes when the underlying data structures exceed 2GB.

Do this:
mkfs -F vxfs -m /dev/vg01/lvol3 (substituting the correct device node for the filesystem in question). The -m option directs mkfs to display the command line arguments used to create the filesystem BUT be absolutely certain to specify -m so a new filesystem will not be created. You should see a value "ninode=unlimited" or "ninode=0"; these indicate dynamic inode allocation; any other non-zero value indicates static inode allocation.

[A. Clay Stephenson · Accepted solution]

Unless you specifically overrode the default behavior when the vxfs filesystem was created, the inodes are automatically added "on the fly" -- as long as extents are available and in the rare case of largefiles not being enabled it is possible to limit the number of inodes when the underlying data structures exceed 2GB.

Do this:
mkfs -F vxfs -m /dev/vg01/lvol3 (substituting the correct device node for the filesystem in question). The -m option directs mkfs to display the command line arguments used to create the filesystem BUT be absolutely certain to specify -m so a new filesystem will not be created. You should see a value "ninode=unlimited" or "ninode=0"; these indicate dynamic inode allocation; any other non-zero value indicates static inode allocation.

Man mkfs_vxfs for details.