bdf lies about how many inodes are free

[Ken Stailey]

I have an 11i system that logs messages about running out of inodes but when I run "bdf -i" (or GNU "df -i") there are over 100,000 free inodes reported.

This is not a timing error. When the filesystem is in this state "touch" will fail to create new files.

I think this bug existed in 11.00 and possibly in 10.20 too. These are very large filesystems as witnessed by the "fstyp" output. We never run either of the fsadm "-d" or "-e" functions.

Particulars:

Running "/usr/sbin/mkfs -F vxfs -m /dev/vgfbr2/rlvol17" reports:

mkfs -F vxfs -o ninode=unlimited,bsize=1024,version=4,inosize=256,logsize=1024,nolargefiles /dev/vgfbr2/rlvol17 131072000

Running "fstyp -v /dev/vgfbr2/rlvol17" reports:
vxfs
version: 4
f_bsize: 8192
f_frsize: 1024
f_blocks: 131072000
f_bfree: 4655454
f_bavail: 4364745
f_files: 9547348
f_ffree: 1163860
f_favail: 1163860
f_fsid: 1074200593
f_basetype: vxfs
f_namemax: 254
f_magic: a501fcf5
f_featurebits: 0
f_flag: 0
f_fsindex: 7
f_size: 131072000

Running "fsadm -F vxfs -D /mnt" reports that there are:
dirs searched 1842422
total blocks 161121
immed dirs 1763792
immeds to add 31
dirs to reduce 1128
blocks to reduce 1157

Running "fsadm -F vxfs -E /mnt" reports that there are:
Total Files 4355682
Average File Blks 28
Average Extents 1
Total Free Blks 4654666
blocks used for indirects: 176
% Free blocks in extents smaller than 64 blks: 5.26
% Free blocks in extents smaller than 8 blks: 0.53
% blks allocated to extents 64 blks or larger: 88.34

[Sandman!]

What does bdf -i tell you?

[Pete Randall]

Have you checked your kernel "ninode" parameter?


Pete

[Pete Randall]

By the way, if you have Glance installed, the System Tables Report will show you your ninode utilization.


Pete

[Ken Stailey]

Have you checked your kernel "ninode" parameter?

http://docs.hp.com/en/939/KCParms/KCparam.Ninode.html

says << ninode defines the maximum number of open inodes that can be in memory. >>

This parameter cannot possibly be the issue. First, ninode controls the size of a table which is a RAM cache so that every iget() operation does not require a disk read. When the limit described by ninode is reached the oldest entries in the inode cache in RAM are expired making room for more.

Next, only one of over 90 filesystems on this box has run out of inodes. There is nothing that associates ninode with a single filesystem.

[A. Clay Stephenson · Accepted solution]

First of all, the kernel tunable ninode only applies to hfs filesystems so that is not applicable. I suspect that your problem is that you created the filesystem with the "nolargefiles" option. That has the intended effect of not allowing individual files to exceed 2GB BUT it has the unintended effect of limiting the internal filesystem data structures which house the inodes to 2GB as well eventhough you used the default unlimited inode allocation option. You will need to run fsadm and set "largefiles". NOTE: This is not the same as the largefiles mount option and, in fact, that mount option does not enable largefiles as the name seems to imply.

[Ken Stailey]

What does bdf -i tell you?

Good question but difficult to answer in that the filesystems on this box are holding live, production data and we currently have to move data between filesystems and make symlinks

If the applications expect /mnt/product/dir1/dir2/file

We have a "secret" indirection layer

/mnt/1/product/dir1/dir2/file

and symlinks like /mnt/product/dir1 -> /mnt/1/product/dir1

so we can shuffle directories around between filesystem on live data by mounting a new filesystem

/mnt/2

and copying /mnt/1/product/dir1 to /mnt2/product/dir1

and then changing the symlink so the update is atomic (indivisible in time.)

/mnt/product/dir1 -> /mnt/2/product/dir1

Then going back and reclaming /mnt/1/product/dir1 via rm -r

New incoming data happens at any time so the inode count is constantly changing. I neglected to save the "bdf -i" data when the limit was hit but right now it looks like this due to clean up:

Filesystem kbytes used avail %used iused ifree %iuse Mounted on
/dev/vgfbr2/lvol17 131072000 126309067 4484901 97% 8305907 1190733 87% /mnt

Using GNU df:
gdf -i .
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vgfbr2/lvol17 9496660 8305931 1190729 87% /repository/rep24

[Ken Stailey]

<< That has the intended effect of not allowing individual files to exceed 2GB BUT it has the unintended effect of limiting the internal filesystem data structures which house the inodes to 2GB as well >>

Oh, this might be it. I'll get back to you after I see about running fsadm to turn on large files.

[Steven E. Protter]

Shalom,

Sounds like a global limit in the kernel may be hit. The an inode limit for the system is being breached.

ninode 476 - ((NPROC+16+MAXUSERS)+32+(2*NPTY))
vnode_cd_hash_locks 128 - 128
vnode_hash_locks 128 - 128
vx_ninode 0 - 0

vx_ninode is probably not set to zero like on this system, which would allow it to go to any number it likes. The problem with leaving it like the above system is there is a huge performance cost.

A. Clay's analysis of filestructure is also a likely cause, except the individual filesystems do not show being out of inodes. I can't explain that discrepency.

btw, computer programs are incapable of lying. They merely produce inaccurate results. Have you looked for patches for this issue?

SEP

[Ken Stailey]

This is not an NINODE issue. I hate to say it but it appears that NINODE is poorly understood.

This appears to be due to hitting the 2GB limit on the size of the structures used to hold inodes on disk.