discrepancy between bdf and du on migrating filesystem - DNLC

[INCS Dept.]

Hi guys and girls,

We do have a problem. We are running omnistorage (OST) 3.12 on 11.0 which uses a filesystem which is 36 GB in size (vxfs). The command du shows that approx. 5 GB is in use, bdf shows about 33 GB. The remaining 28 GB seems te be lost.

We've asked HP about this, but the experts have no clue. The question pops up that the DNLC keeps the open files from being released and talks about fsdb etc. etc.

Does anybody have a clue ? About OST and missing GB's ? About fsdb ?

Any suggestions are welcome.

Thanks in advance,

INCS

[Robert-Jan Goossens]

Hi INCS,

Just some things that i'm thinking. But I must say 28 GB is a large discrepancy (:-)

open files
You could use lsof to check.

files underneath the filesystem.
umount the filesystem and check the mountpoint (directory)

Maybe more later.

Regards,
Robert-Jan

[Fabio Ettore]

Hi,

the explaination is the following:

'du' checks for file existence, whereas 'bdf' also accounts for reserved
space. The 'bdf' command will not report available disk space until the
process that reserved the space closes. For example, if you remove the
syslog.log file, 'bdf' will not report the space until syslogd dies.

This is an example, anyway if you search in technical knowledge base with keybords 'du and bdf' you should find many many docs about this gap between two commands.
fsck or a reboot should delete this gap.

Anyway 28 GB is really big!!!
Have you deleted very big files in latest days?

Best regards,
Ettore

[Umapathy S]

I feel a combination of fuser and lsof will do the trick. You can know by sure the bdf is still reporting with the holding space.

This has been discussed many times previously.
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=138609

HTH,
Umapathy

[Bruno Ganino]

I agree with Ettore.
The difference is due to sparse files. These are files that are only
partially populated with data.

"bdf" shows the directory structure
info which includes data areas that could be occupied if the areas
were filled in.

"du" shows the actual space.

If you copy the /home
directory to another location, the sparse files will be filled in
and then bdf and du will agree.

HTH
Bruno

[INCS Dept.]

Hi,

No big files (filesystem contains about 3.6 million small files, lsof did not work, unmounting and a reboot did not work either.

We installed the latest patches for du, bdf, zdf, zdu, vxfs, etc.

There are no other filesystems mounted on top of this filesystem, so no files are hidden.

Yes, I know the difference between bdf and du. I've read lots about this.

So, nothing that is mentioned above gives a clue. I start thinking about DNLC and debugging.

Anybody ?

Thanks,

INCS.

[KapilRaj]

Is it due to a large fragment size for the filesystem which u r using and copied large number of small files in it ???..

Just a wild guess still 28 GB is a bit high !

lsof |grep "" will list out processes who has opened this filesystem .. check for any process and try terminating one by one !!

Kaps

[Michael Schulte zur Sur]

Hi,

have you checked the filesystem for inconsistencies, and the mountpoint directory for otherwise hidden files as Robert suggested? Though I would tend to think, that in this case the loss would accounted to root. One radical advice, move the file to another place and redo the filesystem. Can you post the output of bdf and du -k as attachment?

greetings,

Michael

[Bruno Ganino]

I don't know well, but look the dnlc_hash_locks and ncsize Tunable Kernel Parameter.
In documentation i read:

dnlc_hash_locks - number of locks for the Directory Name Lookup Cache

The minimum value allowed is 16. The maximum value allowed is 8192.
The value is further constrained in that it must be be a power of 2, and it must be equal to or less than one eighth the number of DNLC entries (ncsize >= 8 * dnlc_hash_locks).

ncsize - number of Directory Name Lookup Cache entries

The minimum value allowed is 128. The maximum value allowed is memory limited.
The value is further constrained in that it must be equal to or greater than eight times the value of the number of locks for the DNLC (ncsize >= 8 * dnlc_hash_lock).

HTH
Bruno

[Bruno Ganino]

INCS, Sorry I have mistaken forum...
Not to read my previous answer.
Bruno

[Tony Horton]

3,600,000 * 8KB is roughly = to 28GB :-)

Now I'm not sure but I suspect maybe you have your vxfs blocksize set at 8KB. Maybe a good idea to try changing it to 1KB and see whether your useage drops to around 3.5GB

Regards,

Tony.

[INCS Dept.]

When I review the answers given until now it seems not totally clear that we are dealing with a MFS which consists at this moment of 1 36 GB harddisk and 10 WORM's (9 GB each). So, the total MFS space is ~ 126 GB.

[Michael Schulte zur Sur]

Hi,

so far you havent been very precise in your statements and giving everybody for their efforts to help you is hard. :-(
Also you havent commented on the block size issue. Can anyone comment on what MFS is? I couldnt find anything on it.

dont give up on us,

Michael

[INCS Dept.]

Hi,

MFS stands for migrating filesystem. It's an extended vxfs filesystem so files can be put/retrieved to/from optical disks. It's just an ordinary filesystem with default blocksize. It can hold large files though.

So it is a filesystem that needs to have extra attibutes (like: on which optical platter is my file so i can migrater it back onto magnetic disk). It looks like file are on magnetic disk, but if I check this they are completely on optical disk.

That's why I and HP started to lokk at other things like DNLC and fsdb.

Thanks,

INCS.

[Bruno Ganino]

Michael, i know that MFS is the file system original of Macintosh.
It has been replaced from HFS/HFS+

INCS is true?

Bruno

[INCS Dept.]

MFS stands for Migrating File System and has nothing to do with Mac or HFS.

MFS is used in conjunction with an optical jukebox and the HP OmniStorage product. The data residing in a MFS can be transparantly migrated between harddisk and MO(=Magneto-Optical).

[A. Clay Stephenson]

This is actually typical behavior for OmniStorage. What you are seeing is only the magnetic cache values. You should install the OST 3.12 cumulative patch PHSS_28461 which implements the "zdu" command that will display total usage, magnetic, optical, and tape (if applicable).

[INCS Dept.]

This patch has been already installed for a couple of weeks.

And indeed the issue is the magnetic cache utilisation.

zdu -kls gives:
total of 56613275 existing of 128512 on magnetic cache and 56484763 on WORM

du -ks gives (on magnetic cache):
128512

However bdf gives:
35557376 33224728 2314432 93%

So according to (z)du the utilisation of the 36GB disk is 0,36% and to bdf the utilisation is 93,44%. And the big question is what happened to the other 93,08%??????

[Fabio Ettore]

- Have you already tried to umount/fsck/mount that filesystem?
- Have you tried a reboot?

Of course they are not good solutions but as 'extrema ratio' you could try them.

Best regards,
Ettore

[A. Clay Stephenson]

Your fundamental problem is that your are expecting vanilla commands (bdf, du, df, ...) to work with VBFS (very Big FileSystem/ Migrating File Systems). The only commands that should be used on VBFS filesystems are the 'z' commands. While it is true that VBFS is built on top of a conventional vxfs filesystem the vanilla commands are ignorant of the migrating aspects of the filesystem.

You should also be aware that bdf, du, and df can sometimes give seemingly inconsistant result even on vanilla vxfs filesystems because of the differences in the ways some of the values are calculated and whether or not files which have been unlinked have actually been closed by all processes which had them open.

[INCS Dept.]

Ettore Rossi,

you're suggestions have been done a dozen of times!!!! (HP support also suggested this :-< ). Yeah, right.

A. Clay Stephenson,
if we want to have only information about the magnetic cache the "vanilla" commands should be working fine.
Looking at the output of du and zdu you will see that this is true.
zdu gives info about used blocks on the optical platters in which we are not interested. Therefore, du can be used; we want info on the MFS filesystem which is on disk. It is foolish to say that zdu must be used, LOL.

The linking problems you suggest should be gone after a reboot and this is already performed several times.

HP is also thinking (for a couple of weeks already) about a good explanation for the described behaviour and the impact on the working of the application.

Let's see who is first with a solution....
(and not the solution of extending the volumegroup with extra harddisks ;-< )

[Bart Beeren · Accepted solution]

Hi,

up to some point I do agree with Tony Horton.

Definitely something went wrong with setting up your filesystem. All the inodes needed for referencing the 3.6 million smal files are consuming more diskspace than necessary. Setting up a filesystem of 36 GB gives a standard blocking factor of 8 kB and 1 inode consumes 1 block! (Is your filesystem setup with a blocking factor of 8 kB?!?)

Since an inode on a MFS uses approximately 1.8 kB, it is advised to recreate the filesystem with a blocking factor of 2 kB. This is the optimal blocking factor with regard of performance for a MFS with lots of small files.

After this exercise your overhead will drop from 28 GB to 7 GB.

(p.s.: don't forget to reassign Tony's points)