Discrepancy between du and bdf

[albert_hua]

Hi, everyone.
I have encountered one big problem.
Here is the output of du and bdf
# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 188416 181664 6389 97% /

# du -akx / | sort -nr | more
35729 /
17717 /sbin
17689 /etc
10857 /etc/lvmconf
4689 /etc/opt
4526 /etc/opt/resmon
3532 /sbin/fs
3447 /etc/opt/resmon/lbin
2488 /etc/opt/resmon/lbin/mibmond
1657 /sbin/fs/vxfs
1454 /sbin/fs/hfs
971 /etc/lvmconf/pcemc2.conf
957 /etc/lvmconf/pcemc5.conf.old
957 /etc/lvmconf/pcemc5.conf
955 /etc/lvmconf/pcemc3.conf
953 /etc/lvmconf/pcemc7.conf
951 /etc/lvmconf/pcemc4.conf.old
....
....

So there is a big discrepancy between their output.

So I am very confused about such an phenomenon.
And I tried to use 'lsof' to find the reason.

nfskd 1762 root 2w VREG 64,0x3 145633280 3046 / (/dev/vg00/lvol3)
bin 2946 root 1w VREG 64,0x3 145633280 3046 / (/dev/vg00/lvol3)
bin 2946 root 2w VREG 64,0x3 145633280 3046 / (/dev/vg00/lvol3)
uvdlockd 2979 root 1w VREG 64,0x3 145633280 3046 / (/dev/vg00/lvol3)
uvdlockd 2979 root 2w VREG 64,0x3 145633280 3046 / (/dev/vg00/lvol3)


145,633,280 must be abnormal because normally the size of the / filesystem is 40MB or so.

So could you give me some invaluable advises?
Thanks a lot.


Al.

[Christian Gebhardt]

Hi

there was the same question this day http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x2b8b402f24d5d61190050090279cd0f9,00.html

Please read and tell us if there is a similar problem

Chris

[albert_hua]

No Christian:
I have met the phenomenon you mentioned. That 's the question of nested file system. But my situation is not the same. The / filesystem begins to be full suddenly.


Al.

[Animesh Chakraborty]

Hi,
The du command gives the number of 512-byte blocks allocated for all
files and (recursively) directories within each directory and file
specified by the name operands.
So 35729 * 512 =18293248 Bytes means ~182 MB which nearly your bdf output.
Regarding the "sudden fillup of /" ,look for any core file or /dev/rmt/om instead of Zero it is O.

[albert_hua]

Hi,animesh:
I have used '-k' option.
So 35729 * 512 =18293248 Bytes
18,293,248 it should be 18MB.

And no large and new core files are found in / filesystem.


Al.

[harry d brown jr]

(1) using lsof, find what files DON'T exist any more but are being REPORTED by bdf

(2) don't have a stroke if bdf and du show differences, its very possible and maybe in your situation, normal. Files can be open and consuming space, but du can't see them, and bdf can, especially files that have been deleted while still open!!!!!

(3) The / filesystem is 188 MEGS not 40 MEGS

(4) What does 145633280 have to do with disk space?

live free or die
harry

[harry d brown jr]

The -k option gives it in 1024 multiples.

Again, the two numbers will NEVER be the same. It's just one of those things you just accept as normal, even though its strange - like allowing democrats to vote.

live free or die
harry

[Bill Hassell · Accepted solution]

The du vs. bdf question is fairly common and the answer is easy: sparse files. A sparse file is one that has been created with 'holes' or unwritten parts. Consider writing a file with record #1, then by seeking to record one million, writing another record and then closing the file. The file has but two records and occupies only a couple of blocks but the missing parts are not stored nor counted in bdf(1) or du(1). Depending on the size of the file and the spareseness, the difference in apparent versus actual size may be VERY large.

Create your own sparse file with:

dd if=/etc/issue of=/var/tmp/sparse bs=2048k seek=1

where you will see the original file is just a few dozen bytes, the result with ls -l or wc -c shows a 2 meg file, but du(1) will show the file as occupying just a bit more than the original /etc/issue file. A cp(1) of the file will create a new file that is the same size (using ls -l or wc -c) but du(1) will now show a MUCH larger size than the original file.

A common file that is often sparse is a core file from a crashed program. Another file that might be sparse is /etc/mail/aliases.pag. And of course, any programmer could write special code that creates a sparse file.

Now to troubleshoot your / filesystem, use du as in:

du -kx / | sort -rn | head

Look for big directories at the top of the list. This is fairly normal:

35855 /
18926 /sbin
14000 /etc
6811 /etc/opt

NEVER allow non-HP-UX directories to exist in the / filesystem (mountpoints are fine).

Then look for big files with:

ll /sbin | sort -rnk5 | more

[Hrvoje Dubravica]

Hi,

I had similar problem. I had users with one aplication which intensively writes to some files and then deletes those files. Problem is that space on filesystem is not free up until process is not finished ( du thinks it is because read from inode table but superblock is not updated ) precisely until process release file. We solve problem by restarting aplication periodicly. Same problem is with oracle database. When dba deletes one datafile, space on filesystem is not free up, until this process die ( release file). Workarround is (db case) to truncate this file, and then delete it.

I hope this help.

BR,

Hrvoje

[Theresa Patrie]

Hi Harry,
Just reading throught this thread and got a good chuckle from your comment. Anytime you can get a good dig about a democrat! 10 points for that one!
Theresa

[albert_hua]

Hi,Bill:
Really appreciate your excellent reply.
It tells me a lot.
I know your meaning is there must be some files with large size which can not be countered by du(1). We can find them by 'll'.
I think I can write one shell to find out the files for me.

Hi,Harry:
(4) What does 145633280 have to do with disk space?

nfskd 1762 root 2w VREG 64,0x3 145633280 3046 / (/dev/vg00/lvol3)

145,633,280 is the size of the file opened by nfskd and / is the name of the mount point on which the file resides so I am confused which file occupies so large space.

bin 2946 root 1w VREG 64,0x3 145633280 3046 / (/dev/vg00/lvol3)
bin 2946 root 2w VREG 64,0x3 145633280 3046 / (/dev/vg00/lvol3)
uvdlockd 2979 root 1w VREG 64,0x3 145633280 3046 / (/dev/vg00/lvol3)
uvdlockd 2979 root 2w VREG 64,0x3 145633280 3046 / (/dev/vg00/lvol3)

There are also other processes opening such a large 'file'.

Because I am not very familiar with the usage of 'lsof'. So the output really makes me lost.

Can anyone explain it for me?
Any advises will be appreciated.


Al.