Missing free file system space

[Rob Beglinger]

I have a file system that is mysteriously missing space. bdf and df are reporting more space used than du.

root@foo $ bdf .
Filesystem kbytes used avail %used Mounted on
/dev/vg01/lv_out
524288 341542 182746 65% /foo/out
root@foo $ pwd
/foo/out
root@foo $ du -ks
244468 .
root@foo $ df .
/foo/out (/dev/vg01/lv_out): 365270 blocks 382290 i-nodes

ls -alR does not show any "Hidden Files" that would account for the missing 100 MB.

I need to reclaim lost space, any suggestions?

[Sandman!]

Common problem owing to space held by a process holding open a deleted file. Look for it with lsof as follows:

# lsof +aL1 /dev/vg01/lv_out

[James R. Ferguson]

Hi Rob:

To add, see the manpages for 'unlink(2)' for more information.

A very common programming technique is to create a temporary file and immediately unlink() it. This leaves the file (and its space) available for the duration of the program but automatically causes its removal when the program using it terminates. One advantage is that no epilog (cleanup) code is necessary to write.

Regards!

...JRF...

[Rob Beglinger]

Sandman:

The lsof reveals nothing. When I run the command it runs for a couple seconds and the just returns to the command prompt. And fuser doesn't show anything out of the ordinary either.

James:

If I had a file that had been unlinked but hasn't been released, how would I track it down?

[James R. Ferguson]

Hi (again) Rob:

# lsof -a +L1 /dev/vg01/lv_out

(or)

# lsof +D /dev/vg01/lv_out +L1

Look for any files with an NLINK value of zero (0). These would be files with a zero link count that will vanish when the last process terminates. The SIZE/OFFSET column will offer the character size of the file in question.

Regards!

...JRF...

[Sandman!]

Another thing to check would be to find out if "/foo/out" is a nested mount point i.e. has other mount points below it. If that's the case then cmds below should give identical results for the "kbytes used" column.

# bdf /foo/out
# du -kxs /foo/out

the "-x" token prevents du(1) from crossing mount points.

[Rob Beglinger]

James

The version of lsof I have 4.76 did not show any NLINK value from either command, so I downloaded the latest version 4.78 and compiled it. Here is the command and output that I used.

root@foo $ ./lsof -a +L1 /dev/vg01/lv_out
root@foo $ ./lsof +D /foo/out
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ksh 7471 bill cwd DIR 64,0xa0003 9216 8256 /foo/out/DIR
ksh 20055 bill cwd DIR 64,0xa0003 18432 8257 /foo/out/DIR/ARCHIVE
ftp 24761 bill cwd DIR 64,0xa0003 9216 8256 /foo/out/DIR
ftp 24761 bill 6u REG 64,0xa0003 3898348 8216 /foo/out/DIR/filename

Sandman
This is not a nested mount point.
root@foo $ bdf |grep /foo/out
524288 348982 175306 67% /foo/out

[James R. Ferguson]

Hi (again) Rob:

# lsof +D /mountpoint +L1

...I think that you forgot to specify "+L1".

For example, if you do:

# perl -e '$f="/tmp/sparse";open(FH,">",$f) or die;seek(FH,1024*1024,1);print FH "ok\n";unlink $f;sleep 120' &

# lsof +D /tmp +L1 -o | grep perl

...you will see something like:

perl 27248 root 3w VREG 64,0x6 0t1048576 0 66 /tmp (/d
ev/vg00/lvol6)

...where the "0t1048576" is the file OFFSET value and the zero (0) in the next column is the NLINK value.

Regards!

...JRF...

[Sandman!]

The lsof command line should be...

# lsof +aL1 /foo/out

not -> lsof -a +L1

[Rob Beglinger]

James

Thanks for the added clarification on your command, I am now seeing the output you expected, but not what is using the space.

root@foo $ ./lsof +D /foo/out +L1 -o |grep out
_appl 11372 bill 33u REG 64,0xa0003 0t3625984 1 8249 /foo/out/DIR/filename
root@foo $ bdf /foo/out
Filesystem kbytes used avail %used Mounted on
/dev/vg01/lv_out
524288 189760 334528 36% /foo/out
root@foo $ du -ks /foo/out
95111 /foo/out


Sandman:

When I run your command it returns nothing.

[Patrick Wallek]

What do you get if you omit the '|grep out' from your lsof command? Any change?

[Rob Beglinger]

Patrick,

When I omit the |grep out, I get every file access on the server.

[James R. Ferguson]

Hi Rob:

Wait a second.

When you do 'du -ks' you are reporting in 1K blocks, NOT chunks of 512. Try:

# bdf /foo/out
# du -s /foo/out
# df /foo/out

...The values should be fairly close.

Regards!

...JRF...

[Rob Beglinger]

James,

Here's what I get:

root@foo $ bdf /foo/out
Filesystem kbytes used avail %used Mounted on
/dev/vg01/lv_out
524288 349152 175136 67% /foo/out
root@foo $ du -s /foo/out
503910 /foo/out
root@foo $ df /foo/out
/foo/out (/dev/vg01/lv_out): 350470 blocks 382518 i-nodes

Thank you all for your continuing help.

BTW the platform I am running on is HP-UX 11.11 PA-RISC in a MC-ServiceGuard cluster. Not sure it that should make a difference for this problem.

[Sandman!]

If taking this filesystem offline is okay then you could stop all processes that are using it; unmount "/foo/out" and mount it back (alongwith starting every process that uses "/foo/out") to see if that fixes the problem.

[Rob Beglinger]

Sandman

I was hoping to avoid that, but I think you may be correct. I will let you know how it goes.

Thanks for all the help,

Rob

[Rob Beglinger]

The unmount remount worked thanks for all the help

[Rob Beglinger]

Unmounting and remounting the file system worked.