topic Re: Missing free file system space in Operating System - HP-UX

Missing free file system space

Rob Beglinger — Wed, 30 May 2007 13:01:47 GMT

I have a file system that is mysteriously missing space. bdf and df are reporting more space used than du.

root@foo $ bdf .
Filesystem kbytes used avail %used Mounted on
/dev/vg01/lv_out
524288 341542 182746 65% /foo/out
root@foo $ pwd
/foo/out
root@foo $ du -ks
244468 .
root@foo $ df .
/foo/out (/dev/vg01/lv_out): 365270 blocks 382290 i-nodes

ls -alR does not show any "Hidden Files" that would account for the missing 100 MB.

I need to reclaim lost space, any suggestions?

Re: Missing free file system space

Sandman! — Wed, 30 May 2007 13:06:42 GMT

Common problem owing to space held by a process holding open a deleted file. Look for it with lsof as follows:

# lsof +aL1 /dev/vg01/lv_out

Re: Missing free file system space

James R. Ferguson — Wed, 30 May 2007 13:17:04 GMT

Hi Rob:

To add, see the manpages for 'unlink(2)' for more information.

A very common programming technique is to create a temporary file and immediately unlink() it. This leaves the file (and its space) available for the duration of the program but automatically causes its removal when the program using it terminates. One advantage is that no epilog (cleanup) code is necessary to write.

Regards!

...JRF...

Re: Missing free file system space

Rob Beglinger — Wed, 30 May 2007 14:05:19 GMT

Sandman:

The lsof reveals nothing. When I run the command it runs for a couple seconds and the just returns to the command prompt. And fuser doesn't show anything out of the ordinary either.

James:

If I had a file that had been unlinked but hasn't been released, how would I track it down?

Re: Missing free file system space

James R. Ferguson — Wed, 30 May 2007 14:38:20 GMT

Hi (again) Rob:

# lsof -a +L1 /dev/vg01/lv_out

(or)

# lsof +D /dev/vg01/lv_out +L1

Look for any files with an NLINK value of zero (0). These would be files with a zero link count that will vanish when the last process terminates. The SIZE/OFFSET column will offer the character size of the file in question.

Regards!

...JRF...

Re: Missing free file system space

Sandman! — Wed, 30 May 2007 14:39:32 GMT

Another thing to check would be to find out if "/foo/out" is a nested mount point i.e. has other mount points below it. If that's the case then cmds below should give identical results for the "kbytes used" column.

# bdf /foo/out
# du -kxs /foo/out

the "-x" token prevents du(1) from crossing mount points.

Re: Missing free file system space

Rob Beglinger — Wed, 30 May 2007 15:05:39 GMT

James

The version of lsof I have 4.76 did not show any NLINK value from either command, so I downloaded the latest version 4.78 and compiled it. Here is the command and output that I used.

root@foo $ ./lsof -a +L1 /dev/vg01/lv_out
root@foo $ ./lsof +D /foo/out
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ksh 7471 bill cwd DIR 64,0xa0003 9216 8256 /foo/out/DIR
ksh 20055 bill cwd DIR 64,0xa0003 18432 8257 /foo/out/DIR/ARCHIVE
ftp 24761 bill cwd DIR 64,0xa0003 9216 8256 /foo/out/DIR
ftp 24761 bill 6u REG 64,0xa0003 3898348 8216 /foo/out/DIR/filename

Sandman
This is not a nested mount point.
root@foo $ bdf |grep /foo/out
524288 348982 175306 67% /foo/out

Re: Missing free file system space

James R. Ferguson — Wed, 30 May 2007 15:44:18 GMT

Hi (again) Rob:

# lsof +D /mountpoint +L1

...I think that you forgot to specify "+L1".

For example, if you do:

# perl -e '$f="/tmp/sparse";open(FH,">",$f) or die;seek(FH,1024*1024,1);print FH "ok\n";unlink $f;sleep 120' &

# lsof +D /tmp +L1 -o | grep perl

...you will see something like:

perl 27248 root 3w VREG 64,0x6 0t1048576 0 66 /tmp (/d
ev/vg00/lvol6)

...where the "0t1048576" is the file OFFSET value and the zero (0) in the next column is the NLINK value.

Regards!

...JRF...

Re: Missing free file system space

Sandman! — Wed, 30 May 2007 15:50:36 GMT

The lsof command line should be...

# lsof +aL1 /foo/out

not -> lsof -a +L1

Re: Missing free file system space

Rob Beglinger — Wed, 30 May 2007 16:00:26 GMT

James

Thanks for the added clarification on your command, I am now seeing the output you expected, but not what is using the space.

root@foo $ ./lsof +D /foo/out +L1 -o |grep out
_appl 11372 bill 33u REG 64,0xa0003 0t3625984 1 8249 /foo/out/DIR/filename
root@foo $ bdf /foo/out
Filesystem kbytes used avail %used Mounted on
/dev/vg01/lv_out
524288 189760 334528 36% /foo/out
root@foo $ du -ks /foo/out
95111 /foo/out

Sandman:

When I run your command it returns nothing.

Re: Missing free file system space

Patrick Wallek — Wed, 30 May 2007 16:07:40 GMT

What do you get if you omit the '|grep out' from your lsof command? Any change?

Re: Missing free file system space

Rob Beglinger — Wed, 30 May 2007 16:10:30 GMT

Patrick,

When I omit the |grep out, I get every file access on the server.

Re: Missing free file system space

James R. Ferguson — Wed, 30 May 2007 16:10:51 GMT

Hi Rob:

Wait a second.

When you do 'du -ks' you are reporting in 1K blocks, NOT chunks of 512. Try:

# bdf /foo/out
# du -s /foo/out
# df /foo/out

...The values should be fairly close.

Regards!

...JRF...

Re: Missing free file system space

Rob Beglinger — Wed, 30 May 2007 16:16:10 GMT

James,

Here's what I get:

root@foo $ bdf /foo/out
Filesystem kbytes used avail %used Mounted on
/dev/vg01/lv_out
524288 349152 175136 67% /foo/out
root@foo $ du -s /foo/out
503910 /foo/out
root@foo $ df /foo/out
/foo/out (/dev/vg01/lv_out): 350470 blocks 382518 i-nodes

Thank you all for your continuing help.

BTW the platform I am running on is HP-UX 11.11 PA-RISC in a MC-ServiceGuard cluster. Not sure it that should make a difference for this problem.

Re: Missing free file system space

Sandman! — Wed, 30 May 2007 17:06:43 GMT

If taking this filesystem offline is okay then you could stop all processes that are using it; unmount "/foo/out" and mount it back (alongwith starting every process that uses "/foo/out") to see if that fixes the problem.

Re: Missing free file system space

Rob Beglinger — Wed, 30 May 2007 17:10:09 GMT

Sandman

I was hoping to avoid that, but I think you may be correct. I will let you know how it goes.

Thanks for all the help,

Rob

Re: Missing free file system space

Rob Beglinger — Thu, 13 Sep 2007 16:14:50 GMT

The unmount remount worked thanks for all the help

Re: Missing free file system space

Rob Beglinger — Thu, 13 Sep 2007 16:15:53 GMT

Unmounting and remounting the file system worked.