Can someone explain this lsof output?

[Jason Martens]

My /var filesystem keeps filling up, and I can't seem to track down the file/process that is responsible. However, using "lsof +f -- /var", it shows some entries that look like this:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sh 13525 root 1w REG 64,0xd 1268427726 27161 /var (/dev/vg00/lvol20)
sh 13525 root 2w REG 64,0xd 1268427726 27161 /var (/dev/vg00/lvol20)

That looks like a 1.2 GB file on /var somewhere, but I can't find it anywhere! Also, what does the 1w and 2w mean for the FD? I think the w means write lock, but what is the number?
Thanks!

[A. Clay Stephenson · Accepted solution]

Sometimes it helps to be a C programmer because then the file descriptor numbers would instantly make sense. File descriptors are small integers that are returned by the open(), creat(), pipe(), socket(), and dup() system calls. A few, normally have designated purposes but these can also be closed and reopened in non-standard ways. In your case, 1w is the shell's stdout, 2w is the shell's stderr. (O is stdin). The 'w' simply means open for writing only.

Man lsof and look under the "OUTPUT" section for a description of the FD column. A man 2 open, man 2 read, man 2 write would also prove helpful.

One of the reasons that you may not see these files is a directory is that they have been unlinked (rm'ed). It is a rather standard UNIX idiom to open a file and immediately unlink it. This has the effect of creating a temporary file that although not longer listed in the directory is none the less available for use until all the processes which opened the file either terminate or close the file descriptor.

Man 2 unlink for this explanation.

[Jason Martens]

Thanks, that makes perfect sense.

[Jeff_Traigle]

How did you try to find it? The NODE value is the inode number so the easiest way to find it is:

find /var -inum 27161

If that fails to return the file name, then most likely it was unlinked as Clay explained. In that case, kill the sh process that has the non-existent file open.

kill -15 13525

[Jason Martens]

Indeed, the find /var -inode command does not find anything. I'll kill the processes. Thanks for your help!