Deleted files not freeing space

[support_5]

Hi folk,

As we should all be aware, if a file is open by a process, but gets deleted, the directory reference to that file gets deleted, and the link count gets decremented by one (usually to 0), but the space won't be freed unless the process holding the file open closes the file.

A classic example is /var/adm/syslog/syslog.log. if that file is deleted, the space isn't recovered until we send a HUP signal to syslogd.

But here is the problem: If a file gets deleted, how can you tell what that file-name was and how can you tell what process you need to kill to free the space?

fuser is useless, as it only reports files you tell it about, although it does list processes, but it's not very useful if you have hundreds of them (which I do in this instance).

Also, lsof is a handy tool, but again useless since it simply tells you that a process has a file open on a particular filesystem, but not the particulars of the file, so once again, if you have hundreds of open files, this isn't very helpful.

What I'm looking for is some sort of tool or methodology where we can easily determine what process(es) is locking a large file so that we can free it up somehow.

I say this because often when a file gets deleted, the space doesn't free up and the all of a sudden the only way to recover that space is to kill every process with files open on that filesyste...which isn't very helpful if it means a system outtage. Surely there is an easy way to determine a single process to kill off to redeem the space?

Any ideas/suggestions?

The only think I can think of is where lsof tells you some information about the size of a file, which may give enough information for you to make an educated guess.

Others?

Thanks.

- Andy Gray

[Duncan Edmonstone · Accepted solution]

If you have access to the HP knowledgbase, have a look at article KBRC00008039

If you have a deleted file in /tmp then

lsof +aL1 /tmp

should find the process responsible.

Also, this is from the lsof FAQ:

Why doesn't lsof report path names for unlinked (rm'd) files?

Lsof never reports a path names for a file that has been
unlinked from its parent directory -- e.g., deleted via
rm, or the unlink() system call -- even when some process
may still hold the file open. That's because the path name
is erased from name caches and the parent directory file
when the file is unlinked.

Unlinked open files are sometimes used by applications for
temporary, but invisible storage (i.e., ls won't show them,
and no other process can open them.) However, they may
occasionally consume disk space to excess and cause concern
for a system administrator, who will be unable to locate
them with find, ls, du, or other tools that rely on finding
files by examining the directory tree.

By using lsof's +L option you can see the link count of
open files -- in the NLINK column. An unlinked file will
have an NLINK value of zero. By using the option +L1 you
can tell lsof to display only files whose link count is
less than one (i.e., zero).


HTH

Duncan

[harry d brown jr]

Using "lsof" ON the file to be DELETED will tell you WHAT has the file open:

lsof /var/adm/syslog/syslog.log

will produce something like this:

[root@rndspt01 /opt/appl]# lsof /var/adm/syslog/syslog.log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
syslogd 558 root 10u REG 64,0x9 11324 24486 /var/adm/syslog/syslog.log
[root@rndspt01 /opt/appl]#


IF a file is OPEN, and you DELETE it, the space it OCCUPIES will NOT be released back to the filesystem UNTIL that process that has it opened closes the file. THIS IS NOT A BUG, just a mere fact of how things are supposed to operate.

USE this in the future to SHRINK files:

cat /dev/null > /var/adm/syslog/syslog.log

This will truncate the file, RELEASING the space back to the filesystem.


live free or die
harry d brown jr

[harry d brown jr]

Options:

(1) Change "rm" and "mv" to be scripts that FIRST verify that the file is not open, before performing the action.

(2) Modify "lsof", as it is opensource and thus available for ENHANCEMENTS.

(3) keep people out of SHELL's

(4) for log files use logrotate: http://hpux.ee.ualberta.ca/hppd/hpux/hpux/Sysadmin/logrotate-2.5/


Other possibilities - see attachment


live free or die
harry d brown jr