filesystem full

[Rusty Sapper]

I'm seeing some strange behavior on an L-class server running 11.0

I got a report that the /home filesystem was full. I ran a bdf and indeed it reported that the filesystem was 100% full. I then ran a du -k on /home and much to my surprise, it report I was using just over 12,000 K on this 1GB filesystem.

I ran a fuser -c and there were about 20 processes listed. I then ran a fuser -ck to kill the processes.

I again ran a bdf and ,again, much to my surprise, it listed /home as using 1%.

Any Ideas what is going on?

TIA

Rusty

[Jeff Machols · Accepted solution]

It looks like there was a process that had an open file. If a process is writing to a file, but has not closed it yet, it will be using blocks of disk space, but the inodes have not been updated, so an ls, or du may not show the file even though the space is being used. When you killed the process, the file was not longer open and sinced it was never closed (inodes writen), the file was not saved so the disk space was freed up

[A. Clay Stephenson]

Hi Rusty:

This is what is actually happening. I'll do it in C but it applies in to any process/language.

1) Create a file
fdes = open("/home/myfile",O_RDWR | O_CREAT,0600);
2)unlink("/home/myfile");
3) Then do all the i/o you like.
4) Finally close(fdes).

Until the file is closed, the file can be used by this process eventhough it has been unlinked (rm'ed). The file continues to occupy space in the filesystem until all processes which have the file open, close it.

This is a standard idiom to use a temp file and in fact there a library function which does just this.

Re\gards, Clay

[harry d brown jr]

Rusty,

You will also see this when a user has opened a file for writing, then it is deleted, but because the file is still open, the file space isn't released back to the system.

If you don't already have "lsof", I'd suggest getting it:

http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.55/


live free or die
harry

[Marcin Wicinski]

Hi,
Sometimes described behaviour happends. It is caused by processes (especially zombie)holding filesystem. Then bdf cannot estimate properly the filesystem utilization. Sometimes after removing a hudge amount od data from filesystem similar bahaviour can occure (befor LVM update info).
Later,

[Wim Rombauts]

I agrre with Jeff

[Darrell Allen]

Hi Rusty,

This often happens when a log file is removed while another process has it open. First time I saw it was when /var/adm/syslog/syslog.log was deleted without stopping syslogd. The system knows the space was allocated and a process has it opened. du is simply doing math based on the inodes it finds and there is none for the deleted file.

Darrell

[Kurtkarl]

Hi,

bdf sometimes shows incorrect informations about fs usage. Try umounting /home and mount it again. Try running also fuser for there might might be some proceses keeping your /home. Also, try to checking on the permission of your filesystem.

Jose

[Joseph A Benaiah_1]

Rusty,

It looks like someone removed sone files that were open.

Basically, a file has 2 parts, the inode and the data. When an open file is removed, the inode is removed but the space is not claimed backup on the filesystem until the process that had the file open has ended or been killed.

The fuser command is useful if you are the root user but as this happenned in /home, it is possible that the offending files and processes were not owned by root.

I would advise non-root users to do the following to check if a file that they wish to remove is not being written to:

cp /dev/null filename
ll filename

If the file starts to grow, it is still being written to.

Cheers,

Joseph