File system full

[Lora Ganeva]

I have a HP-UX 10.20. There is a message in /var/adm/syslog/syslog.log, reporting that my /home filesystem is 100% full and it the output of "bdf" command shows its full, but when I look thhrough the files in thi f.s., there are few files of small size.What could be the cause?
Every help wolud be appreciated.
P.S.And we all pray for America

[David Almada_1]

hi lora
you may have a core file,try to use the comand
find /home -name core

david

[Christopher McCray_1]

First thing I would do is a du -ks * in /home to see which of the home directories is taking up the most space. Then I would look for any core files ( which can be created by users that run programs out of their home directory) and either remove them, or move them to a location where there is space available, be careful of filling other fs. send out notifications to the offending parties to clean up there files, or you can do it for them. These files can be restored if you guessed wrong from a backup (if you have done so). hope this is a good starting point for you. Another note, what types of messages were in syslog prior to the full fs messages? Anything out of the ordinary? Just a thought.

Good luck
Chris

[Richard Woolley]

try:
cd /home
du -k .
it gives output in Kb of all the directories under home, look for something unusual, if not use a bit of maths and compare the figure at the end of the output to the bdf.

[Lora Ganeva]

Hi,
I do not have a core file and anything unusual in /var/adm/syslog/syslog.log.The only unusual is that the total size of all files in /home directory is 130Kbytes and the file system is 12Mbytes.The output of the command:
cd /home
du -k
8 lost+found
132 /data(directory with the files)
141 .
Any ideas?

[linuxfan]

Hi Lora,

1. First of check the sizes of the files/directories in /home

du -ks /home/* |sort -nr

(would sort the sizes in reverse order, you the highext user would be at the end of the list)

2. Look for core files
find /home -name core -exec ll {} \;


If you are certain you don't need the core files in /home, (check with the users first)
find /home -name core -exec rm {} \;

3. Sometimes even though you remove a file, if a process is writing to it the free space is not recovered.
do a
fuser -cu /home
to see which processes are writing to the filesystem.

-HTH
Ramesh

[Lora Ganeva]

I do not have core files.
And my greatest problem is that the toatl size of my files is about 10 times than the file system size.How could be full??

[Christopher McCray_1]

I agree with Ramesh; do a du -ks /home/* | sort -nr to get the highest directories and also a n fuser -cu to find any processes that may be killing you. After that, go into those directories and look for logs, software, tar files, anything that could be put to tape or somewhere else. Stepping back for a moment, you said that your /home is only 12MB? Is there any possibility for increasing this? what type of server do you have?

Let us know what you find.

Regards,
Chris

[Marcin Wicinski]

Hi,

bdf sometimes shows incorrect informations about fs usage. If its possible umount /home and mount again. Mayb some proceses keep /home - check it with fuser.

Later,

[Santosh Nair_1]

Lora,

Sometime even though a file has been removed via the rm command, if there are processes still holding the file open, the file will continue to take up space on the filesystem.
You would have to do a fuser -cu /home (assume that home is a separate filesystem) and then try to determine which process might be holding on to a large file. Or alternatively, just kill off all the processes listed from teh fuser command (if that's an option).

Hope this helps.

-Santosh

[Sanjay_6 · Accepted solution]

Hi Lora,

Can you unmount the filesystem and do an fsck on that and see if it helps.

Thanks

[Jim Turner]

Hi Lora,

Would you please post the results of these two commands?

bdf /home

du -kxs /home

du -kx /home | sort -rn | head -20

All the best,
Jim

[someone_4]

Hello
from /home run
#find * -type f -size +1000000c | xargs ll | sort -n +4.0

this find command will find the biggest files in the home dir and every sub dir.

Richard

[linuxfan]

Hi Lora,

I have seen cases where bdf reports wrong information (HP recommends installing patches) but we before install any patches, you might want to check the output of du and bdf of other filesystems)

From what you have said so far, du definitely reports (141K) much less usage than the actual filesystem size (12MB). But did you do a "fuser -cu /home" and try killing the processes "fuser -kcu /home"?
Did you try unmounting/mounting the filesystem?

-Ramesh

[Alan Riggs]

I have seen this happen when a file is removed while a running process has an open file pointer into the file. The space claimed by that running process will not be freed (as reported by bdf), but file query commands like du and ll will not reveal the space.

fuser /home
fuser might reveal such a process. If so, killing the process will free the space. Umount /home accomplishes basically the same thing, since the umount cannot take place until all attached processes are killed.

[Lora Ganeva]

Hi,
Thanks a lot to all You. I have a sh running in this f.s. ane writing to a log file.I could not kill the processes cause this would stop the work in my network.Any other ideas?
The log file is 106000bytes long. Such f.s. with -sh running and log files are on 5 machines in my network, but the only machine with file system full is this one.
Unmounting is impossible-the work shouldn't be stooped.

[Animesh Chakraborty]

Hi lora,
You can try to increase the file system size online provided you have installed online JFS.


Thanks
Animesh

[Lora Ganeva]

the problem is that this is a HFS file system

[Joe Doe Sr]

lsof is also a nice command to see the open files. This specially because you can remove a filename but the file still can be used.

[Santosh Nair_1]

Lora,

Did you mean to say that there is a shell script that's writing to a log file or the acutal shell is writing to the log file?
If its the former, then I would think zeroing the log file would be relatively okay since I know of very few shell scripts that keep files open indefinitely.
In fact, more shell scripts do automic writes to log files, i.e. a quick redirect to a file, etc.

Also, in any case, 106000 bytes is not very big...its only 106k, so I don't think this is the culprit.
Can you run the following short script which will generate a list of processes running on the filesystem:

!#/bin/sh
for PROC in $(fuser -cu )
do
ps -ef|grep $PROC|grep -v grep
done


From here you can try to narrow down which process is hogging the space.

-Santosh

[Lora Ganeva]

thank You very much for the script..
But it didn'y show anything unusual-just the shell script, writing to my log file

[Lora Ganeva]

I have to tell You something else.
One of the functions of the shell script is to check whether interface is up or down-if it is down the script crteates an empty file in the same directory with the command "touch" and when the interface of the machine is up again, it removes this empty file with "rm " command.
Maybe this has something to do with my problem

[Lora Ganeva]

Hi again,
I would like to tell You that I talked to my boss and he allowed me to stop the user defined shell script. When I did this, the file system usage falled down to 1%.After that I started the script again and the usage remained low.I don't have explanation..but help You very much for Your help.
Lora Ganeva