topic Re: File system full in Operating System - HP-UX

File system full

Lora Ganeva — Wed, 12 Sep 2001 10:09:03 GMT

I have a HP-UX 10.20. There is a message in /var/adm/syslog/syslog.log, reporting that my /home filesystem is 100% full and it the output of "bdf" command shows its full, but when I look thhrough the files in thi f.s., there are few files of small size.What could be the cause?
Every help wolud be appreciated.
P.S.And we all pray for America

Re: File system full

David Almada_1 — Wed, 12 Sep 2001 10:18:02 GMT

hi lora
you may have a core file,try to use the comand
find /home -name core

david

Re: File system full

Christopher McCray_1 — Wed, 12 Sep 2001 10:20:53 GMT

First thing I would do is a du -ks * in /home to see which of the home directories is taking up the most space. Then I would look for any core files ( which can be created by users that run programs out of their home directory) and either remove them, or move them to a location where there is space available, be careful of filling other fs. send out notifications to the offending parties to clean up there files, or you can do it for them. These files can be restored if you guessed wrong from a backup (if you have done so). hope this is a good starting point for you. Another note, what types of messages were in syslog prior to the full fs messages? Anything out of the ordinary? Just a thought.

Good luck
Chris

Re: File system full

Richard Woolley — Wed, 12 Sep 2001 10:21:04 GMT

try:
cd /home
du -k .
it gives output in Kb of all the directories under home, look for something unusual, if not use a bit of maths and compare the figure at the end of the output to the bdf.

Re: File system full

Lora Ganeva — Wed, 12 Sep 2001 10:45:55 GMT

Hi,
I do not have a core file and anything unusual in /var/adm/syslog/syslog.log.The only unusual is that the total size of all files in /home directory is 130Kbytes and the file system is 12Mbytes.The output of the command:
cd /home
du -k
8 lost+found
132 /data(directory with the files)
141 .
Any ideas?

Re: File system full

linuxfan — Wed, 12 Sep 2001 10:51:10 GMT

Hi Lora,

1. First of check the sizes of the files/directories in /home

du -ks /home/* |sort -nr

(would sort the sizes in reverse order, you the highext user would be at the end of the list)

2. Look for core files
find /home -name core -exec ll {} \;

If you are certain you don't need the core files in /home, (check with the users first)
find /home -name core -exec rm {} \;

3. Sometimes even though you remove a file, if a process is writing to it the free space is not recovered.
do a
fuser -cu /home
to see which processes are writing to the filesystem.

-HTH
Ramesh

Re: File system full

Lora Ganeva — Wed, 12 Sep 2001 11:03:37 GMT

I do not have core files.
And my greatest problem is that the toatl size of my files is about 10 times than the file system size.How could be full??

Re: File system full

Christopher McCray_1 — Wed, 12 Sep 2001 11:16:33 GMT

I agree with Ramesh; do a du -ks /home/* | sort -nr to get the highest directories and also a n fuser -cu to find any processes that may be killing you. After that, go into those directories and look for logs, software, tar files, anything that could be put to tape or somewhere else. Stepping back for a moment, you said that your /home is only 12MB? Is there any possibility for increasing this? what type of server do you have?

Let us know what you find.

Regards,
Chris

Re: File system full

Marcin Wicinski — Wed, 12 Sep 2001 11:17:12 GMT

Hi,

bdf sometimes shows incorrect informations about fs usage. If its possible umount /home and mount again. Mayb some proceses keep /home - check it with fuser.

Later,

Re: File system full

Santosh Nair_1 — Wed, 12 Sep 2001 11:22:05 GMT

Lora,

Sometime even though a file has been removed via the rm command, if there are processes still holding the file open, the file will continue to take up space on the filesystem.
You would have to do a fuser -cu /home (assume that home is a separate filesystem) and then try to determine which process might be holding on to a large file. Or alternatively, just kill off all the processes listed from teh fuser command (if that's an option).

Hope this helps.

-Santosh

Re: File system full

Sanjay_6 — Wed, 12 Sep 2001 12:27:42 GMT

Hi Lora,

Can you unmount the filesystem and do an fsck on that and see if it helps.

Thanks

Re: File system full

Jim Turner — Wed, 12 Sep 2001 12:35:30 GMT

Hi Lora,

Would you please post the results of these two commands?

bdf /home

du -kxs /home

du -kx /home | sort -rn | head -20

All the best,
Jim

Re: File system full

someone_4 — Wed, 12 Sep 2001 13:30:32 GMT

Hello
from /home run
#find * -type f -size +1000000c | xargs ll | sort -n +4.0

this find command will find the biggest files in the home dir and every sub dir.

Richard

Re: File system full

linuxfan — Wed, 12 Sep 2001 13:45:05 GMT

Hi Lora,

I have seen cases where bdf reports wrong information (HP recommends installing patches) but we before install any patches, you might want to check the output of du and bdf of other filesystems)

From what you have said so far, du definitely reports (141K) much less usage than the actual filesystem size (12MB). But did you do a "fuser -cu /home" and try killing the processes "fuser -kcu /home"?
Did you try unmounting/mounting the filesystem?

-Ramesh

Re: File system full

Alan Riggs — Wed, 12 Sep 2001 15:07:32 GMT

I have seen this happen when a file is removed while a running process has an open file pointer into the file. The space claimed by that running process will not be freed (as reported by bdf), but file query commands like du and ll will not reveal the space.

fuser /home
fuser might reveal such a process. If so, killing the process will free the space. Umount /home accomplishes basically the same thing, since the umount cannot take place until all attached processes are killed.

Re: File system full

Lora Ganeva — Thu, 13 Sep 2001 05:24:27 GMT

Hi,
Thanks a lot to all You. I have a sh running in this f.s. ane writing to a log file.I could not kill the processes cause this would stop the work in my network.Any other ideas?
The log file is 106000bytes long. Such f.s. with -sh running and log files are on 5 machines in my network, but the only machine with file system full is this one.
Unmounting is impossible-the work shouldn't be stooped.

Re: File system full

Animesh Chakraborty — Thu, 13 Sep 2001 05:42:11 GMT

Hi lora,
You can try to increase the file system size online provided you have installed online JFS.

Thanks
Animesh

Re: File system full

Lora Ganeva — Thu, 13 Sep 2001 06:09:16 GMT

the problem is that this is a HFS file system

Re: File system full

Joe Doe Sr — Thu, 13 Sep 2001 07:53:48 GMT

lsof is also a nice command to see the open files. This specially because you can remove a filename but the file still can be used.

Re: File system full

Santosh Nair_1 — Thu, 13 Sep 2001 08:28:12 GMT

Lora,

Did you mean to say that there is a shell script that's writing to a log file or the acutal shell is writing to the log file?
If its the former, then I would think zeroing the log file would be relatively okay since I know of very few shell scripts that keep files open indefinitely.
In fact, more shell scripts do automic writes to log files, i.e. a quick redirect to a file, etc.

Also, in any case, 106000 bytes is not very big...its only 106k, so I don't think this is the culprit.
Can you run the following short script which will generate a list of processes running on the filesystem:

!#/bin/sh
for PROC in $(fuser -cu )
do
ps -ef|grep $PROC|grep -v grep
done

From here you can try to narrow down which process is hogging the space.

-Santosh