topic Re: / full in Operating System - HP-UX

/ full

Gilles Salvin — Fri, 14 Jun 2002 07:37:20 GMT

Hello
My root file-system / is growing and growing
more and more every day but I can't find
which files are growing-up ???
I've created a file called "NEW"
cd /
touch /NEW and 1 day later
find . -xdev -newer NEW ===> I can't find
what is growing-up so many ...
I've also performed du -kx / | sort -n ===> neither find what's growing-up
Any idea please ?

Re: / full

Paula J Frazer-Campbell — Fri, 14 Jun 2002 07:48:50 GMT

Hi

Normally log files are to blame.

Do a bdf then target the dirs from root that are not mount points.

HTH

Paula

Re: / full

Paula J Frazer-Campbell — Fri, 14 Jun 2002 07:53:16 GMT

Hi

Have a look in /dev - a common problem is doing backups to /dev/om instard of /dev/0m.

Also do a ls -lrt in /dev and look for latest files.

HTH

Paula

Re: / full

Steven Sim Kok Leong — Fri, 14 Jun 2002 07:54:22 GMT

Hi,

Perform an lsof on /. Check for opened files that have already been deleted.

It could be that you have an opened file that you have removed but the process is still writing to it.

Examples of such possible files are mail.log, syslog.log etc. If you remove syslog.log without restarting syslogd, then the process is still writing to the opened file, consuming filesystem space in the process.

Hope this helps. Regards.

Steven Sim Kok Leong

Re: / full

S.K. Chan — Fri, 14 Jun 2002 07:56:32 GMT

You probably got an active process that still writes to a file that was either removed of zeroed out. If that is the case you got an "invinsible" file that will keep on growing and eating up your disk space unless you find that process and kill it or reboot the system.

Re: / full

Sebastian Galeski_1 — Fri, 14 Jun 2002 07:56:52 GMT

Hi
check /etc folder a lot of application put there logs files, maybe You have OpenView software installed??? then check /etc/opt/OV, /etc/opt/omni

seba

Re: / full

Gilles Salvin — Fri, 14 Jun 2002 08:32:45 GMT

Hello Steven Sim Kok Leong
I think you are right, I'm not use to pratice
lsof ; are there any options I need to use ?
Thanhs

Re: / full

Steven Sim Kok Leong — Fri, 14 Jun 2002 08:47:08 GMT

Hi,

You can download the HP-UX port of lsof from here:

http://gatekeep.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.61/

lsof is a must-have utility for any system administrator. You will have to rely on it to find the deleted opened file that is still being actively written into by the process.

# for file in `lsof | grep '/' | awk '{print $9}' | sort | uniq` ; do if [ ! -e $file ] ; then echo $file is a non-existing opened file ; echo "the process(es) holding onto $file is `lsof $file`" ; fi ; done

Once you have identified the process, stop (using kill if no shutdown command is available for this process) and restart it.

Hope this helps. Regards.

Steven Sim Kok Leong

Re: / full

Steven Sim Kok Leong — Fri, 14 Jun 2002 08:50:09 GMT

Hi,

If it is an important system process, then do not terminate it now and there. Schedule downtime first.

The easy way out, without using lsof, is to simply reboot your server. This will clear the opened files. Again, you need to schedule downtime.

Hope this helps. Regards.

Steven Sim Kok Leong

Re: / full

Gilles Salvin — Fri, 14 Jun 2002 09:58:07 GMT

Hello Steven Sim Kok Leong
Thank you for your help but I don't find
anything ...
BR

Re: / full

Gilles Salvin — Fri, 14 Jun 2002 10:02:23 GMT

Hello S.K. Chan
You are probably right
any idea about how to find the faulty process ?
Thanks

Re: / full

Bill Hassell — Fri, 14 Jun 2002 10:13:16 GMT

Most likely, there are directories in / that simply do not belong there. / is a static directory and won't grow without root's help. Often, a poorly written application program will usurp the sysadmin's responsibility to manage the system by storing itself in the root directory. There are only 3 directories in / anhd they are: etc dev sbin

That's all! Anything else (other than root's $HOME) are misplaced and need to be moved. As mentioned, if du can't see any growth, then likely there is a program writing to a logfile (logfiles *NEVER* go into /) and perhaps you rm'ed the file a while ago, but as long as the program keeps the file open, it will grow--and won't be visible

The only solution is a reboot. Save a copy of the sorted du output, then reboot and run it again (along with bdf). If / is back to normal (about 30-60 megs) then the above scenario was the case. It's time to cleanup / and move everything that isn't a mount point (except etc sbin and dev) to another disk. If an application (again, poorly written) can't tolerate being moved to /opt, then you can use a symlink to 'fool' the program. (you may also want to use the telephone to complain about the application's impact on your operation)

Another thing to do is to move root's $HOME. It doesn't belong in / at all. root droppings (the junk that sysadmins leave in $HOME) need to be checked regularly and if it is /, it is is difficult to see what needs to be purged. Instead, create /root, then move *all* files to that directory (no files should be in /). Finally, edit /etc/passwd and change the $HOME directory. Just to make sure everything was done correctly, stay logged in and start another session from a new location by logging in as root. If there is a problem (spelling, etc) then the current root login will let you fix it easily.

Re: / full

Pete Randall — Fri, 14 Jun 2002 10:15:49 GMT

Gilles,

One other thing to look for is an unmounted file system. For example, if /blah is a separate file system and it becomes unmounted, user Bozo may still going to be writing output into /blah - except it's now going to end up in / instead.

Good luck,
Pete

Re: / full

Juan Manuel López — Fri, 14 Jun 2002 10:23:57 GMT

I think you have to reboot the machine because, maybe, there is a open file that you have delete so it have a process still running on it.
Other way to repair you situation is to find the files open on / with lsof
Also it is interesting to look for the files that are greater than a determinated size.
So the problem is how to find this files only on / filesystem.
There is a little modification to your command that maybe can help you a little more:
du -kax | sort -n

Try this.

Juanma.

Re: / full

Juan Manuel López — Fri, 14 Jun 2002 11:20:04 GMT

Give points to all of us, please.

Thanks in advance.

Juanma.

Note. How is now your / file system ? Are you still on troubles ?

Re: / full

Gilles Salvin — Fri, 14 Jun 2002 11:40:37 GMT

I've rebooted the server
the / is now used at only 35% ...
I kept about 100Mo with the reboot
I'll continue to check it and hope I'll
finally find what is wrong
Thank you very much everybody

BR