1834722 Members
2573 Online
110069 Solutions
New Discussion

/ full

 
Gilles Salvin
Frequent Advisor

/ full

Hello
My root file-system / is growing and growing
more and more every day but I can't find
which files are growing-up ???
I've created a file called "NEW"
cd /
touch /NEW and 1 day later
find . -xdev -newer NEW ===> I can't find
what is growing-up so many ...
I've also performed du -kx / | sort -n ===> neither find what's growing-up
Any idea please ?
What could I say ?
15 REPLIES 15
Paula J Frazer-Campbell
Honored Contributor

Re: / full

Hi

Normally log files are to blame.

Do a bdf then target the dirs from root that are not mount points.

HTH

Paula
If you can spell SysAdmin then you is one - anon
Paula J Frazer-Campbell
Honored Contributor

Re: / full

Hi

Have a look in /dev - a common problem is doing backups to /dev/om instard of /dev/0m.

Also do a ls -lrt in /dev and look for latest files.


HTH

Paula
If you can spell SysAdmin then you is one - anon
Steven Sim Kok Leong
Honored Contributor

Re: / full

Hi,

Perform an lsof on /. Check for opened files that have already been deleted.

It could be that you have an opened file that you have removed but the process is still writing to it.

Examples of such possible files are mail.log, syslog.log etc. If you remove syslog.log without restarting syslogd, then the process is still writing to the opened file, consuming filesystem space in the process.

Hope this helps. Regards.

Steven Sim Kok Leong
S.K. Chan
Honored Contributor

Re: / full

You probably got an active process that still writes to a file that was either removed of zeroed out. If that is the case you got an "invinsible" file that will keep on growing and eating up your disk space unless you find that process and kill it or reboot the system.
Sebastian Galeski_1
Trusted Contributor

Re: / full

Hi
check /etc folder a lot of application put there logs files, maybe You have OpenView software installed??? then check /etc/opt/OV, /etc/opt/omni

seba
Gilles Salvin
Frequent Advisor

Re: / full

Hello Steven Sim Kok Leong
I think you are right, I'm not use to pratice
lsof ; are there any options I need to use ?
Thanhs
What could I say ?
Steven Sim Kok Leong
Honored Contributor

Re: / full

Hi,

You can download the HP-UX port of lsof from here:

http://gatekeep.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.61/

lsof is a must-have utility for any system administrator. You will have to rely on it to find the deleted opened file that is still being actively written into by the process.

# for file in `lsof | grep '/' | awk '{print $9}' | sort | uniq` ; do if [ ! -e $file ] ; then echo $file is a non-existing opened file ; echo "the process(es) holding onto $file is `lsof $file`" ; fi ; done

Once you have identified the process, stop (using kill if no shutdown command is available for this process) and restart it.

Hope this helps. Regards.

Steven Sim Kok Leong
Steven Sim Kok Leong
Honored Contributor

Re: / full

Hi,

If it is an important system process, then do not terminate it now and there. Schedule downtime first.

The easy way out, without using lsof, is to simply reboot your server. This will clear the opened files. Again, you need to schedule downtime.

Hope this helps. Regards.

Steven Sim Kok Leong
Gilles Salvin
Frequent Advisor

Re: / full

Hello Steven Sim Kok Leong
Thank you for your help but I don't find
anything ...
BR
What could I say ?
Gilles Salvin
Frequent Advisor

Re: / full

Hello S.K. Chan
You are probably right
any idea about how to find the faulty process ?
Thanks
What could I say ?
Bill Hassell
Honored Contributor

Re: / full

Most likely, there are directories in / that simply do not belong there. / is a static directory and won't grow without root's help. Often, a poorly written application program will usurp the sysadmin's responsibility to manage the system by storing itself in the root directory. There are only 3 directories in / anhd they are: etc dev sbin

That's all! Anything else (other than root's $HOME) are misplaced and need to be moved. As mentioned, if du can't see any growth, then likely there is a program writing to a logfile (logfiles *NEVER* go into /) and perhaps you rm'ed the file a while ago, but as long as the program keeps the file open, it will grow--and won't be visible

The only solution is a reboot. Save a copy of the sorted du output, then reboot and run it again (along with bdf). If / is back to normal (about 30-60 megs) then the above scenario was the case. It's time to cleanup / and move everything that isn't a mount point (except etc sbin and dev) to another disk. If an application (again, poorly written) can't tolerate being moved to /opt, then you can use a symlink to 'fool' the program. (you may also want to use the telephone to complain about the application's impact on your operation)

Another thing to do is to move root's $HOME. It doesn't belong in / at all. root droppings (the junk that sysadmins leave in $HOME) need to be checked regularly and if it is /, it is is difficult to see what needs to be purged. Instead, create /root, then move *all* files to that directory (no files should be in /). Finally, edit /etc/passwd and change the $HOME directory. Just to make sure everything was done correctly, stay logged in and start another session from a new location by logging in as root. If there is a problem (spelling, etc) then the current root login will let you fix it easily.


Bill Hassell, sysadmin
Pete Randall
Outstanding Contributor

Re: / full

Gilles,

One other thing to look for is an unmounted file system. For example, if /blah is a separate file system and it becomes unmounted, user Bozo may still going to be writing output into /blah - except it's now going to end up in / instead.

Good luck,
Pete

Pete
Juan Manuel López
Valued Contributor

Re: / full

I think you have to reboot the machine because, maybe, there is a open file that you have delete so it have a process still running on it.
Other way to repair you situation is to find the files open on / with lsof
Also it is interesting to look for the files that are greater than a determinated size.
So the problem is how to find this files only on / filesystem.
There is a little modification to your command that maybe can help you a little more:
du -kax | sort -n

Try this.

Juanma.
I would like to be lie on a beautiful beach spending my life doing nothing, so someboby has to make this job.
Juan Manuel López
Valued Contributor

Re: / full

Give points to all of us, please.

Thanks in advance.

Juanma.

Note. How is now your / file system ? Are you still on troubles ?

I would like to be lie on a beautiful beach spending my life doing nothing, so someboby has to make this job.
Gilles Salvin
Frequent Advisor

Re: / full

I've rebooted the server
the / is now used at only 35% ...
I kept about 100Mo with the reboot
I'll continue to check it and hope I'll
finally find what is wrong
Thank you very much everybody

BR
What could I say ?