root file system fills up, reboot goes back to 7 %

[James Henderson_1]

Hello,
Please help be figure out why my root fs fills up preriodically. I can't find any obvious reasons and when the system is restarted bdf shows / at 7%.

[A. Clay Stephenson]

First do a bdf. Are all the filesystems mounted? It might be that a filesystem, e.g. /tmp is not mounted and thus everything is actually going to /. The most common cause of this is that temporary files are being written and when the process dies the space is reclaimed. In a well configured system, / should be all be static in size because essentially nothing should write to it.

[John Payne_2]

You could have some sort of large open file that gets deleted, but the space is not freed. (A log file of some sort or something.) The reboot ends the process that is holding this file open, which frees the space. For example, you could have something filling your /etc/rc.log, making it huge. Then the file gets deleted, and the space does not get freed, because the system is up. After reboot, there is no /etc/rc.log to move to /etc/rc.log.old, so the system continues. (Or there is no space to copy, so the system throws the old one away. It doesn't necessary need to be something manually deleted...) This would cause your / filesystem to fill, and the space would be reclaimed at reboot.


I have actually seen this happen to a system having a problem. (Where /etc/rc.log fills /.) You may want to use the command 'du' to look and see what is getting to be very large on your system as / starts filling.

[Jeff Schussele]

Hi James,

It could be any number of things.
You need to look at the system - specifically the / filesystem before it's full to determine just what's filling it.
Use this command:

du -akx | sort -nr | more

and it will list in descending size order the files/dirs in / & this will help you/us determine just what's eating up the space.
Clay's correct - nothing *should* be writing to / to the point that it would fill it up.

Rgds,
Jeff

[Martin Johnson]

When / starts filling up check /dev for any large files. Sometimes processes write to a file instead of a device (like /dev/rmt/om instead of /dev/rmt/0m).

HTH
Marty

[Allan Pincus]

Jim,

I am really surprised that after reboot the bdf is back at 7%.

I had a problem where the rpc.lockd.log file in /var/adm was filling up. There was a patch to fix the problem.

Don't know if that helps, but it may be a place to look.

- Allan

[Rich Wright]

Do a "ll /dev/null" and be sure that it looks like ...
crw-rw-rw- 1 bin bin 3 0x000002 Sep 12 15:20 /dev/null

If it does not, then recreate it using mknod command.

Rich