Filesytem Filling Up

[Peggy Pyburn]

This morning the /var filesystem was 100% full. I deleted out all temporary files and trimmed log files. Deleted anything I didn't need anymore and got the file size down to 74% full. However, it continues to grow. It adds a percent about every 15 minutes and is now up to 89%. There is nothing written to the syslog file since I trimmed it. Does anyone have any ideas as to what is going on!!!?

[James R. Ferguson]

Hi Peggy:

Are you doing a large sort perhaps? (check with 'ps -ef|grep sort').

By default, sort uses /var/tmp as a workspace unless the environmental variable TMPDIR is set or the sort is specified with the '-T' option.

...JRF...

[A. Clay Stephenson]

I think we need a little more data. Please do something like this so that we can zero in on the culprit:
cd /var
du > /tmp/list1
wait a few minutes
du > /tmp/list2
diff /tmp/list1 /tmp/list2

We should then be able to at least see what's changing and then perhaps do an lsof on some of the files/directories in question.

[Patrick Wallek]

How big is /var? Is there anything wrong with the system that might be writing a LOT of messages into your /var/adm/syslog/syslog.log file? Anything else odd going on with the system?

You might try doing an 'fuser -cu /var' and look at each process that is returned and see if it is a normal system process or something out of the ordinary.

[Shannon Petry]

I see lots of people refer to the percentage of disk. This is a poor method of description as 1% of 1GB is much different that 1% of 100K!!!!

Remember that in /var, lots of data is constantly succeptable to change!

You said you looked at the syslog, but how about wtmp, btmp, /var/spool/mail, /var/spool/lp, etc....

Everything under /var is pretty well defined by name what it is for, so look at the changes to var to deternime!
I.E.
> cd /var
>du -sk *
>sleep 120
>du -sk *

if /var/adm is larger, then this is either syslog or accounting files
if /var/spool is growing, see what is growing with the spool by doing just what is above
>cd /var/spool
>du -sk *
>sleep 120
>du -sk *
is /var/spool/lp growing? is it /var/spool/cron?

If it is /var/dt, then something is wrong with the X Server, so look at /var/dt/Xerrors!

I think you get the idea anyway.

Just remember that many files if they are plain old deleted can have adverse effects on the OS, so look at man pages to see how to reset!

I.E. It is not good to remove wtmp! simply running accounting will reset those files properly! However, it creates other files in the process which can be deleted after accounting is done!

Regards,
Shannon

[Volker Borowski]

Hi,
there was a recent thread on this with an informix DB. Are you runnning Informix ?
You should search the forum for it.
Volker

[John Bolene]

So syslog is not the problem, check /var/tmp for a bunch of small files, temp files either go in /tmp or here.

Check /var/spool for print jobs.

Check /var/spool/cron for cron output and log files.

Check /var/crash for dumps to delete.

do the following
cd /var
du -x|sort -rn|more

check what files are at the top of the list

[Darrell Allen]

If one of the files was still open by a process when you deleted or trimmed it then that process may still be writing to the "file" even though you don't see it any longer with ls. An example could be by erroneously trimming syslog by removing the old and creating a new syslog while syslogd was still running.

You might compare bdf /var and du -sk /var output to see if there's much difference in size. If so and since you aren't getting new messages to syslog...

[Vincenzo Restuccia]

find /var -size +xxx

[Peggy Pyburn]

Hi All,
Thank you everyone for responding. You've all been very helpful in one way or another. I discovered that the source of the problem was cron. There were tmp files in /var/spool/cron/tmp that I found by using the command du -kx /var | sort -nr | grep ^....[0-9]. What I'm trying to figure out now is why did that happen. There were no cron jobs scheduled for that day. There wasn't anything written to the syslog.log file or in mail. Any suggestions.
Thanks,
Peggy