cancel
Showing results for 
Search instead for 
Did you mean: 

Filesytem Filling Up

Peggy Pyburn
Occasional Contributor

Filesytem Filling Up

This morning the /var filesystem was 100% full. I deleted out all temporary files and trimmed log files. Deleted anything I didn't need anymore and got the file size down to 74% full. However, it continues to grow. It adds a percent about every 15 minutes and is now up to 89%. There is nothing written to the syslog file since I trimmed it. Does anyone have any ideas as to what is going on!!!?
11 REPLIES
James R. Ferguson
Acclaimed Contributor

Re: Filesytem Filling Up

Hi Peggy:

Are you doing a large sort perhaps? (check with 'ps -ef|grep sort').

By default, sort uses /var/tmp as a workspace unless the environmental variable TMPDIR is set or the sort is specified with the '-T' option.

...JRF...
A. Clay Stephenson
Acclaimed Contributor

Re: Filesytem Filling Up

I think we need a little more data. Please do something like this so that we can zero in on the culprit:
cd /var
du > /tmp/list1
wait a few minutes
du > /tmp/list2
diff /tmp/list1 /tmp/list2

We should then be able to at least see what's changing and then perhaps do an lsof on some of the files/directories in question.
If it ain't broke, I can fix that.
Patrick Wallek
Honored Contributor

Re: Filesytem Filling Up

How big is /var? Is there anything wrong with the system that might be writing a LOT of messages into your /var/adm/syslog/syslog.log file? Anything else odd going on with the system?

You might try doing an 'fuser -cu /var' and look at each process that is returned and see if it is a normal system process or something out of the ordinary.
Shannon Petry
Honored Contributor

Re: Filesytem Filling Up

I see lots of people refer to the percentage of disk. This is a poor method of description as 1% of 1GB is much different that 1% of 100K!!!!

Remember that in /var, lots of data is constantly succeptable to change!

You said you looked at the syslog, but how about wtmp, btmp, /var/spool/mail, /var/spool/lp, etc....

Everything under /var is pretty well defined by name what it is for, so look at the changes to var to deternime!
I.E.
> cd /var
>du -sk *
>sleep 120
>du -sk *

if /var/adm is larger, then this is either syslog or accounting files
if /var/spool is growing, see what is growing with the spool by doing just what is above
>cd /var/spool
>du -sk *
>sleep 120
>du -sk *
is /var/spool/lp growing? is it /var/spool/cron?

If it is /var/dt, then something is wrong with the X Server, so look at /var/dt/Xerrors!

I think you get the idea anyway.

Just remember that many files if they are plain old deleted can have adverse effects on the OS, so look at man pages to see how to reset!

I.E. It is not good to remove wtmp! simply running accounting will reset those files properly! However, it creates other files in the process which can be deleted after accounting is done!

Regards,
Shannon
Microsoft. When do you want a virus today?
Volker Borowski
Honored Contributor

Re: Filesytem Filling Up

Hi,
there was a recent thread on this with an informix DB. Are you runnning Informix ?
You should search the forum for it.
Volker
John Bolene
Honored Contributor

Re: Filesytem Filling Up

So syslog is not the problem, check /var/tmp for a bunch of small files, temp files either go in /tmp or here.

Check /var/spool for print jobs.

Check /var/spool/cron for cron output and log files.

Check /var/crash for dumps to delete.

do the following
cd /var
du -x|sort -rn|more

check what files are at the top of the list
It is always a good day when you are launching rockets! http://tripolioklahoma.org, Mostly Missiles http://mostlymissiles.com
Darrell Allen
Honored Contributor

Re: Filesytem Filling Up

If one of the files was still open by a process when you deleted or trimmed it then that process may still be writing to the "file" even though you don't see it any longer with ls. An example could be by erroneously trimming syslog by removing the old and creating a new syslog while syslogd was still running.

You might compare bdf /var and du -sk /var output to see if there's much difference in size. If so and since you aren't getting new messages to syslog...
"What, Me Worry?" - Alfred E. Neuman (Mad Magazine)
Vincenzo Restuccia
Honored Contributor

Re: Filesytem Filling Up

find /var -size +xxx
Peggy Pyburn
Occasional Contributor

Re: Filesytem Filling Up

Hi All,
Thank you everyone for responding. You've all been very helpful in one way or another. I discovered that the source of the problem was cron. There were tmp files in /var/spool/cron/tmp that I found by using the command du -kx /var | sort -nr | grep ^....[0-9]. What I'm trying to figure out now is why did that happen. There were no cron jobs scheduled for that day. There wasn't anything written to the syslog.log file or in mail. Any suggestions.
Thanks,
Peggy
Bill Thorsteinson
Honored Contributor

Re: Filesytem Filling Up

I have had problems with programs spinning on an error
condtion. If each try produces an error message then the device with the log file rapidly fills up.

Run tail and head on the file if you still have it. It could have been running for a day or two before you ran into space problems.

The cron log might be for an 'at' job. I can't check for
HP-UX right now.

Some programs will fill up the log device, then stay in
an output wait state until space is available. Once you
make space they resume their output.
Tim D Fulford
Honored Contributor

Re: Filesytem Filling Up

I would use
ksh
for i in $( fuser -c /var 2>/dev/null)
do
ps -fp $i | grep $i
done

This will list all active proccesses & PID's on /var. You can then investigate where they are writing.

I usually start with a large list & it soon trimms down to only a few "unsure of" process.

Tim
-