/var 100%

[bhavin asokan]

hi all,
i have a problem with L class server running hp-ux 11.00. the server is having sybase as database. when the /var filesystem (1GB) became full i checked for large files and found /var/spool/cron/tmp/croutQAAa01174 with 420MB .i tried to pass null to make this file zero.then i deleted this file and created a new file with same owner & permissions (then /var came to 60%) .on that day it was working fine. on the next day /var again came to 100% .while checking it was not showing any bigger files .after rebooting it again came to 60%.what is the reason for this?

[Geoff Wild]

You say you tried to "pass null" to the file - did that work?

cat /dev/null > /var/spool/cron/tmp/croutQAAa01174

Sounds like maybe there was a lock or open process trying to write to that file - doing what you did (deleting) may have left the system state a little messed up....and the reboot fixed it....I would monitor this for a few days...

Rgds...Geoff

[Mark Grant]

My guess is that on your second day, a file was created by a process that filled up space. Either that process or another deleted the file but the original process still had the file open. It will still take up the space but you won't see it until the original process closes the file. Re-booting tends to get files closed.

A common procedure for creating tempory files is to open a file and then immediately delete it. This a) makes it hard for another process to create a similarly named file and b) means if your program dies for some reason, the tempory file will not be left on disk.

[Pete Randall]

First, the file you're seeing is a log file from a cron job. There's no reason to retain this information. You might want to schedule a cron job to clean these up periodically with the find command: "find /var/spool/cron/tmp -mtime +2 -exec rm {} \;".

Secondly, you had an open file that was filling /var. The file may have been removed, but the process had not completed so the space was not yet reclaimed.

You're going to need to do some more investigating if this continues to happen. Perhaps a tool like lsof could be of use to find the process that is filling /var.


Pete

[Jeff Schussele · Accepted solution]

Hi,

You may have to kill the cron daemon for it to release that log file. The cron job may be hung, but the cron daemon will be the parent so killing it *should* release the file lock.

My 2 cents,
Jeff

[Bill Hassell]

/var is a critical filesystem and requires constant monitoring. Looking for big files is a common mistake since you'll miss directories that have hundreds of small files. Since every installation is different, you'll need to see where the biggest directories are located:

du -kx /var | sort -rn | more

/var/spool will be particularly difficult because bad cron jobs (as you have already seen) can create massive amounts of data and these jobs must be fixed as soon as possible. The tmp/crout.... file was created by some cron job so look at the beginning of the file (use more or page, not cat) to see what it is doing and what cron job is causing this. Then stop the job (because it will just continue filling /var) and fix the code.

/var/spool also has printer jobs which may grow rapidly. /var/adm has most of the logfiles in the system and /var/adm/sw has installation software and patch recovery files. /var/mail can possibly be a problem. You'll need to monitor /var regularly and search for the big directories and then big files inside the directory. As mentioned, filespace is not returned even when the file is deleted until the process that has the file open closes the file or is terminated.

[bhavin asokan]

passing null string was not working.

earlier also the /var file system became full and after restarting it came to 60%.(the file was not deleted on that time).

the cron is containing 1)system activity report by sadc 2)adding & removing routes

the croutQAAa01174 was showing zero only when /var become full on second day.now also
it is showing not contents.

if the lock was existing with the specific file and the file was open how it became 60% after deleting the file.

if a lock is existing is there any way to see it and the respective process.

[Bill Hassell]

The cron jobs should be changed as soon as possible to send their output to a specific file. You can run fuser to see if it can tell what process is keeping the file open. Normally, cron jobs run for a short time (seconds, minutes). If the cron job that collects data runs for hours, it is written incorrectly.