General
cancel
Showing results for 
Search instead for 
Did you mean: 

Strange prob with /var filesystem!!!!

Sandip Ghosh
Honored Contributor

Strange prob with /var filesystem!!!!

OS - HP-UX 10.20
System - K460
/var - 262 MB (80% full)
EMS - configured
Predictive - configured

Everyday at a particular time the var file system is getting filled up by 100%. And immidiately after that it is getting cleared up automatically. I had tried to catch hold of the process, but I couldn't do it. I have checked the cron logs also, but nothing is there. Can anybody please help me to idetify the process or the file which is getting filled up?

Thanks in advance,

Sandip
Good Luck!!!
15 REPLIES
Helen French
Honored Contributor

Re: Strange prob with /var filesystem!!!!

I would think of the following possibilities:
1) lp spooler files in the queue. A large print request which waits in the queue and remove the entry when it's printed.
2) crontab file from a particular user (/var/spool/cron). Check all users cron files and check if any programs causing this problem.
3) Apply latest patches to the system.
4) log files from stm, EMS etc
Life is a promise, fulfill it!
John Poff
Honored Contributor

Re: Strange prob with /var filesystem!!!!

Hi,

Another reason to consider having separate filesystems for /var/tmp, and maybe /var/spool/lp if your print jobs are big enough.

JP
Shannon Petry
Honored Contributor

Re: Strange prob with /var filesystem!!!!

The var directory is used for lots of stuff. Namely:
system administration and
spooling

look at
/var/adm/
check the *acct files and syslog dir for build up.
/var/spool
large print jobs spooled up will usually cause these problems
/var/mail
check for junkmail sent to local server etc....


Run a cronjob every 30 minutes to snapshot the var directory and put the file into a /tmp file.
I.E.
#!/bin/sh
LOG=/tmp/varmon.txt
#create log if not there
touch $LOG
cd /var
echo "System Snapshot" >>$LOG
date >>$LOG
echo "___________________________________" >>$LOG
du -sk * >>$LOG
echo "" >>$LOG ; echo "">>$LOG

Didnt test it, but should work.

Regards,
Shannon
Microsoft. When do you want a virus today?
MANOJ SRIVASTAVA
Honored Contributor

Re: Strange prob with /var filesystem!!!!

Hi Sandip


Like /tmp is used by vi etc as a scratch pad , /var/tmp is used by applications as scratch pad specially if u ahve oracle wiht lots of users , I woul recommend that u clean /var/tmp when the system comeup after booting all the time as this is not done autoamaitcally . also a crude way to find out waht is eating the the space can be to do

cd /var

ls -lR | grep log
and check for the size every 5 seconds to know what is filling up

or you can do like this

while true
ls -lR /var | grep log >> /tmp/abc
sleep 10
done

run this for the time there is a probelm and u can monitor what log files get bulky , but I think the problems is wiht /var/tmp


Manoj Srivastava
A. Clay Stephenson
Acclaimed Contributor

Re: Strange prob with /var filesystem!!!!

One other thought: This might be a remsh'ed process croned from another server.
If it ain't broke, I can fix that.
A. Clay Stephenson
Acclaimed Contributor

Re: Strange prob with /var filesystem!!!!

Because filling up the /var filesystem can bring everything to a halt, you should really consider creating separate mountpoints for /var/spool/lp, /var/mail, and /var/tmp. Your system is much more robust like that. What is probably happening is that some process (probably cron or at) is creating a temp file. Moreover, the process may even unlink the file after opening it so that it is invisible to you but still occupies space until the last process that had it open, closes the file. When the filesystem fills up the process dies and the space is returned.
If it ain't broke, I can fix that.
Sandip Ghosh
Honored Contributor

Re: Strange prob with /var filesystem!!!!

Shiju,

1. If it is from lp-spooler it may not happen at the same particular time.
2. I have looked at the crontab log and couldn't find anything. Whatever it is there, it runs every 15 or 20 min. But the problem occurs particularly at 9:40 AM. And I have looked at all the entries for the cron users and nothing has been set particularly for 9:40 AM.
3. Latest patch has to be applied within 1-2 weeks. Looking for downtime.
4.Looked at the STM and EMS log, but couldn't find anything.

Is it possible to dump something by the ups_mond daemon? As far as I could remeber it is happening shortly after installing the new UPS. I am not sure about that.

Sandip
Good Luck!!!
Leif Halvarsson_2
Honored Contributor

Re: Strange prob with /var filesystem!!!!

Hi
Do you know exact at what time this happens. Perhaps you can get an idea if you start a script short before which monitor the /var subdirectorys where you can suspect something is going on. For example:

for dir in tmp spool cron ...
du -s $dir
sleep 5
done >logfile

Kill the script and browse the logfile. Perhaps you can see if one directory suddenly begins to grow.

Leif Halvarsson_2
Honored Contributor

Re: Strange prob with /var filesystem!!!!

Sorry,

Ow course this must be run in an infinity loop

while true
do
for dir in tmp spool cron ...
do
du -s $dir
sleep 5
done
done >logfile

Sandip Ghosh
Honored Contributor

Re: Strange prob with /var filesystem!!!!

In fact I had suspected the directory /var/tmp, because this is the only directory where the log files can be deleted automatically. On the other directories if you write any thing it stay over there but not vanishes as soon as the process completed.

Mr. Clay,
I was also thinking in the same way. Some process is creating a file on /var/tmp which is invisible to me. In fact today I had tried to catch hold of the file by giving the following command

while true
do
ll >>/tmp/ll_file
date>>/tmp/ll_file
done

After this also I could not catch hold of the file. Any idea to catch hold of the process like this?

Sandip
Good Luck!!!
Shannon Petry
Honored Contributor

Re: Strange prob with /var filesystem!!!!

You can use lsof to view open files not visible if unlinked.

Regards,
Shannon
Microsoft. When do you want a virus today?
Martin Johnson
Honored Contributor

Re: Strange prob with /var filesystem!!!!

It is probably /var/tmp filling up. I would make a file system for /var/tmp and monitor it. Probably "ll -R /var/tmp" every 30 seconds should get you the culprit (erdirect to a file not in /var/tmp). If not, use lsof.

HTH
Marty
Bill Hassell
Honored Contributor

Re: Strange prob with /var filesystem!!!!

First (and MOST important), 262 megs for /var is WAY TOO SMALL. /var should start at 750 megs and depending on what you are running, may need to be 2000 megs! The program and operational problems seen when /var fills up are far more expensive than a couple of gigabytes of disk.

To find the problem are, use du to analyze exactly which directory is growing. It could be a simple cp of a large file which is larger than the 50 megs left on the 262 meg filesystem. The process starts, fills the filesystem then aborts and the file is removed. No wonder it is hard to find.

So run du in a script like this every 5 seconds:

#!/usr/bin/sh
echo "Start /var log" > /tmp/var.log
while :
do
date >> /tmp/var.log
du -kx /var | sort -rn >> /tmp/var.log
sleep 5
done

This should catch the directory that is growing. Then change the script to report on the directory by file size and run the next day just before the event:

#!/usr/bin/sh
echo "Start /var log" > /tmp/var.log
while :
do
date >> /tmp/var.log
ll /var/whatever | sort -rnk5 >> /tmp/var.log
sleep 5
done





Bill Hassell, sysadmin
Animesh Chakraborty
Honored Contributor

Re: Strange prob with /var filesystem!!!!

Hi Sandip,
It is quite normal that the space in /var is variable.I have seen it in many times.Even while doing vi a large file sometime /var/preserve directory gets full.
Since you have only 262MB(very low)space that is why it is bothering you.Try to increase the /var file system in next downtime.
To identify the culprit.
put this line in crontab
#crontab -e
38 09 * * * /sbin/ls -ltR /var/tmp >/tmp/text1.txt
40 03 * * * /sbin/ls -ltR /var/tmp >/tmp/text2.txt

Then compare these two files.
#diff text1.txt text2.txt

Did you take a backup?
john korterman
Honored Contributor

Re: Strange prob with /var filesystem!!!!

Hi Sandip,
if all other possibilities fail, you could try a long shot: do you have measureware on the machine. If yes, could it be the automatic logfile maintenance made at "mainttime" in mwa's parmfile - normally /var/opt/perf/parm. If measureware is not installed, please ignore this posting.

regards,
John K.
it would be nice if you always got a second chance