File System Continually going Full

[Andrew Pollard]

Hi,

We have an RX7640 running 11.31 and the /home (lvol5)filesystem is always going full. However it only stays full for a few seconds and then it is cleaned up again. We are not able to investigate fast enough to find out what is causing the file system to fill. The syslog and dmesg provide no information, is there another log we can look into or a flag to turn on so that this type of information is captured in syslog?

Thanks

Andrew Pollard

[Steven E. Protter]

Shalom,

When its full:

du -k | sort -rn | more

The /home file system is normally where users home directory is located. So it could be a user process.

SEP

[TTr]

No other log to look at. What you need to do is find out if any of your users are running any processes that may be creating large temp files that would fill up the filesystem. In most cases when this happens and the filesystem fills up the process traps the full filesystem, cleans up and exits. See if you have any user complains about not being able to run such a process.

Another thing to look for is if any of the users have set their TMPDIR variable and point it to their home directory. If this happened anything that utilizes the TMPDIR setting (vi and elm are two common programs that do this) would save everytjhing back in /home.

Other things to try is to run ps or du listings via cron to find out what is running when the filesystem fills up but it may not be easy. At least the du commands will create a lot of unecessary i/o load.

[Andrew Pollard]

Hi SEP,

Yes we have tried that, we even created an automatic action in Openview to run the du command once the alerts appears and that template runs every 10 seconds but whatever file is causing the file system to fill is being removed just as quickly as it appears and we can never get any good information from the du.

Thanks

Andrew

[Rita C Workman]

If it's always going full...can you narrow down the time it is always going full?

If you can, then can you check for any automated (or cron'd) jobs running around that time?

Or next it time it happens, see who's logged in and/or running something? The issue may have cleared, but the "culprit" may be the person still logged in.

Just a thought,
Rita

[Court Campbell]

You may want to get a list of users logged in at that time. If you don't have many you could possibly narrow it down after a few phone calls. You might also check out any cron jobs that run. I guess you could write a loop in a script that prints du output to a file and hope that it gives you a lead.

[Court Campbell]

I will also offer this. You will need a linux box to do this. But in some cases I will use the watch command with ssh to monitor things on my hpux box. Ex.

watch -d ssh server du -sk /home/*

[Victor Fridyev]

Hi,

Try to use fuser in order to see the users which open files on the file system.
Run something like this:
if [ $(bdf /home| awk 'NR>1 {print $(NF-1)}' | sed 's/%//') -gt 90 ] ; then
fuser -cu /home >> /var/logfile
fi

If number of users is not too big, you can log out them and find the criminal

HTH

[Sp4admin]

hello,
Maybe you can use disk quotes on the users in the /home directory. This might not resolve what keeps filling it up. But it should stop it from filling up the directory completely.

sp,

[Andrew Pollard]

Thank you all for your suggestions. I think I will try the "watch" command from linux to see if I can catch the ID causing the issue.

Andrew Pollard