topic File System Continually going Full in Operating System - HP-UX

File System Continually going Full

Andrew Pollard — Fri, 08 May 2009 13:58:07 GMT

Hi,

We have an RX7640 running 11.31 and the /home (lvol5)filesystem is always going full. However it only stays full for a few seconds and then it is cleaned up again. We are not able to investigate fast enough to find out what is causing the file system to fill. The syslog and dmesg provide no information, is there another log we can look into or a flag to turn on so that this type of information is captured in syslog?

Thanks

Andrew Pollard

Re: File System Continually going Full

Steven E. Protter — Fri, 08 May 2009 14:08:13 GMT

Shalom,

When its full:

du -k | sort -rn | more

The /home file system is normally where users home directory is located. So it could be a user process.

SEP

Re: File System Continually going Full

TTr — Fri, 08 May 2009 14:22:29 GMT

No other log to look at. What you need to do is find out if any of your users are running any processes that may be creating large temp files that would fill up the filesystem. In most cases when this happens and the filesystem fills up the process traps the full filesystem, cleans up and exits. See if you have any user complains about not being able to run such a process.

Another thing to look for is if any of the users have set their TMPDIR variable and point it to their home directory. If this happened anything that utilizes the TMPDIR setting (vi and elm are two common programs that do this) would save everytjhing back in /home.

Other things to try is to run ps or du listings via cron to find out what is running when the filesystem fills up but it may not be easy. At least the du commands will create a lot of unecessary i/o load.

Re: File System Continually going Full

Andrew Pollard — Fri, 08 May 2009 14:22:41 GMT

Hi SEP,

Yes we have tried that, we even created an automatic action in Openview to run the du command once the alerts appears and that template runs every 10 seconds but whatever file is causing the file system to fill is being removed just as quickly as it appears and we can never get any good information from the du.

Thanks

Andrew

Re: File System Continually going Full

Rita C Workman — Fri, 08 May 2009 14:26:34 GMT

If it's always going full...can you narrow down the time it is always going full?

If you can, then can you check for any automated (or cron'd) jobs running around that time?

Or next it time it happens, see who's logged in and/or running something? The issue may have cleared, but the "culprit" may be the person still logged in.

Just a thought,
Rita

Re: File System Continually going Full

Court Campbell — Fri, 08 May 2009 15:16:53 GMT

You may want to get a list of users logged in at that time. If you don't have many you could possibly narrow it down after a few phone calls. You might also check out any cron jobs that run. I guess you could write a loop in a script that prints du output to a file and hope that it gives you a lead.

Re: File System Continually going Full

Court Campbell — Fri, 08 May 2009 15:19:22 GMT

I will also offer this. You will need a linux box to do this. But in some cases I will use the watch command with ssh to monitor things on my hpux box. Ex.

watch -d ssh server du -sk /home/*

Re: File System Continually going Full

Victor Fridyev — Fri, 08 May 2009 16:00:54 GMT

Hi,

Try to use fuser in order to see the users which open files on the file system.
Run something like this:
if [ $(bdf /home| awk 'NR>1 {print $(NF-1)}' | sed 's/%//') -gt 90 ] ; then
fuser -cu /home >> /var/logfile
fi

If number of users is not too big, you can log out them and find the criminal

HTH

Re: File System Continually going Full

Sp4admin — Fri, 08 May 2009 17:40:26 GMT

hello,
Maybe you can use disk quotes on the users in the /home directory. This might not resolve what keeps filling it up. But it should stop it from filling up the directory completely.

sp,

Re: File System Continually going Full

Andrew Pollard — Wed, 13 May 2009 11:13:28 GMT

Thank you all for your suggestions. I think I will try the "watch" command from linux to see if I can catch the ID causing the issue.

Andrew Pollard

Re: File System Continually going Full

Bill Hassell — Thu, 14 May 2009 00:59:50 GMT

You need to find the largest directories first. That's what du -ksx/home/* is for. Run this command into a logfile every minute or so:

(date ; du -ksx /home/* | sort -rn | head) >> /var/tmp/duhome.log

There's no point in looking at all the directories, just the biggest ones. And with this logfile, you'll see which one is growing. The name of the directory that grows rapidly is the problem user. This may be a logged-in user or it could be an cron or at job.