System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

File System Continually going Full

Andrew Pollard
Super Advisor

File System Continually going Full

Hi,

We have an RX7640 running 11.31 and the /home (lvol5)filesystem is always going full. However it only stays full for a few seconds and then it is cleaned up again. We are not able to investigate fast enough to find out what is causing the file system to fill. The syslog and dmesg provide no information, is there another log we can look into or a flag to turn on so that this type of information is captured in syslog?

Thanks

Andrew Pollard
10 REPLIES
Steven E. Protter
Exalted Contributor

Re: File System Continually going Full

Shalom,

When its full:

du -k | sort -rn | more

The /home file system is normally where users home directory is located. So it could be a user process.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
TTr
Honored Contributor

Re: File System Continually going Full

No other log to look at. What you need to do is find out if any of your users are running any processes that may be creating large temp files that would fill up the filesystem. In most cases when this happens and the filesystem fills up the process traps the full filesystem, cleans up and exits. See if you have any user complains about not being able to run such a process.

Another thing to look for is if any of the users have set their TMPDIR variable and point it to their home directory. If this happened anything that utilizes the TMPDIR setting (vi and elm are two common programs that do this) would save everytjhing back in /home.

Other things to try is to run ps or du listings via cron to find out what is running when the filesystem fills up but it may not be easy. At least the du commands will create a lot of unecessary i/o load.
Andrew Pollard
Super Advisor

Re: File System Continually going Full

Hi SEP,

Yes we have tried that, we even created an automatic action in Openview to run the du command once the alerts appears and that template runs every 10 seconds but whatever file is causing the file system to fill is being removed just as quickly as it appears and we can never get any good information from the du.

Thanks

Andrew
Rita C Workman
Honored Contributor

Re: File System Continually going Full

If it's always going full...can you narrow down the time it is always going full?

If you can, then can you check for any automated (or cron'd) jobs running around that time?

Or next it time it happens, see who's logged in and/or running something? The issue may have cleared, but the "culprit" may be the person still logged in.

Just a thought,
Rita
Court Campbell
Honored Contributor

Re: File System Continually going Full

You may want to get a list of users logged in at that time. If you don't have many you could possibly narrow it down after a few phone calls. You might also check out any cron jobs that run. I guess you could write a loop in a script that prints du output to a file and hope that it gives you a lead.
"The difference between me and you? I will read the man page." and "Respect the hat." and "You could just do a search on ITRC, you don't need to start a thread on a topic that's been answered 100 times already." Oh, and "What. no points???"
Court Campbell
Honored Contributor

Re: File System Continually going Full

I will also offer this. You will need a linux box to do this. But in some cases I will use the watch command with ssh to monitor things on my hpux box. Ex.

watch -d ssh server du -sk /home/*
"The difference between me and you? I will read the man page." and "Respect the hat." and "You could just do a search on ITRC, you don't need to start a thread on a topic that's been answered 100 times already." Oh, and "What. no points???"
Victor Fridyev
Honored Contributor

Re: File System Continually going Full

Hi,

Try to use fuser in order to see the users which open files on the file system.
Run something like this:
if [ $(bdf /home| awk 'NR>1 {print $(NF-1)}' | sed 's/%//') -gt 90 ] ; then
fuser -cu /home >> /var/logfile
fi

If number of users is not too big, you can log out them and find the criminal

HTH
Entities are not to be multiplied beyond necessity - RTFM
Sp4admin
Trusted Contributor

Re: File System Continually going Full

hello,
Maybe you can use disk quotes on the users in the /home directory. This might not resolve what keeps filling it up. But it should stop it from filling up the directory completely.

sp,
Andrew Pollard
Super Advisor

Re: File System Continually going Full

Thank you all for your suggestions. I think I will try the "watch" command from linux to see if I can catch the ID causing the issue.

Andrew Pollard
Bill Hassell
Honored Contributor

Re: File System Continually going Full

You need to find the largest directories first. That's what du -ksx/home/* is for. Run this command into a logfile every minute or so:

(date ; du -ksx /home/* | sort -rn | head) >> /var/tmp/duhome.log

There's no point in looking at all the directories, just the biggest ones. And with this logfile, you'll see which one is growing. The name of the directory that grows rapidly is the problem user. This may be a logged-in user or it could be an cron or at job.


Bill Hassell, sysadmin