Linux Filesystem Space Utilization Won't Drop

[g28days]

1. The problem
The space utilization for /appl/xxx on xmen.hello.com spikes to over 90% almost every weekend (no special processes are running at this period). Also there are similiar instances of the same application running on the same server and other servers with no issues.
On the application level, we have already housekept everything there is to housekeep; there is nothing else to housekeep. We even tried to stop and start our processes.

You can see; there is nothing there that will cause the space utilzation to go up to 90%; We would like to check if there is anything locking the filesystem.

user@xmen:/appl/xxx >df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/vx/dskxxxx/xxx
20G 14G 6.2G 69% /appl/xxx

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vx/dskxxxx/xxx
1758860 97508 1661352 6% /appl/xxx

user@xmen:/appl/xxx >du -sk .
46867 .

user@xmen:/appl/xxx >du -ax | sort -rn | more
24451 .
8378 ./xmen_bin
8087 ./xmen_stats
7124 ./xmen_logs
7016 ./xmen_stats/stats1210
6273 ./xmen_logs/id1210
683 ./xmen_bin/hcs
636 ./xmen_bin/ks
473 ./xmen_bin/kc2
455 ./xmen_bin/ku
440 ./xmen_db
427 ./xmen_logs/id1209.gz
414 ./xmen_logs/id1208.gz
388 ./xmen_bin/loc
374 ./xmen_stats/stats1208.gz
364 ./xmen_stats/stats1207.gz
341 ./xmen_db/xmenrte.db
332 ./xmen_stats/stat1209.gz
328 ./xmen_bin/tcpr
319 ./xmen_bin/kips

FS was unmounted and then mounted again. Space utilization immediately dropped to 1% from 90%


After processes have been stopped; Did a “lsof +D /appl/xxx” as root; no processes. (Did it as both root and user id)

Did a “lsof +aL1 /appl/xxx” nothing is locking this FS. (Did it as both root and user id)

uname -a
Linux xmen 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64
x86_64 GNU/Linux

more /etc/redhat-release
Red Hat Enterprise Linux Server release 5.4 (Tikanga)

Please help.

Thanks and regards.,

[Jeff Hodge]

When the problem occurs run:

lsof | grep deleted

Neither of the lsof commands you ran will list deleted files that are being held open for some reason.

[Alzhy]

Hmm.. a VxVM based RHEL system.. nice.

Well, a 20GB filesystem can easily fill up in a blink of an eye. What you can do is perhaps do a "du -ax" on that FS say every 5 minutes and for sure you'll catch the culprit growing file. Once id'd, yo can employ fuser on said file to get the PID/name of the process.

My bet would be any one of your log or stats subdir is filling up with an overly verbose process being the culprit. You may want to employ intelligent log watchers and rotaters -- noting that these ASCII / text log files compress up to 97%..

[g28days]

Hi Jeff,
Thanks for your reply. I got a list of deleted processes; upon entering that command; and managed to kill it. But the space still stays the same.

Hi Alzhy,
Thanks for your reply. I will try your suggestion. The thing is we simiiar processes with similiar diskspace on other servers as well on this server; but the utilization never goes pass 2%. The application/process which runs will not accumulate that much space at all.

Thank you again.

[Jeff Hodge]

Between Alzhy's suggestion and the lsof commands you should be able to find what is taking up all the space. The lsof command should show you the size of the file that is open regardless of whether or not it is deleted. Check that size prior to killing the process to see if it matches up with the space that you expect to reclaim.

You can see which column will represent the size by doing a quick "lsof | head".

I agree with Alzhy. The most likely suspect in this type of situation is a logrotate. Depending on the log rotate config, a log file can be completely copied prior to being compressed. You might check the configurations of you log rotate scripts (/etc/logrotate.d) and the time frames during which they will run (/etc/cron*).

Another less fruitful approach but "good-to-know" would be a search for "D" state processes during the issue. Those are normally a result of storage locations that become unavailable and whose state is referred to as "uninterruptible sleep".

Good luck.

[Lucifer Megacruel]

Hi g28days,

Linux might show the size incorrectly if the file is sparse with lot of zero's. You can do a hexdump of the file see its contents. As far as i know there is nothing you can do to fix it in a straight forward way :(

Truly Evil
Lucifer Megacruel