/var filesystem 100% full

[Carles Viaplana]

Hello,

I've an HP-UX 11i with /var filesystem full:

# bdf /var
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol8 2609152 2609152 0 100% /var
#

But when I execute du -ks * from /var I see files use less than filesystem capacity:

# du -ks /var
271088 /var
#

I think it could be due some files had been deleted but any process was using them yet.

What do you think?
Is there any way to fix it without rebooting system (this is the only solution I've found till now)?

Thanks in advance for your help.
Regards,

Carles

[MarkSyder]

You are right, some deleted files may still be in use by processes.

If you want to clear space look at the following:

/var/adm/btmp
/var/adm/sulog
/var/adm/wtmp

Do not delete these files! Safest to empty them thus:

> /var/adm/btmp

/var/adm/cron/log and OLDlog - particularly OLDlog. If you haven't rebooted in a while, you can safely get rid of this.

/var/adm/lp/log and oldlog - see above comment

/var/adm/syslog/mail.log, syslog.log, and OLDsyslog.log

Log files can be trimmed in sam - this way you keep the most recent information in case you need to refer to it.

Mark Syder (like the drink but spelt different)

[Pete Randall]

- and look at /var/tmp. Probably all of it should be delete-able.


Pete

[Peter Godron]

Carles,
you can also search for large files:
cd /var
find . | xargs ll -d | sort -nrk5 | more

bdf and du are notorious for showing different space.

[melvyn burnard]

also check /var/adm/crash
and you could think about cleaning up /var/adm/sw/save

[Pete Randall]

I'm sure Melvyn meant to say that you could clean up /var/adm/sw/save using the "cleanup" command, right Melvyn?

;^)


Pete

[A. Clay Stephenson]

Because /var is such a critical filesystem and when it fills up many critical processes, it is good practice to divide the /var filesystem into several mountpoints depending upon the what your box actually does such as /var/spool, /var/mail, /var/tmp, /var/adm, /var/opt/omni, ... .

[Carles Viaplana]

Thanks to all for your messages.

In fact I already did all you suggested and /var filesystem is still 100% full.

As I wrote in my 1st post, filesystem is 2,5 GB size and when I execute "du -ks /var" output is just 270 MB(!).

We deleted some temp files this morning and we decreases /var to less than 20%, but few minutes later we detected it increases to 100% again.

Could deletion failed due other processes were using them and space won't be available till these processes finish?

Thanks again for your help.

regards,

Carles

[MarkSyder]

Could deletion failed due other processes were using them and space won't be available till these processes finish?

Yes.

fuser will help. I'm no expert in the use of fuser, but plenty of people on here are. Or you could use the man page if you don't want to wait for an expert.

If space if filling up quickly you might like to try the following commands:

cd /var
find . -mtime -1 -print

or, in an affected directory,

ll -tr

which will show the most recent files at the bottom of the screen.

Mark

[Heironimus · Accepted solution]

Yes, most likely some file was removed while it was still in use. Sometimes this happens by accident (somebody rm's without running fuser first), but it's also a common idiom in UNIX programming for temp files that are supposed to be small and short-lived so a bad programmer can also cause it. Killing the processes that have the file open will free the space.

lsof should help you find the processes.

[Carles Viaplana]

Hi all!

Finally I used "lsof +aL1 /var" command and I found processes were locking deleted files. I killed them and all is fine now.

Thanks to all for your suggestions.

Regards,

Carles