FS /var is full but is not FULL

[Riccardo BIgoni]

Hi,
I have a problem, my /var is 100% full, but:
/var# du -ks .
592868 .
/var# bdf .
../lvol7 2097152 2097152 0 100% /var
using lsof utility i found this process
tnslsnr 4075 ora817 2w REG 64,0x7 1593196544 8933 /var (/dev/vg00/lvol7)
after this operetation the situation returned regular. The problem is, this situation has succeeded various times with always various processes (oracle, application etc). What can be?

[Pete Ledger]

Are you deleting files which are in use by an application. If a process has a file open and the file is deleted the space used by the file is not reclaimed until the process exits.

Try checking that files are not in used before removing them and if you can't stop the process first empty the file.

Did the space get freed after you stopped the tnslsnr ? Is that the operation you are talking about ?

[Ravi_8]

Hi,

lsof shows the opened ports but occupied by process which can be removed (killed)

in your case tnslsnr (TNS listerner) with pid 4075 is using the port 8933 which is open.

o/p of lsof no where connected to your file system being full

[Riccardo BIgoni]

Not, anch' I thought this thing, but e' already succeeded some times on the same machine. I "think" is not a human (or crontab script) error.
Thanks,

[Pete Randall]

This is a relatively common occurence. The space occupied by a removed file is not freed while a process still has the file open until the process completes. If this is happening regularly, as you say, then you either need to do some serious housekeeping in /var to free up some space, or increase the size of /var.

Start with the housekeeping. Use SAM to clean up log files. Use "du -sk /var/*" to identify the largest subdirectories under /var and drill down into them (again with du) to find large files that can be removed. Most everything in /var/tmp can be removed. Run the cleanup command ("cleanup -c 1", for example) to commit patches occupying space in /var/adm/sw (do not manually delete anything under /var/adm/sw). Use the find command to find and remove core files ("find /var -name core -exec rm {} \;"). Probably the best advice I can give you is to proceed cautiously and make sure you have a backup first so that you can restore anything that shouldn't have been removed.


Pete

[Bernhard Mueller]

Hello,

Pete is right, this typically occurs when files are in use (typically logfiles) and while being in use they are deleted.

The space gets available to the filesystem only after the process writing to that file
has been stopped.

So I would suggest you check your run-scripts
which are executed at boot time e.g. /sbin/init.d/oracle and see what is being logged to which file.

Next you should watch all files that are constantly being written to. For that you use lsof (I take it you have been using this tool).

You can also use find with the -mtime option and check sizes
e.g.
find /var -type f -mtime 0 | xargs ll | awk '{print $5" "$9}' | sort -nr

This will list and sort all files in /var which have been modified today by their size and print size and name.

[Riccardo BIgoni]

thanks to anyone, my problem is various:
- Oracle does not write (and after to cancel) a log of 1,5Gb in /var.
- Not there are crontab.
- I have not cancelled file in the var.
- already it has succeeded (with a called program Spazio) of Sunday to the 6:00.....

Thanks

[Bruno Vidal]

Hi,
The other are rights, your problem is that you have somewhere on your system one or several process that have some file open, but the inode has been removed from, so the space will not be free until the process end. I've attache a small binary named uli (for UnLInked files). It browse the entire process table for file descriptor with unliked inode. With this small tool you will be able to find wich process is using this kind of files, and so stop it.

Cheers.

[V. Nyga]

Hi,

also check /var/adm/crash, if there are subdirectories core.1, core.2 ...
These directories take much space (core dump) and are written if something is not ok with the workstation (for example at shutdown). If they are with a newer date you should open a call at HP. If they are older HP told me to delete them. More interesting for HP would be the files from /var/tombstones, which also show errors of a workstation, written by online diagnostics.

Regards
Volkmar

[GerGon]

One more not fixed yet. This not a solution, it is a experience only.

All days in the morning, I copy files from server A to server B, about 10Gb. Before, I do a rm *

Sometimes, my disk is at 100%, but the real % is 10%.

I do a shutdown of server and then it come up, the space are correct, of course, this server is for backup only.
I don't know Why it happend. NO files open, no active cron. Actually, I don't know How can I fix it?

On one thead that I saw, it tell about to apply a patch.

This is a reference to u.

[Massimo Bianchi]

Hi Riccardo,

are oracle log files pointing to the /var filesystem ?


Expecially the listener can log many many lines, and space is not released untill you stop/start it.

Can you post a bdf and a du -ks /var* ?

just to see what is installed...
Other things that logs very much are the performance monitor of the middaemon, they spool to the /var/opt/perf...something.


Other temporary occupation may arrive from the spool, they lay in the /var/spool, and a big spool can fill your system pretty fast.


HTH,
Massimo

[Mark Greene_1]

You may want to consider setting up /var/tmp, or whereever else in /var that Oracle is logging, as a seperate file system if you have the space to do so.

I'd also cron a process to run lsof and fuser to look for delete but not yet space-freed files in these directories. I'd run every 5 minutes for about 8 hours and log the results. Then you can see how random or not this is.

HTH
mark