Freed disk space but bdf does not show it

[hydrocct]

My / was filling up due to a package log file growing too fast (had verbose logging turned on), so I modified the verbose switch, stopped the package, gzipped the package log file and renamed it, restarted the package, but the source of the problem is that I had tail -f running on the log file. The result is that the space is freed, but bdf does not say so (and neither do the ITO/performance monitors). I know a reboot will solve this, but this system is in production. Is there cleaner solution to have bdf reflect reality?

[Sridhar Bhaskarla]

Hi,

Can you see the tail -f process in your process list 'ps -ef'? If so, find the PID and kill it.

If not, then you will have to use "lsof" to find the processes that opened the log file.

-Sri

[G. Vrijhoeven]

Hi,

There must be a process (tail)that keeps the file open. If you manage to find/kill the process ( lsof, fuser ) the disk space will clear up ( in bdf).

Gideon

[A. Clay Stephenson]

This is normal UNIX behavior and it really must work this way. When you rm a file (unlink in UNIX speak) what really happens it that the link count is reduced by one. If the link count reaches zero, the entry is removed from the directory BUT the space is not actually freed until the processes which have the file open either close the files or terminate themselves.

[hydrocct]

I forgot to mention I had killed the tail process, but before I did that, I restarted the package, so the log file is now attached to service guard processes, which I cannot kill. In other words, I deleted a file which was attached to process A (tail), and before stopping A, process B claimed that same file (i.e, the log file is recreated by service guard if it no longer exists). Now bdf is all confused.

[Sridhar Bhaskarla]

Well, in that case reboot (as mentioned by you) is not the ultimate solution. The solution is the same thing you did before. Shut the package and make sure there are no processes associated with it and you should get the space back.

-Sri

[hydrocct]

This a very radical solution. Otherwise, I leave the system as it is, will the OS eventually complain that / is full (vxfs message is syslog.log) even though it isn't, or is this problem limited to bdf (which would mean that ITO templates use bdf).

[Sridhar Bhaskarla]

The space is not going to be freed until the process releases the file. So, 'bdf' is going to
show the reality accounting for that file.

You are going to receive a page once it reaches the configured threshold.

-Sri

[G. Vrijhoeven]

Hi,

Do you start customer defined run commands CDRC in the foregound? In other words if you cmviewcl do you get,package starting or running. If it is starting change the CDRC so they start in the background or give back a exit 0. ( script that starts the software)


Gideon

[Jeff Schussele]

Hi,

You might be able to fee up space elsewhere in /.
Run the following
du -akx | sort -rn | more

This will show the largest files/dirs in descending order. You may be able to remove/trim something just enough to get you by until you can halt the pkg again.

HTH,
Jeff

[Elmar P. Kolkman]

If you have lsof available, you could try to find the process giving the problem... Do a 'lsof | grep vg00/lvol3 | grep -v VREG'.
This might help you pinpoint the process, because it will report processes using the / directory, but your process will not show up in 'fuser /'.

[Jean-Louis Phelix]

Hi,

Or use this small script to find out which process has an opened indoe but no dir entry associated (rm has been done but space not freed until process dies)

Regards.