topic Re: Freed disk space but bdf does not show it in Operating System - HP-UX

Freed disk space but bdf does not show it

hydrocct — Mon, 19 Jan 2004 16:22:45 GMT

My / was filling up due to a package log file growing too fast (had verbose logging turned on), so I modified the verbose switch, stopped the package, gzipped the package log file and renamed it, restarted the package, but the source of the problem is that I had tail -f running on the log file. The result is that the space is freed, but bdf does not say so (and neither do the ITO/performance monitors). I know a reboot will solve this, but this system is in production. Is there cleaner solution to have bdf reflect reality?

Re: Freed disk space but bdf does not show it

Sridhar Bhaskarla — Mon, 19 Jan 2004 16:29:49 GMT

Hi,

Can you see the tail -f process in your process list 'ps -ef'? If so, find the PID and kill it.

If not, then you will have to use "lsof" to find the processes that opened the log file.

-Sri

Re: Freed disk space but bdf does not show it

G. Vrijhoeven — Mon, 19 Jan 2004 16:33:46 GMT

Hi,

There must be a process (tail)that keeps the file open. If you manage to find/kill the process ( lsof, fuser ) the disk space will clear up ( in bdf).

Gideon

Re: Freed disk space but bdf does not show it

A. Clay Stephenson — Mon, 19 Jan 2004 16:37:09 GMT

This is normal UNIX behavior and it really must work this way. When you rm a file (unlink in UNIX speak) what really happens it that the link count is reduced by one. If the link count reaches zero, the entry is removed from the directory BUT the space is not actually freed until the processes which have the file open either close the files or terminate themselves.

Re: Freed disk space but bdf does not show it

hydrocct — Mon, 19 Jan 2004 16:41:42 GMT

I forgot to mention I had killed the tail process, but before I did that, I restarted the package, so the log file is now attached to service guard processes, which I cannot kill. In other words, I deleted a file which was attached to process A (tail), and before stopping A, process B claimed that same file (i.e, the log file is recreated by service guard if it no longer exists). Now bdf is all confused.

Re: Freed disk space but bdf does not show it

Sridhar Bhaskarla — Mon, 19 Jan 2004 16:45:28 GMT

Well, in that case reboot (as mentioned by you) is not the ultimate solution. The solution is the same thing you did before. Shut the package and make sure there are no processes associated with it and you should get the space back.

-Sri

Re: Freed disk space but bdf does not show it

hydrocct — Mon, 19 Jan 2004 16:50:21 GMT

This a very radical solution. Otherwise, I leave the system as it is, will the OS eventually complain that / is full (vxfs message is syslog.log) even though it isn't, or is this problem limited to bdf (which would mean that ITO templates use bdf).

Re: Freed disk space but bdf does not show it

Sridhar Bhaskarla — Mon, 19 Jan 2004 16:55:56 GMT

The space is not going to be freed until the process releases the file. So, 'bdf' is going to
show the reality accounting for that file.

You are going to receive a page once it reaches the configured threshold.

-Sri

Re: Freed disk space but bdf does not show it

G. Vrijhoeven — Mon, 19 Jan 2004 17:06:47 GMT

Hi,

Do you start customer defined run commands CDRC in the foregound? In other words if you cmviewcl do you get,package starting or running. If it is starting change the CDRC so they start in the background or give back a exit 0. ( script that starts the software)

Gideon

Re: Freed disk space but bdf does not show it

Jeff Schussele — Mon, 19 Jan 2004 17:12:39 GMT

Hi,

You might be able to fee up space elsewhere in /.
Run the following
du -akx | sort -rn | more

This will show the largest files/dirs in descending order. You may be able to remove/trim something just enough to get you by until you can halt the pkg again.

HTH,
Jeff

Re: Freed disk space but bdf does not show it

Elmar P. Kolkman — Tue, 20 Jan 2004 02:58:55 GMT

If you have lsof available, you could try to find the process giving the problem... Do a 'lsof | grep vg00/lvol3 | grep -v VREG'.
This might help you pinpoint the process, because it will report processes using the / directory, but your process will not show up in 'fuser /'.

Re: Freed disk space but bdf does not show it

Jean-Louis Phelix — Tue, 20 Jan 2004 07:36:02 GMT

Hi,

Or use this small script to find out which process has an opened indoe but no dir entry associated (rm has been done but space not freed until process dies)

Regards.