topic Re: Full partition, but nothing in it in Operating System - HP-UX

Full partition, but nothing in it

Kim Kendall — Wed, 05 Feb 2003 14:23:32 GMT

We have a /tmp partition that is 2GB. Last night, it hit 100% full. Someone went in and removed some files.

Later, it hit 100% full again. I went in this time and anything that didn't have a current date stamp was removed. What was left amounted to 1-2MB.

The %used (from bdf command), never seemed to go down. And sure enough, /tmp filled up again. But there are no files or directories that can account for the 2GB.

My hunch is that the first time someone removed files from /tmp, they ended up removing something that was open for write, and the system never gave up that space in memory, and that may be hanging onto (allocated) disk space.

Any other ideas of why? And any ideas how to find the villan?

Re: Full partition, but nothing in it

Pete Randall — Wed, 05 Feb 2003 14:25:42 GMT

Kim,

Probably the only way to find the villain is to use lsof.

Pete

Re: Full partition, but nothing in it

Ken Hubnik_2 — Wed, 05 Feb 2003 14:25:58 GMT

Sounds like the process that was the villiam was still running and even if you deleted the file until the process goes away the space will not show up.

Re: Full partition, but nothing in it

T G Manikandan — Wed, 05 Feb 2003 14:31:01 GMT

The file system /tmp is used for all temporary work area of all the processes.
Probably when the system is rebooted the /tmp is cleared.

Also,

Now /tmp is full is that some process is filling up the fil e system.
you need to identify the process that is causing this.

you can use lsof for this.
Check this link for the download site for lsof.

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x346d402f24d5d61190050090279cd0f9,00.html

shutdown this process and then start.

That should clear up the space

Re: Full partition, but nothing in it

Karthik S S — Wed, 05 Feb 2003 14:33:49 GMT

Try this command,

fuser -ck /tmp

and check for larger files again under that partition ..

Regards,
Karthik

Re: Full partition, but nothing in it

T G Manikandan — Wed, 05 Feb 2003 14:40:33 GMT

steven has a suggestion on to find the number of open files by a process

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xfa2a8f960573d611abdb0090277a778c,00.html

check it

Re: Full partition, but nothing in it

Stanimir — Wed, 05 Feb 2003 14:45:05 GMT

Hi!
I have such problem months ago, and
#fsck -p
was my solution.
Regards, Stan

Re: Full partition, but nothing in it

David_246 — Wed, 05 Feb 2003 14:47:44 GMT

Hi,

At first I would like to strongly un-recommend you to do the "fuser -ck /tmp" !!

This will kill all open processes in /tmp, i.e. it will kill your system.
The best you can do is a "fuser -cu /tmp", This will tell you the current processes open in /tmp.
The reason for your error is that someone has removed a file while a process still had the file open for writing. Therefor always first check a file before deleting it, escpecialy when residing in /tmp. You can do this using "fuser -fu /tmp/".

Ok, what to do :
- bdf /tmp
- cd /tmp; du -ks .

These two should differ a lot (df counts, bdf looks up free space).
If so, do the "fuser -cu /tmp" to see all the current open processes in /tmp. ps -ef | grep all of them to see if this could be your file. When killing the right process bdf immediatly shows you the correct free space.
Of course you can compare this again with "df-ks .".

Never delete a file in /tmp without checking the file first with "fuser -fu /tmp/
And if the file is in use, but has to be deleted, then always use : "> /tmp/"

This will only zero the file and will not change the inode, etc.

Good luck !!

Regs David

Re: Full partition, but nothing in it

Kim Kendall — Wed, 05 Feb 2003 15:14:08 GMT

>> And if the file is in use, but has to be deleted, then always use : "> /tmp/"

Yeah, I wish he would have done this, or cat'd /dev/null into it.

I already used du to to see what was in tmp as a whole. There's nothing in there besides a few small files.

tried lsof /tmp and the output doesn't give me the files that are open. Just the dir. (see attached)

Re: Full partition, but nothing in it

Kim Kendall — Wed, 05 Feb 2003 15:24:29 GMT

Stanimir -

What kind of system do you use that you can use a -p option with fsck?

Re: Full partition, but nothing in it

Ravi_8 — Wed, 05 Feb 2003 15:36:15 GMT

Hi,

#umount /tmp
#fsck -o full /dev/vg00/lvol4

may solve problem

Re: Full partition, but nothing in it

Kim Kendall — Wed, 05 Feb 2003 15:51:59 GMT

You can't unmount /tmp (unless you're prepared to go to single user)

Also tried lsof -s +D /tmp
(see attached)

Nothing that accounts for a full 2GB fils system.

Re: Full partition, but nothing in it

Vincent Fleming — Wed, 05 Feb 2003 16:19:34 GMT

From some of your attachments, it may be oracle that has a file open. Can you stop and restart the database?

Re: Full partition, but nothing in it

Kim Kendall — Wed, 05 Feb 2003 18:02:40 GMT

Yeah, that was my guess as well.