Filesystem full issue

[Sean OB_1]

Ok, we've all seen this before, but I wanted to verify my thinking.

I have a system that is showing /tmp at 98% full. I've gone into /tmp and removed everything.

I've done fuser to see who is using it and was able to stop (not kill) all processes using it and unmount /tmp.

I then remount /tmp and still see 98% full.

Is there anything left to do other than reboot the system?

I do have one zombie process on the system.

TIA,

Sean

[Tom Ward_1]

This won't help you this time, but it may help you next time.

I normally use fuser before I remove a file. If there's a process that still has ahold of one of the deleted files and you can find it. That would save you rebooting, but good luck finding it. If you fuser the other files in /tmp you may be able to guess what applicaitons are using /tmp and cycle those.

Even if a process is using a file, such as a log, you can use " > filename" to zero it out wihtout problems.

[Martin Johnson]

If you were able to umount /tmp, then all channels are closed. You should see the files in the fiel system. Do:

cd /tmp
du -kx | sort -rn | more

to see the directory using the most space.

If that doesn't help, I would umount /tmp again and run fsck on it.

HTH
Marty

[Roger Baptiste]

Hi ,

Run lsof tool too see what processes are writing to /tmp filesystem. (lsof is a free tool available from the hp portable archive site). this is probably due to a runaway process dumping onto /tmp, so if you identify it through lsof and kill it, it should be ok.

Otherwise, another option is to make sure the /tmp filesystem is really using that much space. Did you do a lvdisplay on the /tmp lv ?? Does it show that much usage , even after unmounting it?? You can do a lvremove and lvcreate again.

HTH
raj

[harry d brown jr]

Sean,

get "lsof"!!!!!!!!!

fuser is just that (i'll leave that to the imagination"

http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.64/

How did you "stop" processes from using /tmp, temporarily???

live free or die
harry

[Scott Marer]

Here is how /tmp could remain full even after deleting the files :

This problem is occurring because you had files in the directory, and then you deleted them when a process was accessing the files. If a process is accessing a file when the file is deleted, it only removes the directory entry, not the file itself. The file itself will not be deleted until the process accessing it releases it. Once you kill those processes, then the directory will go back to normal size.

With that said, the fact that you were able to unmount /tmp doesn't make sense with the above scenario, since in order for /tmp to be unmounted any active processes would have had to be killed.

I am in agreement with the fsck of the /tmp file system; something does not appear to be in check.

[Sean OB_1]

Ok, have a look here:


cosmo1:/sbin/rc2.d-> bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 143360 116349 25379 82% /
/dev/vg00/lvol1 83733 45680 29679 61% /stand
/dev/vg00/lvol8 1064960 416458 611579 41% /var
/dev/vg00/lvol7 1191936 601032 557386 52% /usr
/dev/vg00/lvol6 1228800 1123960 98341 92% /opt
/dev/omni/varomni 32772096 15592896 16650460 48% /var/opt/omni
/dev/omni/optomni 1024000 738819 267365 73% /opt/omni
/dev/vg00/lvol4 65536 1506 60419 2% /tmp
cosmo1:/sbin/rc2.d-> fuser -c /tmp
/tmp:

cosmo1:/sbin/rc2.d-> umount /tmp
cosmo1:/sbin/rc2.d-> bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 143360 116349 25379 82% /
/dev/vg00/lvol1 83733 45680 29679 61% /stand
/dev/vg00/lvol8 1064960 416458 611579 41% /var
/dev/vg00/lvol7 1191936 601032 557386 52% /usr
/dev/vg00/lvol6 1228800 1123960 98341 92% /opt
/dev/omni/varomni 32772096 15592896 16650460 48% /var/opt/omni
/dev/omni/optomni 1024000 738819 267365 73% /opt/omni
cosmo1:/sbin/rc2.d-> fsck -y /dev/vg00/lvol4
file system is clean - log replay is not required
cosmo1:/sbin/rc2.d-> mount /dev/vg00/lvol4 /tmp
cosmo1:/sbin/rc2.d-> bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 143360 116349 25379 82% /
/dev/vg00/lvol1 83733 45680 29679 61% /stand
/dev/vg00/lvol8 1064960 416458 611579 41% /var
/dev/vg00/lvol7 1191936 601032 557386 52% /usr
/dev/vg00/lvol6 1228800 1123960 98341 92% /opt
/dev/omni/varomni 32772096 15592896 16650460 48% /var/opt/omni
/dev/omni/optomni 1024000 738819 267365 73% /opt/omni
/dev/vg00/lvol4 65536 1506 60419 2% /tmp
cosmo1:/sbin/rc2.d->


To umount /tmp I had to shutdown opc, snmp, and mwa. All done cleanly through the startup files.

Sean

[Sean OB_1]

forgot this:

cosmo1:/tmp-> ls -al /tmp
total 14
drwxrwxrwx 3 bin bin 5120 Sep 25 18:11 .
drwxr-xr-x 29 root root 2048 Sep 25 16:03 ..
drwx------ 2 root sys 96 Sep 25 16:05 .AgentSockets
prw-rw-rw- 1 root root 0 Sep 25 16:36 dhcpfifo.any
prw-rw-rw- 1 root root 0 Sep 25 16:36 dhcpfifo.root
cosmo1:/tmp->

cosmo1:/tmp-> du -sk *
0 dhcpfifo.any
0 dhcpfifo.root
cosmo1:/tmp->

[harry d brown jr · Accepted solution]

Sean,

You are showing that /tmp is only 2% full.

I'm lost now....


live free or die
harry

[Scott Marer]

Ditto what Harry said...

[harry d brown jr]

Scott,

Did you bring the flashlight?

live free or die
harry

[Sean OB_1]

oh my god am I a dumbass today.

Thanks for all your help for making me realize what an idiot I've been.

You'd think after looking at bdf's about a million times I would read it right!

It's been one of those days.

[harry d brown jr]

sean,

It has nothing to do about being a d'ass. Sometimes we just get lost in the glaze off our monitor. I usually walk into my lab wondering what the hell i went in there for - then turn around and go get a cup of coffee until I figure it out - thank god I don't play chess that way (though not much better).

Sean,

Go outside and get some fresh air, a cup of coffee, cig, candy, whatever, and take a breather.


live free or die
harry

[Michael J. Sabal]

Actually, I've noticed that bdf (in 10.20) doesn't give an accurate percentage. Use df -b instead, and give the system about 10-15 minutes after deleting a large amount of data before checking again. I've had a lot more success (and happy end users) with this method.

[Tom Danzig]

Take the rest of the day off ... sounds like you need it. Tell your boss I said it was OK ;^)

[Sean OB_1]

What a day yesterday was!

After all of this, I was getting ready to leave around 4:30 when the campus had a series of power outages that took down the network, and consequently their service guarded machines!

[Sridhar Bhaskarla]

Sean -

A person without frequent brain freezes is not qualified to be a system administrator.

I was completely bowled by your post and didn't know how it could happen after removing everything and remounting.

No points for this post. Just wanted to share.

-Sri