topic Re: bdf /tmp shows 100% full in Operating System - HP-UX

bdf /tmp shows 100% full

Ravi S. Banda — Mon, 16 May 2005 15:36:37 GMT

bdf /tmp shows 100% full!!!
On our PROD server, I don't understand why.

I did ll -R /tmp and very few files. I did
ls -altr shows very few files and directories. I went into the directories and hidden directories, NOTHING!!!

du -ks /tmp shows:
71 /tmp

bdf shows 100% full.

/dev/vg00/lvol4 819200 819200 0 100% /tmp

Why is this? This is our PROD. Please help!!!

Re: bdf /tmp shows 100% full

A. Clay Stephenson — Mon, 16 May 2005 15:40:11 GMT

Because some processes have some files open even if they have been rm'ed the space is only freed when the processes which have them open either terminate or close the filoes. You can use lsof (or fuser) to see what process(es) have the file(s) open.

By the way, /tmp is not supposed to be used for user temp files; that is the role of /var/tmp these days. /tmp is strictly for OS related processes.

Re: bdf /tmp shows 100% full

John Poff — Mon, 16 May 2005 15:40:28 GMT

Hi,

You probably have a process that is holding open a file in /tmp, but the file has been removed. The disk space won't be released until the process finishes. The trick now is to figure out which process is holding the disk space. I'd use Glance or lsof to try to find the offending process.

JP

Re: bdf /tmp shows 100% full

Ravi S. Banda — Mon, 16 May 2005 15:49:16 GMT

fuser /tmp shows:

/tmp: 922o 930o 938o

are 922, 930 and 938 process ids? I'm sorry, I just inherited this system from the unix sysadmin.

If so,
root 938 1 0 Jan 26 ? 0:00 /usr/sbin/trapdestagt

root 930 1 0 Jan 26 ? 11:51 /usr/sbin/mib2agt

root 922 1 0 Jan 26 ? 0:00 /usr/sbin/hp_unixagt

Thanks!
Ravi.

Re: bdf /tmp shows 100% full

A. Clay Stephenson — Mon, 16 May 2005 15:53:02 GMT

Yes, those are PID's.

Re: bdf /tmp shows 100% full

Ravi S. Banda — Mon, 16 May 2005 15:55:36 GMT

root 938 1 0 Jan 26 ? 0:00 /usr/sbin/trapdestagt

root 930 1 0 Jan 26 ? 11:51 /usr/sbin/mib2agt

root 922 1 0 Jan 26 ? 0:00 /usr/sbin/hp_unixagt

Any ideas as to what these "agt" processes are?

Thanks for all your help.
-- Ravi.

Re: bdf /tmp shows 100% full

Robert-Jan Goossens — Mon, 16 May 2005 16:04:04 GMT

agents or daemons

check this doc.

http://docs.hp.com/en/B7612-90015/apbs05.html

Regards,
Robert-Jan

Re: bdf /tmp shows 100% full

Bill Hassell — Mon, 16 May 2005 21:01:46 GMT

These are SNMP (Simple Network Management Protocol) and should only be enabled if your network department is running special monitoring software such as HP OpenView tools. If yo don't know anything about this, disable all SNMP processes as they can be a security risk. These processes provide information about your system to other computers. The following commands will terminate the SNMP processes:

/sbin/init.d/SnmpFddi4 stop

/sbin/init.d/SnmpMaster stop

/sbin/init.d/SnmpTrpDst stop

/sbin/init.d/SnmpHpunix stop

/sbin/init.d/SnmpMib2 stop

These processes are enabled by default and most sysadmins will turn them off unless a Network Operations Center is monitoring systems. To make these SNMP process not run at bootup, edit the file in /etc/rc.config.d that start with: Snmp Change all XXXXX=1 lines to XXXXX=0 and now SNMP agents will not be started.

Now if your system has not been reasonably patched, mib2agt had a fairly serious bug that caused it to grow extremely large. The reason that the space in /tmp isn't being returned is that rm does not really remove a file--it decrements the link count. For a simple file, the link count goes to zero and if the file is not open, it is removed. The problem is that rm will succeed even though a file is still open. So until you kill all processes that have the file open, the file appears to have been deleted but the space has not been returned.

Re: bdf /tmp shows 100% full

Nguyen Anh Tien — Tue, 17 May 2005 00:25:51 GMT

/tmp is used for temporary only.
so you can delete un needed files.
1, search big files
#cd /tmp
#du -sk *|sort -nr
#rm -rf bigfiles (big files listed at above command)
2, extend /tmp lv
see
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=881438

Re: bdf /tmp shows 100% full

Ravi S. Banda — Tue, 17 May 2005 07:27:11 GMT

Hello All,
I went through a list of processes to see what each process was doing.
I killed the custom (cron) processes that I know can be killed. After killing one (or some) of these processes, the used space became 8% immediately! I know what the problem with the cron jobs was. I fixed that problem afterwards.
The only thing I don't understand is that "fuser /tmp" did NOT show any of these processes as using /tmp because obviously, killing one of these processes cleared the /tmp.
Thanks!
Ravi.

Re: bdf /tmp shows 100% full

Bill Hassell — Tue, 17 May 2005 07:48:26 GMT

Unfortunately, fuser missies about 50% of processes that have open files on a mountpoint. The tool of choice (until fuser is rewritten) is lsof (list open files). You can download lsof from the Liverpool Archive at: http://hpux.connect.org.uk/
Search for lsof

Re: bdf /tmp shows 100% full

Ravi S. Banda — Tue, 17 May 2005 09:29:55 GMT

Thanks, Bill.
We're on hp-ux 11.00, though. http://hpux.connect.org.uk shows only for 11i pa-risc and itanium 2.

Where can I download for 11.0?
Thanks!
Ravi.

Re: bdf /tmp shows 100% full

Bill Hassell — Tue, 17 May 2005 10:35:12 GMT

The author is Vic Abell and he has a web page for lsof at: http://people.freebsd.org/~abe/

Re: bdf /tmp shows 100% full

A. Clay Stephenson — Tue, 17 May 2005 10:47:40 GMT

And this points to the problem I was alluding to earlier --- you should not be using /tmp for user processes (even cron jobs). The current convention (now well over a decade old) is to use /var/tmp for these temporary files. You should change your scripts/programs to reflect this and see that TMPDIR is not set to /tmp.

Re: bdf /tmp shows 100% full

Sorrel G. Jakins — Tue, 17 May 2005 10:55:50 GMT

You need a utlity called lsof.
then lsof|grep /tmp will show who and what is using /tmp