Urgent Help Please. /tmp 100% full

[Raoof]

No, it is not installed in the server.

[Raoof]

Any advise please.....its really very urgent experts... please help me out...

[sbrews · Accepted solution]

From your earlier post:

===

#fuser -cu /tmp
/tmp: 1493o(root) 1483o(root) 1475o(root) 1467o(root) 13615c(root) 2094o(root) 2042o(root) 2320o(oracle) 2334o(oracle) 14313c(oracle) 14194o(oracle) 28666c(root) 14107c(oracle)

from the above how do i know which pid is of that deleted file i.e. of "HP_perf_vmstat", or do i need to kill all those process?

===

Since you dont have lsof, you will need to look at each individual process to determine what it is running.  For example, take the first entry (after /tmp) :

1493o(root)

All you need from that are the digits - 1493.  Issue a ps command: ps -ef | grep 1493

Look at what it gives back.  Does this look like the process that is/was running  "HP_perf_vmstat"? 

If so, kill that process: kill -15 1493 or kill -9 1493.

If not, look at the next process in the list.  Repeat with each process ID until you find the process that needs to be stopped.

Be sure to carefully examine the process before killing it AND be sure you really want to kill it.  Killing the wrong process may lead to other issues.

As an alternative - was that command kicked off as part of something else? 

IE  is there an overall process/application that can be shutdown: /sbin/init.d/someapplication stop?  This would be the cleaner way if it is an option.

[Raoof]

Hi sbrew,

Thanks for your detail steps. Following are the results of all those processes, but i didn' see any of them shows about the process which i was looking for i.e.   "HP_perf_vmstat".

mcmsdbl1@root@/>#fuser -cu /tmp

/tmp: 1493o(root) 1483o(root) 1475o(root) 1467o(root) 13615c(root) 2094o(root) 2042o(root) 2320o(oracle) 2334o(oracle) 14313c(oracle) 14194o(oracle) 14107c(oracle)

mcmsdbl1@root@/>#ps -ef | grep 1493
root 1493 1 0 Jun 21 ? 0:00 /usr/sbin/trapdestagt
root 11506 8306 1 22:57:32 pts/2 0:00 grep 1493
mcmsdbl1@root@/>#ps -ef | grep 1483
root 1483 1 0 Jun 21 ? 6:42 /usr/sbin/mib2agt
root 11517 8306 0 22:57:46 pts/2 0:00 grep 1483
mcmsdbl1@root@/>#ps -ef | grep 1475
root 1475 1 0 Jun 21 ? 9:03 /usr/sbin/ipv6agt
root 11519 8306 0 22:57:54 pts/2 0:00 grep 1475
mcmsdbl1@root@/>#ps -ef | grep 1467
root 1467 1 0 Jun 21 ? 0:00 /usr/sbin/hp_unixagt
root 11523 8306 1 22:58:01 pts/2 0:00 grep 1467
mcmsdbl1@root@/>#ps -ef | grep 13615
root 13615 13612 0 11:46:41 pts/1 0:05 -sh
root 11544 8306 0 22:58:24 pts/2 0:00 grep 13615
oracle 28942 13615 0 18:01:37 pts/1 0:00 -sh
mcmsdbl1@root@/>#ps -ef | grep 2094
root 2094 1 0 Jun 21 ? 0:00 /bin/sh /sbin/init.d/init.ohasd run
root 11546 8306 0 22:58:38 pts/2 0:00 grep 2094
mcmsdbl1@root@/>#ps -ef | grep 2042
root 2042 1 0 Jun 21 ? 425:59 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
root 11548 8306 0 22:58:47 pts/2 0:00 grep 2042
mcmsdbl1@root@/>#ps -ef | grep 2320
oracle 2320 1 0 Jun 21 ? 29:20 /u01/app/11.2.0/grid/bin/gpnpd.bin
root 11558 8306 0 22:58:55 pts/2 0:00 grep 2320
mcmsdbl1@root@/>#ps -ef | grep 2334
oracle 2334 1 0 Jun 21 ? 420:07 /u01/app/11.2.0/grid/bin/ocssd.bin
root 11560 8306 0 22:59:06 pts/2 0:00 grep 2334
mcmsdbl1@root@/>#ps -ef | grep 14313
oracle 14313 14107 0 Jul 22 pts/4 0:00 sqlplus / as sysdba
root 11584 8306 0 22:59:28 pts/2 0:00 grep 14313
oracle 2768 14313 0 Jul 23 pts/4 0:00 /sbin/sh
mcmsdbl1@root@/>#ps -ef | grep 14194
oracle 14194 1 0 Jul 22 pts/4 9:56 /usr/bin/vmstat -d -n 1
root 11587 8306 0 22:59:40 pts/2 0:00 grep 14194
mcmsdbl1@root@/>#ps -ef | grep 14107
oracle 14313 14107 0 Jul 22 pts/4 0:00 sqlplus / as sysdba
root 11617 8306 0 23:00:00 pts/2 0:00 grep 14107
oracle 14107 9275 0 Jul 22 pts/4 0:00 /sbin/sh
mcmsdbl1@root@/>#

still i see the below message continously going on

vxfs: msgcnt 36206 mesg 001: V-2-1: vx_nospace - /dev/vg00/lvol4 file system full (2 block extent)

server started giving slow response now.

[sbrews]

Of all the processes listed, I would look at the following:

mcmsdbl1@root@/>#ps -ef | grep 14194
oracle 14194 1 0 Jul 22 pts/4 9:56 /usr/bin/vmstat -d -n 1

it isrunning vmstat and circumstantial evidence (the process name) relates it to hp_perf_vmstat AND it appears to have been running since July 22.  Since its being run under the oracle ID, you should contact your DBAs and see what they can tell you about it.

[sbrews]

on a related note, how big is your /tmp mount point?

[Raoof]

Hey sbrews,

Thanks a ton for your great assistance to get my issue resolve. After killing that process it releases the space, now the current status of /tmp is as below.

/dev/vg00/lvol4     524288   98688  423416   19% /tmp

about the size of mb is as above in kb.

Thanks & Best Regards

[Raoof]

FYI, Below the current file system layout.

File System layout

LVM Device file mount point size fs type
/dev/vg00:
/dev/vg00/lvol1 /stand 1792 vxfs
/dev/vg00/lvol2 swap 8192
/dev/vg00/lvol3 / 1024 vxfs
/dev/vg00/lvol4 /tmp 2048 vxfs
/dev/vg00/lvol5 /home 128 vxfs
/dev/vg00/lvol6 /opt 16448 vxfs
/dev/vg00/lvol7 /usr 5824 vxfs
/dev/vg00/lvol8 /var 16384 vxfs
/dev/vg00/lvo19 /u01 61440 vxfs
/dev/vg00/lvo20 /backup 102400vxfs
/dev/vg00/lvo22 57344 unused
/dev/vg00 unallocated 6016

If possible can i insrease the size of /tmp by using some space from the unallocated size i.e. of around 5 to 6gb.

[sbrews]

your /tmp is only 512M in size, you should look into making it larger.  Overall size depends on how your system is being used.  I tend to go with 2G as a minimum and depending on system use, I may make it larger.

[sbrews]

Check this page - it has some examples of adding extending file systems, etc.

http://www.unixguide.net/hp/hplvmtasks.shtml

Also, if you google for hpux file system resizing, you should find lots of other examples.