topic Re: Urgent Help Please. /tmp 100% full in Operating System - HP-UX

Urgent Help Please. /tmp 100% full

Raoof — Thu, 01 Aug 2013 12:40:45 GMT

Hello Experts,

Need urgent help please.

Server Details:HP-UX rx3600 B.11.31 U ia64

I got below message on the sql prompt as well as on user prompt.

vxfs: msgcnt 27130 mesg 001: V-2-1: vx_nospace - /dev/vg00/lvol4 file system full (2 block extent)

when i look for dbf it shows that /tmp directory gets 100% full, tried to free some space but still it is 99 to 100% usage.

HP_perf_vmstat is getting updated frequently, can we create a new file with the same name and drop the existing one by taking the backup of it.

-rw-r--r-- 1 oracle oinstall 429617270 Aug 1 12:51 HP_perf_vmstat

I will be really thankful if you could provide the solution/workaround to fix it as it is production database server.

Thanks & Best Regards,
Syed

Re: Urgent Help Please. /tmp 100% full

Raoof — Thu, 01 Aug 2013 13:02:23 GMT

I erased all unnecessary files. In spite of that the /tmp dir is still showing %100 full. What could be a problem?

Please advise.

Re: Urgent Help Please. /tmp 100% full

Patrick Wallek — Thu, 01 Aug 2013 13:23:17 GMT

If the files that you removed happen to be in use, then the space will not be reclaimed until the process(es) using those files stop.

Re: Urgent Help Please. /tmp 100% full

Raoof — Thu, 01 Aug 2013 13:40:40 GMT

Thanks for the reply.

Please help me out to check for those process and kill it to release the space at earliest.

Regards

Re: Urgent Help Please. /tmp 100% full

Torsten. — Thu, 01 Aug 2013 14:00:10 GMT

>> HP_perf_vmstat is getting updated frequently, can we create a new file with the same name and drop the existing one by taking the backup of it.

You deleted an open file, better keep it and delete the contents only.

Re: Urgent Help Please. /tmp 100% full

Raoof — Thu, 01 Aug 2013 14:24:21 GMT

Dear Torsten,

I have already deleted the files after taking backup. Please advise what i have do now as am continously getting this below message

vxfs: msgcnt 28998 mesg 001: V-2-1: vx_nospace - /dev/vg00/lvol4 file system full (2 block extent)

now there is 0 space available in the /tmp.

Re: Urgent Help Please. /tmp 100% full

Torsten. — Thu, 01 Aug 2013 14:38:06 GMT

As Patrick said, you need to stop the process that writes to the (deleted) file.

Re: Urgent Help Please. /tmp 100% full

Raoof — Thu, 01 Aug 2013 14:48:37 GMT

#fuser -cu /tmp
/tmp: 1493o(root) 1483o(root) 1475o(root) 1467o(root) 13615c(root) 2094o(root) 2042o(root) 2320o(oracle) 2334o(oracle) 14313c(oracle) 14194o(oracle) 28666c(root) 14107c(oracle)

from the above how do i know which pid is of that deleted file i.e. of "HP_perf_vmstat", or do i need to kill all those process?

Re: Urgent Help Please. /tmp 100% full

Raoof — Thu, 01 Aug 2013 15:18:22 GMT

Awaiting for the reply!!!

Please help me out to solve the issue!!!

Re: Urgent Help Please. /tmp 100% full

Torsten. — Thu, 01 Aug 2013 15:21:29 GMT

Do you have lsof installed?

Re: Urgent Help Please. /tmp 100% full

Raoof — Thu, 01 Aug 2013 16:10:53 GMT

No, it is not installed in the server.

Re: Urgent Help Please. /tmp 100% full

Raoof — Thu, 01 Aug 2013 17:11:03 GMT

Any advise please.....its really very urgent experts... please help me out...

Re: Urgent Help Please. /tmp 100% full

sbrews — Thu, 01 Aug 2013 17:46:52 GMT

From your earlier post:

===

#fuser -cu /tmp
/tmp: 1493o(root) 1483o(root) 1475o(root) 1467o(root) 13615c(root) 2094o(root) 2042o(root) 2320o(oracle) 2334o(oracle) 14313c(oracle) 14194o(oracle) 28666c(root) 14107c(oracle)

from the above how do i know which pid is of that deleted file i.e. of "HP_perf_vmstat", or do i need to kill all those process?

===

Since you dont have lsof, you will need to look at each individual process to determine what it is running. For example, take the first entry (after /tmp) :

1493o(root)

All you need from that are the digits - 1493. Issue a ps command: ps -ef | grep 1493

Look at what it gives back. Does this look like the process that is/was running "HP_perf_vmstat"?

If so, kill that process: kill -15 1493 or kill -9 1493.

If not, look at the next process in the list. Repeat with each process ID until you find the process that needs to be stopped.

Be sure to carefully examine the process before killing it AND be sure you really want to kill it. Killing the wrong process may lead to other issues.

As an alternative - was that command kicked off as part of something else?

IE is there an overall process/application that can be shutdown: /sbin/init.d/someapplication stop? This would be the cleaner way if it is an option.

Re: Urgent Help Please. /tmp 100% full

Raoof — Thu, 01 Aug 2013 20:27:14 GMT

Hi sbrew,

Thanks for your detail steps. Following are the results of all those processes, but i didn' see any of them shows about the process which i was looking for i.e. "HP_perf_vmstat".

mcmsdbl1@root@/>#fuser -cu /tmp

/tmp: 1493o(root) 1483o(root) 1475o(root) 1467o(root) 13615c(root) 2094o(root) 2042o(root) 2320o(oracle) 2334o(oracle) 14313c(oracle) 14194o(oracle) 14107c(oracle)

mcmsdbl1@root@/>#ps -ef | grep 1493
root 1493 1 0 Jun 21 ? 0:00 /usr/sbin/trapdestagt
root 11506 8306 1 22:57:32 pts/2 0:00 grep 1493
mcmsdbl1@root@/>#ps -ef | grep 1483
root 1483 1 0 Jun 21 ? 6:42 /usr/sbin/mib2agt
root 11517 8306 0 22:57:46 pts/2 0:00 grep 1483
mcmsdbl1@root@/>#ps -ef | grep 1475
root 1475 1 0 Jun 21 ? 9:03 /usr/sbin/ipv6agt
root 11519 8306 0 22:57:54 pts/2 0:00 grep 1475
mcmsdbl1@root@/>#ps -ef | grep 1467
root 1467 1 0 Jun 21 ? 0:00 /usr/sbin/hp_unixagt
root 11523 8306 1 22:58:01 pts/2 0:00 grep 1467
mcmsdbl1@root@/>#ps -ef | grep 13615
root 13615 13612 0 11:46:41 pts/1 0:05 -sh
root 11544 8306 0 22:58:24 pts/2 0:00 grep 13615
oracle 28942 13615 0 18:01:37 pts/1 0:00 -sh
mcmsdbl1@root@/>#ps -ef | grep 2094
root 2094 1 0 Jun 21 ? 0:00 /bin/sh /sbin/init.d/init.ohasd run
root 11546 8306 0 22:58:38 pts/2 0:00 grep 2094
mcmsdbl1@root@/>#ps -ef | grep 2042
root 2042 1 0 Jun 21 ? 425:59 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
root 11548 8306 0 22:58:47 pts/2 0:00 grep 2042
mcmsdbl1@root@/>#ps -ef | grep 2320
oracle 2320 1 0 Jun 21 ? 29:20 /u01/app/11.2.0/grid/bin/gpnpd.bin
root 11558 8306 0 22:58:55 pts/2 0:00 grep 2320
mcmsdbl1@root@/>#ps -ef | grep 2334
oracle 2334 1 0 Jun 21 ? 420:07 /u01/app/11.2.0/grid/bin/ocssd.bin
root 11560 8306 0 22:59:06 pts/2 0:00 grep 2334
mcmsdbl1@root@/>#ps -ef | grep 14313
oracle 14313 14107 0 Jul 22 pts/4 0:00 sqlplus / as sysdba
root 11584 8306 0 22:59:28 pts/2 0:00 grep 14313
oracle 2768 14313 0 Jul 23 pts/4 0:00 /sbin/sh
mcmsdbl1@root@/>#ps -ef | grep 14194
oracle 14194 1 0 Jul 22 pts/4 9:56 /usr/bin/vmstat -d -n 1
root 11587 8306 0 22:59:40 pts/2 0:00 grep 14194
mcmsdbl1@root@/>#ps -ef | grep 14107
oracle 14313 14107 0 Jul 22 pts/4 0:00 sqlplus / as sysdba
root 11617 8306 0 23:00:00 pts/2 0:00 grep 14107
oracle 14107 9275 0 Jul 22 pts/4 0:00 /sbin/sh
mcmsdbl1@root@/>#

still i see the below message continously going on

vxfs: msgcnt 36206 mesg 001: V-2-1: vx_nospace - /dev/vg00/lvol4 file system full (2 block extent)

server started giving slow response now.

Re: Urgent Help Please. /tmp 100% full

sbrews — Thu, 01 Aug 2013 20:55:40 GMT

Of all the processes listed, I would look at the following:

mcmsdbl1@root@/>#ps -ef | grep 14194
oracle 14194 1 0 Jul 22 pts/4 9:56 /usr/bin/vmstat -d -n 1

it isrunning vmstat and circumstantial evidence (the process name) relates it to hp_perf_vmstat AND it appears to have been running since July 22. Since its being run under the oracle ID, you should contact your DBAs and see what they can tell you about it.

Re: Urgent Help Please. /tmp 100% full

sbrews — Thu, 01 Aug 2013 21:07:22 GMT

on a related note, how big is your /tmp mount point?

Re: Urgent Help Please. /tmp 100% full

Raoof — Thu, 01 Aug 2013 22:13:59 GMT

Hey sbrews,

Thanks a ton for your great assistance to get my issue resolve. After killing that process it releases the space, now the current status of /tmp is as below.

/dev/vg00/lvol4 524288 98688 423416 19% /tmp

about the size of mb is as above in kb.

Thanks & Best Regards

Re: Urgent Help Please. /tmp 100% full

Raoof — Thu, 01 Aug 2013 22:21:15 GMT

FYI, Below the current file system layout.

File System layout

LVM Device file mount point size fs type
/dev/vg00:
/dev/vg00/lvol1 /stand 1792 vxfs
/dev/vg00/lvol2 swap 8192
/dev/vg00/lvol3 / 1024 vxfs
/dev/vg00/lvol4 /tmp 2048 vxfs
/dev/vg00/lvol5 /home 128 vxfs
/dev/vg00/lvol6 /opt 16448 vxfs
/dev/vg00/lvol7 /usr 5824 vxfs
/dev/vg00/lvol8 /var 16384 vxfs
/dev/vg00/lvo19 /u01 61440 vxfs
/dev/vg00/lvo20 /backup 102400vxfs
/dev/vg00/lvo22 57344 unused
/dev/vg00 unallocated 6016

If possible can i insrease the size of /tmp by using some space from the unallocated size i.e. of around 5 to 6gb.

Re: Urgent Help Please. /tmp 100% full

sbrews — Thu, 01 Aug 2013 22:24:44 GMT

your /tmp is only 512M in size, you should look into making it larger. Overall size depends on how your system is being used. I tend to go with 2G as a minimum and depending on system use, I may make it larger.

Re: Urgent Help Please. /tmp 100% full

sbrews — Thu, 01 Aug 2013 22:27:35 GMT

Check this page - it has some examples of adding extending file systems, etc.

http://www.unixguide.net/hp/hplvmtasks.shtml

Also, if you google for hpux file system resizing, you should find lots of other examples.