System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Urgent Help Please. /tmp 100% full

SOLVED
Go to solution
Raoof
Regular Advisor

Urgent Help Please. /tmp 100% full

 

Hello Experts,

 

Need urgent help please.

 

Server Details:HP-UX rx3600 B.11.31 U ia64

 

 

I got below message on the sql prompt as well as on user prompt.

 

vxfs: msgcnt 27130 mesg 001: V-2-1: vx_nospace - /dev/vg00/lvol4 file system full (2 block extent)

 

when i look for dbf it shows that /tmp directory gets 100% full, tried to free some space but still it is 99 to 100% usage.

 

HP_perf_vmstat is getting updated frequently, can we create a new file with the same name and drop the existing one by taking the backup of it.

 

-rw-r--r--   1 oracle     oinstall   429617270 Aug  1 12:51 HP_perf_vmstat

 

I will be really thankful if you could provide the solution/workaround to fix it as it is production database server.

 

Thanks & Best Regards,
Syed

 

 

21 REPLIES
Raoof
Regular Advisor

Re: Urgent Help Please. /tmp 100% full

I erased all unnecessary files. In spite of that the /tmp dir is still showing %100 full. What could be a problem?

 

Please advise.

Patrick Wallek
Honored Contributor

Re: Urgent Help Please. /tmp 100% full

If the files that you removed happen to be in use, then the space will not be reclaimed until the process(es) using those files stop.

 

 

Raoof
Regular Advisor

Re: Urgent Help Please. /tmp 100% full

Thanks for the reply.

 

Please help me out to check for those process and kill it to release the space at earliest.

 

Regards

Torsten.
Acclaimed Contributor

Re: Urgent Help Please. /tmp 100% full

>> HP_perf_vmstat is getting updated frequently, can we create a new file with the same name and drop the existing one by taking the backup of it.


You deleted an open file, better keep it and delete the contents only.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Raoof
Regular Advisor

Re: Urgent Help Please. /tmp 100% full

Dear Torsten,

 

I have already deleted the files after taking backup. Please advise what i have do now as am continously getting this below message

 

vxfs: msgcnt 28998 mesg 001: V-2-1: vx_nospace - /dev/vg00/lvol4 file system full (2 block extent)

 

now there is 0 space available in the /tmp.

Torsten.
Acclaimed Contributor

Re: Urgent Help Please. /tmp 100% full

As Patrick said, you need to stop the process that writes to the (deleted) file.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Raoof
Regular Advisor

Re: Urgent Help Please. /tmp 100% full

#fuser -cu /tmp
/tmp: 1493o(root) 1483o(root) 1475o(root) 1467o(root) 13615c(root) 2094o(root) 2042o(root) 2320o(oracle) 2334o(oracle) 14313c(oracle) 14194o(oracle) 28666c(root) 14107c(oracle)

 

from the above how do i know which pid is of that deleted file i.e. of "HP_perf_vmstat", or do i need to kill all those process?

Raoof
Regular Advisor

Re: Urgent Help Please. /tmp 100% full

Awaiting for the reply!!!

Please help me out to solve the issue!!!

Torsten.
Acclaimed Contributor

Re: Urgent Help Please. /tmp 100% full

Do you have lsof installed?

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Raoof
Regular Advisor

Re: Urgent Help Please. /tmp 100% full

No, it is not installed in the server.

Raoof
Regular Advisor

Re: Urgent Help Please. /tmp 100% full

Any advise please.....its really very urgent experts... please help me out...
sbrews
Frequent Advisor
Solution

Re: Urgent Help Please. /tmp 100% full

From your earlier post:

 

===

#fuser -cu /tmp
/tmp: 1493o(root) 1483o(root) 1475o(root) 1467o(root) 13615c(root) 2094o(root) 2042o(root) 2320o(oracle) 2334o(oracle) 14313c(oracle) 14194o(oracle) 28666c(root) 14107c(oracle)

 

from the above how do i know which pid is of that deleted file i.e. of "HP_perf_vmstat", or do i need to kill all those process?

===

 

Since you dont have lsof, you will need to look at each individual process to determine what it is running.  For example, take the first entry (after /tmp) :

 

1493o(root)

 

All you need from that are the digits - 1493.  Issue a ps command: ps -ef | grep 1493

 

Look at what it gives back.  Does this look like the process that is/was running  "HP_perf_vmstat"? 

 

If so, kill that process: kill -15 1493 or kill -9 1493.

 

If not, look at the next process in the list.  Repeat with each process ID until you find the process that needs to be stopped.

 

Be sure to carefully examine the process before killing it AND be sure you really want to kill it.  Killing the wrong process may lead to other issues.

 

As an alternative - was that command kicked off as part of something else? 

IE  is there an overall process/application that can be shutdown: /sbin/init.d/someapplication stop?  This would be the cleaner way if it is an option.

 

 

Raoof
Regular Advisor

Re: Urgent Help Please. /tmp 100% full

Hi sbrew,

 

Thanks for your detail steps. Following are the results of all those processes, but i didn' see any of them shows about the process which i was looking for i.e.   "HP_perf_vmstat".

 

mcmsdbl1@root@/>#fuser -cu /tmp

/tmp: 1493o(root) 1483o(root) 1475o(root) 1467o(root) 13615c(root) 2094o(root) 2042o(root) 2320o(oracle) 2334o(oracle) 14313c(oracle) 14194o(oracle) 14107c(oracle)

 

 

mcmsdbl1@root@/>#ps -ef | grep 1493
root 1493 1 0 Jun 21 ? 0:00 /usr/sbin/trapdestagt
root 11506 8306 1 22:57:32 pts/2 0:00 grep 1493
mcmsdbl1@root@/>#ps -ef | grep 1483
root 1483 1 0 Jun 21 ? 6:42 /usr/sbin/mib2agt
root 11517 8306 0 22:57:46 pts/2 0:00 grep 1483
mcmsdbl1@root@/>#ps -ef | grep 1475
root 1475 1 0 Jun 21 ? 9:03 /usr/sbin/ipv6agt
root 11519 8306 0 22:57:54 pts/2 0:00 grep 1475
mcmsdbl1@root@/>#ps -ef | grep 1467
root 1467 1 0 Jun 21 ? 0:00 /usr/sbin/hp_unixagt
root 11523 8306 1 22:58:01 pts/2 0:00 grep 1467
mcmsdbl1@root@/>#ps -ef | grep 13615
root 13615 13612 0 11:46:41 pts/1 0:05 -sh
root 11544 8306 0 22:58:24 pts/2 0:00 grep 13615
oracle 28942 13615 0 18:01:37 pts/1 0:00 -sh
mcmsdbl1@root@/>#ps -ef | grep 2094
root 2094 1 0 Jun 21 ? 0:00 /bin/sh /sbin/init.d/init.ohasd run
root 11546 8306 0 22:58:38 pts/2 0:00 grep 2094
mcmsdbl1@root@/>#ps -ef | grep 2042
root 2042 1 0 Jun 21 ? 425:59 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
root 11548 8306 0 22:58:47 pts/2 0:00 grep 2042
mcmsdbl1@root@/>#ps -ef | grep 2320
oracle 2320 1 0 Jun 21 ? 29:20 /u01/app/11.2.0/grid/bin/gpnpd.bin
root 11558 8306 0 22:58:55 pts/2 0:00 grep 2320
mcmsdbl1@root@/>#ps -ef | grep 2334
oracle 2334 1 0 Jun 21 ? 420:07 /u01/app/11.2.0/grid/bin/ocssd.bin
root 11560 8306 0 22:59:06 pts/2 0:00 grep 2334
mcmsdbl1@root@/>#ps -ef | grep 14313
oracle 14313 14107 0 Jul 22 pts/4 0:00 sqlplus / as sysdba
root 11584 8306 0 22:59:28 pts/2 0:00 grep 14313
oracle 2768 14313 0 Jul 23 pts/4 0:00 /sbin/sh
mcmsdbl1@root@/>#ps -ef | grep 14194
oracle 14194 1 0 Jul 22 pts/4 9:56 /usr/bin/vmstat -d -n 1
root 11587 8306 0 22:59:40 pts/2 0:00 grep 14194
mcmsdbl1@root@/>#ps -ef | grep 14107
oracle 14313 14107 0 Jul 22 pts/4 0:00 sqlplus / as sysdba
root 11617 8306 0 23:00:00 pts/2 0:00 grep 14107
oracle 14107 9275 0 Jul 22 pts/4 0:00 /sbin/sh
mcmsdbl1@root@/>#

 

still i see the below message continously going on

 

vxfs: msgcnt 36206 mesg 001: V-2-1: vx_nospace - /dev/vg00/lvol4 file system full (2 block extent)

 

server started giving slow response now.

sbrews
Frequent Advisor

Re: Urgent Help Please. /tmp 100% full

Of all the processes listed, I would look at the following:

 

mcmsdbl1@root@/>#ps -ef | grep 14194
oracle 14194 1 0 Jul 22 pts/4 9:56 /usr/bin/vmstat -d -n 1

 

it isrunning vmstat and circumstantial evidence (the process name) relates it to hp_perf_vmstat AND it appears to have been running since July 22.  Since its being run under the oracle ID, you should contact your DBAs and see what they can tell you about it.

sbrews
Frequent Advisor

Re: Urgent Help Please. /tmp 100% full

on a related note, how big is your /tmp mount point?

Raoof
Regular Advisor

Re: Urgent Help Please. /tmp 100% full

Hey sbrews,

 

Thanks a ton for your great assistance to get my issue resolve. After killing that process it releases the space, now the current status of /tmp is as below.

 

/dev/vg00/lvol4     524288   98688  423416   19% /tmp

 

about the size of mb is as above in kb.

 

Thanks & Best Regards

Raoof
Regular Advisor

Re: Urgent Help Please. /tmp 100% full

FYI, Below the current file system layout.

 

File System layout

LVM Device file mount point size fs type
/dev/vg00:
/dev/vg00/lvol1 /stand 1792 vxfs
/dev/vg00/lvol2 swap 8192
/dev/vg00/lvol3 / 1024 vxfs
/dev/vg00/lvol4 /tmp 2048 vxfs
/dev/vg00/lvol5 /home 128 vxfs
/dev/vg00/lvol6 /opt 16448 vxfs
/dev/vg00/lvol7 /usr 5824 vxfs
/dev/vg00/lvol8 /var 16384 vxfs
/dev/vg00/lvo19 /u01 61440 vxfs
/dev/vg00/lvo20 /backup 102400vxfs
/dev/vg00/lvo22 57344 unused
/dev/vg00 unallocated 6016

 

If possible can i insrease the size of /tmp by using some space from the unallocated size i.e. of around 5 to 6gb.

sbrews
Frequent Advisor

Re: Urgent Help Please. /tmp 100% full

your /tmp is only 512M in size, you should look into making it larger.  Overall size depends on how your system is being used.  I tend to go with 2G as a minimum and depending on system use, I may make it larger.

 

 

sbrews
Frequent Advisor

Re: Urgent Help Please. /tmp 100% full

Check this page - it has some examples of adding extending file systems, etc.

 

http://www.unixguide.net/hp/hplvmtasks.shtml

 

Also, if you google for hpux file system resizing, you should find lots of other examples.

Raoof
Regular Advisor

Re: Urgent Help Please. /tmp 100% full

Thanks for the information!!!

Dennis Handly
Acclaimed Contributor

Re: Urgent Help Please. /tmp 100% full

>how big is your /tmp mount point?

 

Also, how big is /var/tmp?  That's what should be used instead of /tmp, if on a different filesystem.