unable to find what is filling up /tmp

[NDO]

Hi all!

It seems rather easy problem to solve, but I am struglling to find out what is wrong, please can you help:

I have a server running 11.23 in which:

 #bdf /tmp
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol8    1048576 1012592   35984   97% /tmp

 But when I search for big files in it:

find /tmp -xdev -size +10000c -exec ls -lrt {} \;
-rwx------   1 root       sys          11914 Feb 17 11:09 /tmp/fr/vconf11
-r-xr-xr-x   1 root       sys          98152 Jun 29  2011 /tmp/fr/evainfo_hpux11_23_IA
-rw-------   1 root       sys          35959 Jun 20  2011 /tmp/mana06178
mceldev[721]/ #

#du -kx /tmp | sort -rn | head -30
496     /tmp
272     /tmp/fr
8       /tmp/backup
8       /tmp/PrinterSoftwt
8       /tmp/.oracle
0       /tmp/lost+found
0       /tmp/ign_configure
0       /tmp/.AgentSockets
mceldev[722]/ #

 and :

/tmp #du -sk *
0       67MQLuQXL5
0       ACL.log
8       AUTO.err
8       Core-sd.log
8       PrinterSoftwt
8       backup
8       cc-sd.log
8       da-sd.log
0       dhcpfifo.any
0       dhcpfifo.root
8       docs-sd.log
272     fr
0       ign_configure
0       ipv6agt.crashlog
8       javagui-sd.log
0       llbdbase.dat
0       lost+found
8       ma-sd.log
40      mana06178
8       media_info
8       media_info.tmp
8       mover_files
8       portmap.file
8       rpcbind.file
8       sensor_info
16      set_8265-151211_145156_basenv.ksh
8       system
8       system.prev
8       tmp_cur_shel_8265-151211_145156.env
0       tomala_swap.txt
0       wd_action.dbg

 Pls can you help

FR

[Dennis Handly]

>It seems rather easy problem to solve

It appears someone created a large file then it was removed while still opened.

You need to use lsof to find it and the process that has it opened.

[Chirag Parikh]

Run "fuser -cu /tmp" to find the process id which is using /tmp file system and kill that processor ID. then check bdf for /tmp, it should be normal and actual usage.

Regards,

Chirag Parikh

[Patrick Wallek]

To find unlinked files that are still occupying space use:

# lsof +L1

[NDO]

I dont have at moment lsof, I am downloading it, but when I do fuser -cu /tmp, I got the following:

/tmp #fuser -cu /tmp
/tmp:     1387o(root)    1366o(root)    1377o(root)    1389o(root)    1416o(root)   22320o(root)   16842o(root)   10474o(root)   11626o(root)   12057o(root)   10340o(root)   14448o(root)   12535o(root)   13737o(root)   10206o(root)   10660o(root)   10956o(root)   11264o(root)   13130o(root)   15175o(root)   15954o(root)   18712o(root)   19735o(root)   17746o(root)   20809o(root)   24511o(root)   21992o(root)   23192o(root)   27204o(root)   25832o(root)    7028o(root)   28684o(root)     284o(root)    3570o(root)    1907o(root)    5252o(root)   14631o(root)   16717o(root)    8829o(root)    2935o(root)   12657o(root)   10728o(root)   18815o(root)   20964o(root)   27811o(root)   23172o(root)   25436o(root)   13414o(root)     306o(root)   16176o(root)    5469o(root)    8033o(root)   10710o(root)   27819o(root)   18991o(root)   21854o(root)    7392o(root)    4215o(root)   24827o(root)   13967o(root)    1022o(root)    5384o(root)   17315o(root)   10668o(root)   24187o(root)   20725o(root)   27753o(root)   12872o(root)   20631o(root)    1582o(root)    9094o(root)   16745o(root)   12522o(root)   24578o(root)    2844o(root)   28584o(root)   11235o(root)    3339o(root)   24167o(root)    7037o(root)   19806o(root)   15491o(root)    7908o(root)   28637o(root)   17194o(root)   21960o(root)   11786o(root)   26747o(root)    6813o(root)   27094o(root)   21955o(root)    1844o(root)   16867o(root)    2514o(root)   28261o(root)    7797o(root)   29537o(root)    5334o(root)   13183o(root)   18575o(root)   24026o(root)   10314o(root)   16703o(root)   11008o(root)   22434o(root)   22458o(root)   20895o(root)    4336o(root)   22290o(root)   22303o(root)   22427o(root)   22350o(root)   22413o(root)   22581o(root)   22193o(root)   22406o(root)   22686o(root)   22450o(root)   22614o(root)   22849o(root)   22499o(root)   22590o(root)   22719o(root)   22599o(root)   22940o(root)   22669o(root)   22696o(root)   22881o(root)   22703o(root)   22984o(root)   22858o(root)   22771o(root)   22866o(root)   23408o(root)   23157o(root)   22947o(root)   22959o(root)   22966o(root)   23386o(root)   23120o(root)   23038o(root)   23220o(root)   23244o(root)   23140o(root)   23213o(root)   23129o(root)   23233o(root)   23258o(root)   23682o(root)   23498o(root)   23308o(root)   23395o(root)   23512o(root)   23421o(root)   23669o(root)   23477o(root)   23523o(root)   24025o(root)   23539o(root)   23589o(root)   23695o(root)   23775o(root)   23760o(root)   24107o(root)   23708o(root)   23785o(root)   24035o(root)   23792o(root)   23807o(root)   23927o(root)   24051o(root)   24016o(root)   24121o(root)   24326o(root)   24138o(root)   24289o(root)   24131o(root)   24152o(root)   24437o(root)   24607o(root)   24208o(root)   24593o(root)   24300o(root)   24309o(root)   24412o(root)   24380o(root)   24397o(root)   24419o(root)   24693o(root)   24568o(root)   24580o(root)   24485o(root)   24859o(root)   24670o(root)   24660o(root)   24682o(root)   24707o(root)   24844o(root)   24757o(root)   24869o(root)   24937o(root)   24885o(root)   24961o(root)   24947o(root)   24976o(root)    6332c(root)    8286c(root)

 its a lot of stuff

FR

[Patrick Wallek]

Unless you want to check all of those processes, I recommend using LSOF.  I am not surprised that you  have lots of processes accessing /tmp. 

[NDO]

Hi

I have downloaded and install lsof, but its a lot of stuff : 194 entries:

/usr/local/bin #./lsof +aL1 /tmp | more
COMMAND   PID USER   FD   TYPE DEVICE  SIZE/OFF NLINK NODE NAME
sh        306 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       2844 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       2935 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       3339 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       3570 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh       4215 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       4336 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       5252 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh       5334 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       5384 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       5469 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       6813 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       7028 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh       7037 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       7392 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       7797 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       7908 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       8033 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       8829 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       9094 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      10206 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      10314 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh      10340 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      10474 root    1u   REG 64,0x8 345698572     0  301 /tmp (/dev/vg00/lvol8)
sh      10660 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      10668 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      10710 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      10728 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      10956 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      11008 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh      11235 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      11264 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      11626 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      11786 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh      12057 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)

 I did this:

#./lsof +aL1 /tmp | wc -l
194

 and if I do ps -ef | grep pid, it returns an entry that was in the crontab, but its no longer there

FR

[Dennis Handly]

>but it's a lot of stuff:

>sh 306 root 1u REG 64,0x8 345698744 0 301 /tmp (/dev/vg00/lvol8)

This appears to be the same large file each time, used by the shell.

>and if I do ps -ef | grep pid, it returns an entry that was in the crontab, but its no longer there

It doesn't matter if it's no longer in crontab, if it is still running.

What does this show:

ps -f -p 306 -p 2844 -p 2935 -p 11786 -p 12057

[NDO]

its the same stuff:

ps -f -p 306 -p 2844 -p 2935 -p 11786 -p 12057
     UID   PID  PPID  C    STIME TTY       TIME COMMAND
    root 12057     1  0  Apr 13  ?        78:22 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
    root  2935     1  0  Apr 13  ?        77:46 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
    root   306     1  0  Apr 13  ?        78:46 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
    root 11786     1  0  Apr 13  ?        76:41 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
mceldev[800]/usr/local/bin #

[Patrick Wallek]

Does this file exist: /tmp/fr/ping_date.txt

If not, that is probably your culprit.  Do you really need all of those processes running that are pinging the same IP address?  They have been running since April 13.  I would look and see if you can kill those processes.

[Dennis Handly]

>it's the same stuff:

So it's not very useful, kill them all!  You don't really need 194 processes pinging the same IP.  ;-)

Also, you really should be sending this info to a pipe so you can do data reduction on the fly.  Or only send errors when ping fails.

>Does this file exist: /tmp/fr/ping_date.txt?

That's the missing file, since it's so large.

>I would look and see if you can kill those processes.

I would just wave bye-bye to the unmutual script and kill them all.

You could check to see if all of them are the same IP and output file.

[NDO]

Hi

The /tmp/fr/ping_date.txt file does not exist, and I am not quite sure how can kill 194 processes at the same time, please can you give a hint?

FR

[Patrick Wallek]

You can try to script the process. Otherwise you will just have to go through the list and kill the appropriate processes.

[NDO]

does this:

ps -ef |awk '/pong/{print $2}' | xargs kill -15

will do the job?

[RAJD1]

Nandinho,

Try using lsof on /tmp with sorting SIZE column.

# lsof /tmp > lsof_tmp.out

# cat lsof_tmp.ou | sort -k 8 | more 

You will see the large size , the program using, try to check that program with the PID given. May be you need to bounce that program/application or to kill it .

Hth,

Raj D.

[Dennis Handly · Accepted solution]

>ps -ef | awk '/pong/{print $2}' | xargs kill -15;   will do the job?

Yes, unless you want to limit it to be safe:

ps -fu root | awk '/ping.*pong/ {print $2}' | xargs kill

>Try using lsof on /tmp with sorting SIZE column.

> cat lsof_tmp.out | sort -k 8

I count column 7.  Also no need to use an evil cat:

sort -k 7 lsof_tmp.out

[NDO]

Hi

Thank you Dennis, I used your command, but with a pipe, (I think you missed it)

ps -fu root | awk '/ping.*pong/ {print $2}'| xargs kill

It worked fine, problem solved.

Thank you

FR