unable to find what is filling up /tmp

[NDO]

Hi all!

It seems rather easy problem to solve, but I am struglling to find out what is wrong, please can you help:

I have a server running 11.23 in which:

 #bdf /tmp
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol8    1048576 1012592   35984   97% /tmp

 But when I search for big files in it:

find /tmp -xdev -size +10000c -exec ls -lrt {} \;
-rwx------   1 root       sys          11914 Feb 17 11:09 /tmp/fr/vconf11
-r-xr-xr-x   1 root       sys          98152 Jun 29  2011 /tmp/fr/evainfo_hpux11_23_IA
-rw-------   1 root       sys          35959 Jun 20  2011 /tmp/mana06178
mceldev[721]/ #

#du -kx /tmp | sort -rn | head -30
496     /tmp
272     /tmp/fr
8       /tmp/backup
8       /tmp/PrinterSoftwt
8       /tmp/.oracle
0       /tmp/lost+found
0       /tmp/ign_configure
0       /tmp/.AgentSockets
mceldev[722]/ #

 and :

/tmp #du -sk *
0       67MQLuQXL5
0       ACL.log
8       AUTO.err
8       Core-sd.log
8       PrinterSoftwt
8       backup
8       cc-sd.log
8       da-sd.log
0       dhcpfifo.any
0       dhcpfifo.root
8       docs-sd.log
272     fr
0       ign_configure
0       ipv6agt.crashlog
8       javagui-sd.log
0       llbdbase.dat
0       lost+found
8       ma-sd.log
40      mana06178
8       media_info
8       media_info.tmp
8       mover_files
8       portmap.file
8       rpcbind.file
8       sensor_info
16      set_8265-151211_145156_basenv.ksh
8       system
8       system.prev
8       tmp_cur_shel_8265-151211_145156.env
0       tomala_swap.txt
0       wd_action.dbg

 Pls can you help

FR

[Dennis Handly]

>It seems rather easy problem to solve

It appears someone created a large file then it was removed while still opened.

You need to use lsof to find it and the process that has it opened.

[Chirag Parikh]

Run "fuser -cu /tmp" to find the process id which is using /tmp file system and kill that processor ID. then check bdf for /tmp, it should be normal and actual usage.

Regards,

Chirag Parikh

[Patrick Wallek]

To find unlinked files that are still occupying space use:

# lsof +L1

[NDO]

I dont have at moment lsof, I am downloading it, but when I do fuser -cu /tmp, I got the following:

/tmp #fuser -cu /tmp
/tmp:     1387o(root)    1366o(root)    1377o(root)    1389o(root)    1416o(root)   22320o(root)   16842o(root)   10474o(root)   11626o(root)   12057o(root)   10340o(root)   14448o(root)   12535o(root)   13737o(root)   10206o(root)   10660o(root)   10956o(root)   11264o(root)   13130o(root)   15175o(root)   15954o(root)   18712o(root)   19735o(root)   17746o(root)   20809o(root)   24511o(root)   21992o(root)   23192o(root)   27204o(root)   25832o(root)    7028o(root)   28684o(root)     284o(root)    3570o(root)    1907o(root)    5252o(root)   14631o(root)   16717o(root)    8829o(root)    2935o(root)   12657o(root)   10728o(root)   18815o(root)   20964o(root)   27811o(root)   23172o(root)   25436o(root)   13414o(root)     306o(root)   16176o(root)    5469o(root)    8033o(root)   10710o(root)   27819o(root)   18991o(root)   21854o(root)    7392o(root)    4215o(root)   24827o(root)   13967o(root)    1022o(root)    5384o(root)   17315o(root)   10668o(root)   24187o(root)   20725o(root)   27753o(root)   12872o(root)   20631o(root)    1582o(root)    9094o(root)   16745o(root)   12522o(root)   24578o(root)    2844o(root)   28584o(root)   11235o(root)    3339o(root)   24167o(root)    7037o(root)   19806o(root)   15491o(root)    7908o(root)   28637o(root)   17194o(root)   21960o(root)   11786o(root)   26747o(root)    6813o(root)   27094o(root)   21955o(root)    1844o(root)   16867o(root)    2514o(root)   28261o(root)    7797o(root)   29537o(root)    5334o(root)   13183o(root)   18575o(root)   24026o(root)   10314o(root)   16703o(root)   11008o(root)   22434o(root)   22458o(root)   20895o(root)    4336o(root)   22290o(root)   22303o(root)   22427o(root)   22350o(root)   22413o(root)   22581o(root)   22193o(root)   22406o(root)   22686o(root)   22450o(root)   22614o(root)   22849o(root)   22499o(root)   22590o(root)   22719o(root)   22599o(root)   22940o(root)   22669o(root)   22696o(root)   22881o(root)   22703o(root)   22984o(root)   22858o(root)   22771o(root)   22866o(root)   23408o(root)   23157o(root)   22947o(root)   22959o(root)   22966o(root)   23386o(root)   23120o(root)   23038o(root)   23220o(root)   23244o(root)   23140o(root)   23213o(root)   23129o(root)   23233o(root)   23258o(root)   23682o(root)   23498o(root)   23308o(root)   23395o(root)   23512o(root)   23421o(root)   23669o(root)   23477o(root)   23523o(root)   24025o(root)   23539o(root)   23589o(root)   23695o(root)   23775o(root)   23760o(root)   24107o(root)   23708o(root)   23785o(root)   24035o(root)   23792o(root)   23807o(root)   23927o(root)   24051o(root)   24016o(root)   24121o(root)   24326o(root)   24138o(root)   24289o(root)   24131o(root)   24152o(root)   24437o(root)   24607o(root)   24208o(root)   24593o(root)   24300o(root)   24309o(root)   24412o(root)   24380o(root)   24397o(root)   24419o(root)   24693o(root)   24568o(root)   24580o(root)   24485o(root)   24859o(root)   24670o(root)   24660o(root)   24682o(root)   24707o(root)   24844o(root)   24757o(root)   24869o(root)   24937o(root)   24885o(root)   24961o(root)   24947o(root)   24976o(root)    6332c(root)    8286c(root)

 its a lot of stuff

FR

[Patrick Wallek]

Unless you want to check all of those processes, I recommend using LSOF.  I am not surprised that you  have lots of processes accessing /tmp. 

[NDO]

Hi

I have downloaded and install lsof, but its a lot of stuff : 194 entries:

/usr/local/bin #./lsof +aL1 /tmp | more
COMMAND   PID USER   FD   TYPE DEVICE  SIZE/OFF NLINK NODE NAME
sh        306 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       2844 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       2935 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       3339 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       3570 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh       4215 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       4336 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       5252 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh       5334 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       5384 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       5469 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       6813 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       7028 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh       7037 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       7392 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       7797 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       7908 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh       8033 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       8829 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh       9094 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      10206 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      10314 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh      10340 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      10474 root    1u   REG 64,0x8 345698572     0  301 /tmp (/dev/vg00/lvol8)
sh      10660 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      10668 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      10710 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      10728 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      10956 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      11008 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh      11235 root    1u   REG 64,0x8 345698744     0  301 /tmp (/dev/vg00/lvol8)
sh      11264 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      11626 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)
sh      11786 root    1u   REG 64,0x8 345698830     0  301 /tmp (/dev/vg00/lvol8)
sh      12057 root    1u   REG 64,0x8 345698658     0  301 /tmp (/dev/vg00/lvol8)

 I did this:

#./lsof +aL1 /tmp | wc -l
194

 and if I do ps -ef | grep pid, it returns an entry that was in the crontab, but its no longer there

FR

[Dennis Handly]

>but it's a lot of stuff:

>sh 306 root 1u REG 64,0x8 345698744 0 301 /tmp (/dev/vg00/lvol8)

This appears to be the same large file each time, used by the shell.

>and if I do ps -ef | grep pid, it returns an entry that was in the crontab, but its no longer there

It doesn't matter if it's no longer in crontab, if it is still running.

What does this show:

ps -f -p 306 -p 2844 -p 2935 -p 11786 -p 12057

[NDO]

its the same stuff:

ps -f -p 306 -p 2844 -p 2935 -p 11786 -p 12057
     UID   PID  PPID  C    STIME TTY       TIME COMMAND
    root 12057     1  0  Apr 13  ?        78:22 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
    root  2935     1  0  Apr 13  ?        77:46 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
    root   306     1  0  Apr 13  ?        78:46 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
    root 11786     1  0  Apr 13  ?        76:41 sh -c /usr/sbin/ping 192.168.0.85 | while read pong; do echo "$(date): $pong"; done > /tmp/fr/ping_date.txt
mceldev[800]/usr/local/bin #

[Patrick Wallek]

Does this file exist: /tmp/fr/ping_date.txt

If not, that is probably your culprit.  Do you really need all of those processes running that are pinging the same IP address?  They have been running since April 13.  I would look and see if you can kill those processes.