Something eating space in /usr

[Ranveer]

Dear All
i am facing a strange problem. one day i found my server(it is a BCV) was not allowing the telnet to my server.I had hard booted the server. After rebooting, bdf was showing /usr 100% full(in /usr 8232KB was available).

I deleted some files from /usr so that /usr was 97%.Sytem worked for 4 days and after that i was not able to telnet. Again i have freed 17544 KB. Now i am noticeing this that my spcae is decreasing and came to 14050KB
in last 15 hours.
i have checked my crontab. nothing is there.The /var/adm/syslog/syslog.log not showing any proble. Please help me.The follwing are the process running on server.
ps -eaf
# ps -eaf | more
UID PID PPID C STIME TTY TIME COMMAND
root 0 0 0 Sep 9 ? 0:14 swapper
root 8 0 0 Sep 9 ? 0:00 supsched
root 9 0 0 Sep 9 ? 0:00 strmem
root 10 0 0 Sep 9 ? 0:00 strweld
root 11 0 0 Sep 9 ? 0:00 strfreebd
root 2 0 0 Sep 9 ? 0:00 vhand
root 3 0 0 Sep 9 ? 0:22 statdaemon
root 4 0 0 Sep 9 ? 0:00 unhashdaemon
root 12 0 0 Sep 9 ? 0:00 ttisr
root 13 0 0 Sep 9 ? 0:00 ioconfigd
root 1 0 0 Sep 9 ? 0:00 init
root 19 0 0 Sep 9 ? 0:00 lvmkd
root 20 0 0 Sep 9 ? 0:00 lvmkd
root 21 0 0 Sep 9 ? 0:00 lvmkd
root 22 0 0 Sep 9 ? 0:00 lvmkd
root 23 0 0 Sep 9 ? 0:00 lvmkd
root 24 0 0 Sep 9 ? 0:00 lvmkd
root 25 0 0 Sep 9 ? 0:00 lvmschedd
root 26 0 0 Sep 9 ? 0:00 smpsched
root 27 0 0 Sep 9 ? 0:00 smpsched
root 28 0 0 Sep 9 ? 0:00 sblksched
root 29 0 0 Sep 9 ? 0:00 sblksched
root 1461 1 0 Sep 9 console 0:00 /usr/sbin/getty console console
root 673 1 0 Sep 9 ? 0:00 /usr/sbin/rpcbind
root 35 0 0 Sep 9 ? 0:37 vxfsd
root 726 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 464 1 0 Sep 9 ? 0:10 /usr/sbin/syncer
root 531 1 0 Sep 9 ? 0:00 /usr/sbin/syslogd -D
root 534 1 0 Sep 9 ? 0:00 /usr/sbin/ptydaemon
root 678 0 0 Sep 9 ? 0:00 nfskd
root 556 1 0 Sep 9 ? 0:00 /usr/lbin/nktl_daemon 0 0 0 0 0
1 -2
root 566 1 0 Sep 9 ? 0:00 /usr/lbin/ntl_reader 0 1 1 1 100
0 2 /var/adm/nettl /var/adm/con
root 567 566 0 Sep 9 ? 0:05 /usr/sbin/netfmt -C -F -f /var/a
dm/nettl.LOG000 -c /var/adm/con
root 693 1 0 Sep 9 ? 0:00 /usr/lib/netsvc/yp/ypbind
root 1074 1 0 Sep 9 ? 0:00 /usr/sbin/snmpdm
root 7485 786 0 13:07:35 ? 0:00 instl_bootd
root 711 1 0 Sep 9 ? 0:00 /usr/sbin/keyserv
root 727 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 728 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 729 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 730 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 731 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 732 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 733 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 734 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 735 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 736 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 737 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 738 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 739 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 740 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 741 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
# ps -eaf
UID PID PPID C STIME TTY TIME COMMAND
root 0 0 0 Sep 9 ? 0:14 swapper
root 8 0 0 Sep 9 ? 0:00 supsched
root 9 0 0 Sep 9 ? 0:00 strmem
root 10 0 0 Sep 9 ? 0:00 strweld
root 11 0 0 Sep 9 ? 0:00 strfreebd
root 2 0 0 Sep 9 ? 0:00 vhand
root 3 0 0 Sep 9 ? 0:23 statdaemon
root 4 0 0 Sep 9 ? 0:00 unhashdaemon
root 12 0 0 Sep 9 ? 0:00 ttisr
root 13 0 0 Sep 9 ? 0:00 ioconfigd
root 1 0 0 Sep 9 ? 0:00 init
root 19 0 0 Sep 9 ? 0:00 lvmkd
root 20 0 0 Sep 9 ? 0:00 lvmkd
root 21 0 0 Sep 9 ? 0:00 lvmkd
root 22 0 0 Sep 9 ? 0:00 lvmkd
root 23 0 0 Sep 9 ? 0:00 lvmkd
root 24 0 0 Sep 9 ? 0:00 lvmkd
root 25 0 0 Sep 9 ? 0:00 lvmschedd
root 26 0 0 Sep 9 ? 0:00 smpsched
root 27 0 0 Sep 9 ? 0:00 smpsched
root 28 0 0 Sep 9 ? 0:00 sblksched
root 29 0 0 Sep 9 ? 0:00 sblksched
root 1461 1 0 Sep 9 console 0:00 /usr/sbin/getty console console
root 673 1 0 Sep 9 ? 0:00 /usr/sbin/rpcbind
root 35 0 0 Sep 9 ? 0:37 vxfsd
root 726 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 464 1 0 Sep 9 ? 0:10 /usr/sbin/syncer
root 531 1 0 Sep 9 ? 0:00 /usr/sbin/syslogd -D
root 534 1 0 Sep 9 ? 0:00 /usr/sbin/ptydaemon
root 678 0 0 Sep 9 ? 0:00 nfskd
root 556 1 0 Sep 9 ? 0:00 /usr/lbin/nktl_daemon 0 0 0 0 0
1 -2
root 566 1 0 Sep 9 ? 0:00 /usr/lbin/ntl_reader 0 1 1 1 100
0 2 /var/adm/nettl /var/adm/con
root 567 566 0 Sep 9 ? 0:05 /usr/sbin/netfmt -C -F -f /var/a
dm/nettl.LOG000 -c /var/adm/con
root 693 1 0 Sep 9 ? 0:00 /usr/lib/netsvc/yp/ypbind
root 1074 1 0 Sep 9 ? 0:00 /usr/sbin/snmpdm
root 7485 786 0 13:07:35 ? 0:00 instl_bootd
root 711 1 0 Sep 9 ? 0:00 /usr/sbin/keyserv
root 727 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 728 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 729 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 730 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 731 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 732 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 733 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 734 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 735 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 736 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 737 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 738 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 739 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 740 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 741 1 0 Sep 9 ? 0:00 /usr/sbin/biod 16
root 756 1 0 Sep 9 ? 0:00 /usr/sbin/rpc.lockd
root 750 1 0 Sep 9 ? 0:00 /usr/sbin/rpc.statd
root 1095 1 0 Sep 9 ? 0:04 /usr/sbin/mib2agt
root 1086 1 0 Sep 9 ? 0:00 /usr/sbin/hp_unixagt
root 786 1 0 Sep 9 ? 0:01 /usr/sbin/inetd
root 1104 1 0 Sep 9 ? 0:00 /usr/sbin/trapdestagt
root 1127 1 0 Sep 9 ? 0:00 /usr/sbin/fddi4subagt
root 1193 1 0 Sep 9 ? 0:02 scrdaemon
root 1155 1 0 Sep 9 ? 0:03 /usr/dmi/bin/dmisp
root 1141 1 0 Sep 9 ? 0:02 /opt/dce/sbin/rpcd
root 1164 1 0 Sep 9 ? 0:00 /var/dmi/bin/hpuxci
root 2807 1269 0 Sep 9 ? 0:11 cclogd
root 1186 1 0 Sep 9 ? 0:00 /var/dmi/bin/swci
root 1217 1 0 Sep 9 ? 0:04 /usr/sbin/xntpd
root 7454 7453 0 13:02:06 pts/tb 0:00 -sh
root 1202 1 0 Sep 9 ? 0:00 /usr/sbin/rbootd
root 1227 1 0 Sep 9 ? 0:02 /usr/sbin/pwgrd
root 2906 1 0 Sep 9 ? 0:00 /usr/sbin/stm/uut/bin/tools/moni
tor/dm_FCMS_adapter
root 1272 1 0 Sep 9 ? 0:00 /usr/sbin/envd
root 2816 786 0 Sep 9 ? 0:00 /etc/opt/resmon/lbin/registrar
root 1241 1 0 Sep 9 ? 0:00 /usr/sbin/cron
root 2808 1269 0 Sep 9 ? 0:00 diaglogd
root 1289 1 0 Sep 9 ? 0:01 /usr/sbin/swagentd -r
root 1463 1 0 Sep 9 ? 0:00 /sbin/sfd
root 1269 1 0 Sep 9 ? 7:31 /usr/sbin/stm/uut/bin/sys/diagmo
nd
root 2809 1269 0 Sep 9 ? 0:00 memlogd
root 1303 1 0 Sep 9 ? 0:00 /etc/opt/resmon/lbin/emsagent
root 1388 1 0 Sep 9 ? 0:00 /usr/ecc/exec/mstragent
root 2864 1 0 Sep 9 ? 0:00 /usr/sbin/stm/uut/bin/tools/moni
tor/disk_em
root 1373 1 0 Sep 9 ? 0:04 /opt/VRTSob/bin/vxsvc -r /etc/vx
/isis/Registry
root 7550 7454 1 13:20:38 pts/tb 0:00 ps -eaf
root 1464 1 0 Sep 9 ? 0:00 /usr/sam/lbin/samd
root 1457 1 0 Sep 9 ? 0:00 /sbin/sh /usr/dt/bin/dtrc
root 1462 1 0 Sep 9 ? 0:00 /sbin/krsd -i
root 1456 1388 0 Sep 9 ? 0:28 /usr/ecc/exec/mstragent -s
root 1473 1 0 Sep 9 ? 0:03 /etc/opt/resmon/lbin/p_client
root 3311 1 0 Sep 9 ? 0:00 /usr/sbin/stm/uut/bin/tools/moni
tor/sysstat_em
root 1508 1457 0 Sep 9 ? 0:00 /usr/dt/bin/dtlogin
root 3053 1 0 Sep 9 ? 0:06 /usr/sbin/stm/uut/bin/tools/moni
tor/dm_core_hw
root 2810 1269 0 Sep 9 ? 0:34 psmctd
root 3140 1 0 Sep 9 ? 0:01 /usr/sbin/stm/uut/bin/tools/moni
tor/dm_stape
root 3104 1 0 Sep 9 ? 0:00 /usr/sbin/stm/uut/bin/tools/moni
tor/dm_memory
root 2936 1 0 Sep 9 ? 0:00 /usr/sbin/stm/uut/bin/tools/moni
tor/dm_TL_adapter
root 7453 786 0 13:02:06 pts/tb 0:00 telnetd
root 3272 1 0 Sep 9 ? 0:12 /usr/sbin/stm/uut/bin/tools/moni
tor/lpmc_em
root 7371 7370 0 12:29:12 pts/ta 0:00 -sh
root 7370 786 0 12:29:12 pts/ta 0:00 telnetd

Thanks & Regds
Ranveer

[morganelan]

Hi,
Try to use this command to find out what files that grow up fast.
#du -akx /usr|sort -nr|more
This commans will sort from greatest size of file to smallest size of file.

[Devender Khatana]

Hi,

The contents of /usr are normally static only and should not change. In such event if some open process is using space from /usr then you should be able to find that using lsof.

Also sort the contents as suggested above to find some unwanted files. As in any case total used space should not be more that 80-85% if enough space has been provided for the file system initially. If enough space is not ptovided then you should increase file system size.

HTH,
Devender

[Raj D.]

Hi Ranveer ,

/usr should not grow by itself unless any new package or any sw installs.

You can check for any bigger files if user is putting up there :

# ls -lR | sort +4 -5nr | more

or

# cd /usr
# du -sk * [ And can be monitor which is eating up space ]

Cheers,
Raj.

[Ranveer]

Hi morganelan
The command is showing the files. but how to find the culprit?
Dear Khatana
Can u explain little more on lsof?
i have downloaded it but confused how to get culprit process?

Thanks & regds
Ranveer

[morganelan]

Could you please post the result of du -akx|sort nr|more that I have given to you before?

[morganelan · Accepted solution]

To list all open files, use: #lsof

When -f is specified, all path name arguments will be taken to be simple files. Thus, for example, the ``-f /'' arguments direct lsof to search for open files with a `/' path name, not all open files in the `/' (root) file system.
Be careful to make sure +f is properly terminated and isn't followed by a character (e.g., of the file or file system name) that might be taken as a parameter. For example, use ``--'' after +f as in this example.

#lsof +f -- /file/system/name

To list all open Internet, x.25 (HP-UX), and UNIX domain files, use: #lsof -i -U

To list all open IPv4 network files in use by the process whose PID is 1234, use:
#lsof -i 4 -a -p 1234

Presuming the UNIX dialect supports IPv6, to list only open IPv6 network files, use:
#lsof -i 6

To list all files using any protocol on ports 513, 514, or 515 of host wonderland.cc.purdue.edu, use:
#lsof -i @wonderland.cc.purdue.edu:513-515

To list all files using any protocol on any port of mace.cc.purdue.edu (cc.purdue.edu is the default domain), use:
#lsof -i @mace

To list all open files for login name ``abe'', or user ID 1234, or process 456, or process 123, or process 789, use:
#lsof -p 456,123,789 -u 1234,abe

To list all open files on device /dev/hd4, use: #lsof /dev/hd4

To find the process that has /u/abe/foo open, use: #lsof /u/abe/foo

To send a SIGHUP to the processes that have /u/abe/bar open, use:
#kill -HUP `lsof -t /u/abe/bar`

To find any open file, including an open UNIX domain socket file, with the name /dev/log, use: #lsof /dev/log

To find processes with open files on the NFS file system named /nfs/mount/point whose server is inaccessible, and presuming your mount table supplies the device number for /nfs/mount/point, use:
#lsof -b /nfs/mount/point

To do the preceding search with warning messages suppressed, use:
#lsof -bw /nfs/mount/point

To ignore the device cache file, use:
#lsof -Di

To obtain PID and command name field output for each process, file descriptor, file device number, and file inode number for each file of each process, use:#lsof -FpcfDi

To list the files at descriptors 1 and 3 of every process running the lsof command for login ID ``abe'' every 10 seconds, use:
#lsof -c lsof -a -d 1 -d 3 -u abe -r10

To list the current working directory of processes running a command that is exactly four characters long and has an 'o' or 'O' in character three, use this regular expression form of the -c c option:
#lsof -c /^..o.$/i -a -d cwd

To find an IP version 4 socket file by its associated numeric dot-form address, use:
#lsof -i@128.210.15.17

To find an IP version 6 socket file (when the UNIX dialect supports IPv6) by its associated numeric colon-form address, use:
#lsof -i@[0:1:2:3:4:5:6:7]

To find an IP version 6 socket file (when the UNIX dialect supports IPv6) by an associated numeric colon-form address that has a run of zeroes in it - e.g., the loop-back address - use: #lsof -i@[::1]

[Bill Hassell]

Before we can help, you need to post the results of the du command:

du /usr -kx | sort -rn | head -20

This list shows the biggest directories (not files). This is a fairly normal listing:

822462 /usr
259032 /usr/lib
158704 /usr/sbin
114426 /usr/sbin/stm
90078 /usr/lib/nls
89724 /usr/sbin/stm/uut
82646 /usr/lib/nls/loc
67676 /usr/share
65494 /usr/lib/nls/loc/locales.1
53612 /usr/conf
50004 /usr/share/man
49382 /usr/dt
46408 /usr/sbin/stm/uut/lib
43316 /usr/sbin/stm/uut/bin
43116 /usr/bin

Now, if you find a directory that is in your listing but not in the one above, that is likely the problem. A high probability is /usr/local which often has bad permissions. To find the big files, do NOT use find! Instead, cd to the big directory and sort the files by size:

cd /usr/someBIGdirectory
ll | sort -rnk5 | head -20

Now if there are no big files, they may be in a subdirectory. It will help if you can post the results of your du command as soon as possible.

[Ranveer]

Dear All
very very thanks for your great cooperation.
there was a core file in /usr/ecc of 186MB. file shifted.

[Raj D.]

Hi Ranveer ,

You can delete the core file as it generates due to program fault , error or abnormal termination.

You can check it by :
# file core_filename [ and file type will show core ]

Hope your problem solved .
Now cheer up and you can assign the points.

"I have assigned points to 2 of 15 responses to my questions. "

Try to make it 15 of 15.

Enjoy,
HTH,
Raj.