Disk space usage in the root directory

[Andrew Kaplan]

Hi there --

One of our servers, an rp3440 system running HP-UX 11.11 in a ServiceGuard cluster, has its root directory running low on free space. The server in question acts as the Master NIS, as well as a Samba, and Postgres server.

I started investigating the matter by using the command syntax: du -x / and piping the output to a file. After the file was created, I checked things out, but I did not see anything that was glaringly obvious as a disk space hog. I have included the output file with this posting.

Am I using the wrong command syntax, or the wrong command to determine what file(s) and/or directories in the root folder are taking up the most space? Thanks.

[Shibin_2]

Command du -x only returns the size of directories.

Have you run a find command for core files in / ?

Or else you can use lsof command to determine what all processes holding your space.

[Andrew Kaplan]

Hi there --

Thanks for your reply. I ran the find command in the root directory using the following command syntax:

find . -name core -print

The results were the following:

./etc/core
./opt/hpws/apache/php/bin/core
./opt/omni/gui/help/C/help_topics/core
./usr/local/include/c++/4.2.3/gnu/java/net/protocol/core
./var/opt/wbem/core

When I checked out the above locations, here is what I found:

# cd /etc
# ll |grep core
-rw------- 1 root sys 337308 Sep 28 2008 core
lrwxrwxrwt 1 root sys 14 Apr 14 2006 savecore -> /sbin/savecore
# cd /opt/hpws/apache/php/bin/
# ll |grep core
-rw------- 1 root sys 1328764 Mar 13 2008 core
# cd /opt/omni/gui/help/C/help_topics/
# ll |grep core
drwxr-x--- 3 root sys 96 May 9 2008 core
# cd /usr/local/include/c++/4.2.3/gnu/java/net/protocol/
# ll |grep core
drwxr-xr-x 2 root sys 96 Feb 24 2009 core
# cd /var/opt/wbem/
# ll |grep core
-rw------- 1 root sys 13285244 Nov 3 07:59 core

It appears the core files in the /etc, /opt/hpws/apache/php/bin, and /var/opt/wbem directories are 'true' core files, and can be safely deleted, while those in the other two locations are directories, and should not be touched.

Do you agree?

[Shibin_2]

Yes, you are correct. Delete those core files.

[Andrew Kaplan]

Hi there --

I removed the core files in question, but the amount of free space generated did not amount to very much. I then did a search for lsof, but it is not installed on the system.

Aside from installing lsof, what other steps can I take here?

Thanks.

[Shibin_2]

First you determine, what is causing the utilization.

You can have a comparison of output from du -sk / and df -k / commands.

Here you can see, whether it is really holding by processes or it is real usage.

Another options, /etc/lvmconf directory often keeps old VG conf files. e.g. vg00.conf.old. You can remove those old files. It will give you considerable space.

[James R. Ferguson]

Hi Andrew:

For future use:

# find / -xdev -type f -name core

...will search for files (only) named "core" in the root filesystem ('/', '/etc', '/sbin') but *not* mounted directories like '/var' and '/usr'. This is controlled by 'find's '-xdev'.

For evaluating the *directories* consuming the most space, yse:

# cd / && du -xk | sort -rnk1,1

...which causes 'du' to act like 'find's '-xdev'. The sort orders the *sizes* in descending order to speed your analysis.

Regards!

...JRF...
Regards!

...JRF...

[bullz]

Hi

Alway use the below command to find the files/directory which are consuming more disk space.

#du -akx | sort -nr | more

Also, you need to be careful to the check the current utilization and output of df -k

Sometimes because of the open files, df -k output will show the wrong info.

If everything Ok, you many be delete/move the most consuming data from one disk to another.

[DeafFrog]

Hi Andrew ,

In addition to lvmconf and core alredy mentioned above ,
- In /var/tmp, anything older than the current system uptime can usually be safely deleted.
- Look into sub-directories of /var/spool/:
-- if there are many files in /var/spool/mqueue, your system has/had a problem in sending email messages to other systems, *and* you probably don't have a sendmail daemon running.

Reg ,

[Bill Hassell]

Big files are tempting targets but always start with directories. Use the above example:

du -kx / | sort -rn | head -20

A proper root file system has only two big diretories: /etc and /sbin. NOTHING else. Anything bigger than /etc and /sbin is misplaced and must be moved. The root filesystem is essentially static and should not change except due to mistakes. /dev should be just 1 to 3 megs. Anything more and there are junk files in there. Use this command to find the junk:

find /dev/ -type f -exec ll {} \;

The first step to cleanup is root's home. It should NEVER be in / (despite the default location setup by HP-UX, Linux, Solaris, etc). It is the most dangerous directory for a root user to be in. Create a new directory called /root and move /.profile to /root. Then edit /etc/passwd to change :/: to :/root: and then start a new window logging as root.

Now look at the garbage in /. There should never be any ordinary files in /. It is the the location for /etc, /sbin, /root, and /dev plus mounpoints (empty directories to mount things like /usr, /opt, /var...). Now list the junk files:

ll -dt /*/

The oldest files are at the bottom of the list. Like all file maintenance, the sysadmin must examine each file and determine whether it can be removed or moved.

Here are typical directory sizes for 10.20 through 11.31:

10.20
Dir-Size Directory name
20.0 MB /
14.7 MB /sbin
4.3 MB /etc
3.7 MB /sbin/fs
2.0 MB /etc/opt
1.9 MB /etc/opt/resmon
1.3 MB /sbin/fs/vxfs
1.1 MB /sbin/fs/hfs

11.00
Dir-Size Directory name
36.3 MB /
23.8 MB /sbin
11.0 MB /etc
8.1 MB /sbin/fs
7.9 MB /etc/opt
6.3 MB /etc/opt/resmon
4.3 MB /sbin/fs/vxfs3.3
3.4 MB /etc/opt/resmon/lbin
1.8 MB /sbin/fs/vxfs

11.11
Dir-Size Directory name
184.5 MB /
73.9 MB /etc
64.0 MB /etc/vx
41.6 MB /sbin
33.5 MB /etc/vx/type
29.5 MB /etc/vx/static.d
28.8 MB /etc/vx/static.d/build
14.9 MB /etc/vx/type/static
14.7 MB /root
12.8 MB /root/.swa
12.7 MB /root/.swa/cache
9.2 MB /etc/vx/type/gen
6.9 MB /etc/vx/type/raid5
5.9 MB /sbin/fs
5.2 MB /etc/opt

11.23
Dir-Size Directory name
0.3 GB /
0.2 GB /etc
0.1 GB /etc/vx
0.1 GB /etc/vx/type
98.7 MB /sbin
65.5 MB /etc/vx/static.d
62.5 MB /etc/vx/static.d/build
60.1 MB /root
42.5 MB /etc/vx/type/static
42.1 MB /root/.cpan
28.1 MB /etc/vx/type/gen
25.8 MB /root/.cpan/build
22.4 MB /etc/opt
21.4 MB /etc/vx/type/raid5
17.3 MB /sbin/fs
14.1 MB /root/.swa
13.9 MB /etc/vx/type/fsgen
13.8 MB /root/.swa/cache

11.31 PARISC
Dir-Size Directory name
0.3 GB /
0.2 GB /etc
0.1 GB /etc/vx
0.1 GB /sbin
0.1 GB /etc/vx/type
67.7 MB /etc/vx/static.d
64.2 MB /etc/vx/static.d/build
44.8 MB /etc/vx/type/static
29.0 MB /sbin/fs
28.5 MB /etc/vx/type/gen
26.2 MB /etc/opt
23.2 MB /sbin/fs/vxfs
21.8 MB /etc/vx/type/raid5
21.7 MB /etc/opt/resmon
16.9 MB /root
14.1 MB /etc/vx/type/fsgen
12.9 MB /root/.swa
12.8 MB /root/.swa/cache
10.2 MB /etc/opt/resmon/lib

Dir-Size Directory name
0.3 GB /
0.1 GB /etc
0.1 GB /sbin
0.1 GB /etc/vx
88.0 MB /etc/vx/type
42.1 MB /etc/opt
38.4 MB /etc/vx/type/static
37.6 MB /etc/opt/resmon
37.5 MB /sbin/fs
27.3 MB /sbin/fs/vxfs
23.9 MB /etc/opt/resmon/lib
22.1 MB /etc/vx/type/gen
21.0 MB /root
16.7 MB /etc/vx/type/raid5
16.2 MB /root/.swa
15.9 MB /root/.swa/cache
12.8 MB /etc/vx/static.d
12.1 MB /etc/vx/static.d/build
11.3 MB /etc/opt/resmon/lbin

(FYI, the script to list the directories by size is attached)

[Andrew Kaplan]

Hi there --

I ran the script posted by Bill Hassell, thanks Bill, and the output indicated the following directories were taking up the most space in the root directory:

64056 ./etc/vx
33512 ./etc/vx/type
29512 ./etc/vx/static.d
28800 ./etc/vx/static.d/build

The /etc directory itself is about 120 megabytes which contrasts with the typical size of seventy-four megabytes in a standard HP-UX 11.11 system. Thoughts and suggestions are welcome.

The vx directories appear to be approximately the default size as stated the previous posting, so unless I am wrong there must be something directly under /etc that is taking up an inordinate amount of space.

I ran the ls -lt command, and piped the results to a file. As of this posting, I am still going through the file, but I have included here.

[Bill Hassell]

> 64056 ./etc/vx

This is normal for the vx directory. Can you post the start of the / listing? /etc/vx may be big but we need to see /sbin and make sure most of the space for /sbin + /etc add up to 120 MB. If not, then there are other directories contributing to the usage or there are large files in /etc. The largest files in /etc should be less than 1MB (typically less than 100KB:

-rw-r--r-- 1 root root 103.6 Kb Nov 17 15:14 rc.log.old
-rw-r--r-- 1 root root 91.2 Kb Nov 17 15:19 rc.log
-r--r--r-- 1 bin bin 51.5 Kb Nov 14 2000 disktab
-rw-r--r-- 1 root sys 17.9 Kb Jul 18 2006 #ntp.conf
-r--r--r-- 1 bin bin 17.9 Kb Jul 18 2006 ntp.conf
-r--r--r-- 1 bin bin 17.7 Kb Nov 14 2000 ntp.conf.orig
-r--r--r-- 1 bin bin 17.5 Kb Nov 14 2000 cue.dm
-rw------- 1 root sys 13.1 Kb Dec 4 2008 lvmtab.today
-rw-r--r-- 1 bin bin 11.5 Kb Mar 18 2008 nettlgen.conf
-rw-r--r-- 1 root sys 11.4 Kb Mar 18 2008 oldnettlgen.conf
-rw-rw-r-- 1 root sys 11.2 Kb Nov 17 15:15 ioconfig

The companion script lls (long listing sorted) is attached.

[Andrew Kaplan]

Hi there --

The beginning of the / listing is shown below:

179320 .
119352 ./etc
64056 ./etc/vx
43040 ./sbin
33512 ./etc/vx/type
29512 ./etc/vx/static.d
28800 ./etc/vx/static.d/build/vold.o
28800 ./etc/vx/static.d/build
28512 ./etc/lvmconf
16160 ./etc/opt
14984 ./etc/vx/type/static
11976 ./etc/opt/resmon
9256 ./etc/vx/type/gen
8744 ./etc/opt/resmon/lbin
6944 ./etc/vx/type/raid5
5952 ./cosmos1.err
5928 ./sbin/fs
5184 ./etc/vx/type/static/vxconfigd
4392 ./etc/cmcluster
3880 ./dead.letter
3512 ./sbin/fs/vxfs
2936 ./etc/opt/resmon/lbin/rdbmsmond
2920 ./etc/opt/resmon/lbin/mibmond
2704 ./.mozilla
2688 ./.mozilla/default/15xeidf9.slt
2688 ./.mozilla/default

I also ran the attached script on the /sbin and / directories. The largest file size in the former was 1.5 megabytes, while that on the latter was 5.8 megabytes.