ROOT file system almost full

[Darrell Albee]

We have a new RX2800 system with HP-UX 11i v3 and I noticed the root file system is at 93% full.  We also have an RX2660 with 11i v3 and when I compare all the files/directories that make up the root file system with the RX2800 the RX2660 has more used space but the root file system on the RX2660 is only at 23% full. I can't seem to find what is filling up the root file system on our RX2800.

Any ideas or suggestions?

[Steven Schweda]

> [...] when I compare all the files/directories that make up the root
> file system [...]

   "compare" how, exactly?  How big are these file systems?  "93%" of
not much might easily be smaller than "23%" of some big thing.

> [...] I can't seem to find what is filling up the root file system on
> our RX2800.

   "find" how, exactly?

   "lsof" might be useful if there's some large, open file.

      http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.89/

[Darrell Albee]

By compare I mean I went through each directory in the mount point and did a "du -sk" to get the total size for the directory and in most cases the exact same directory on our RX2660 was the same size or slightly larger. On our RX2800 the current size of / is 1.37GB and on our RX2660 the current size of / is 1.04GB. The % used on our RX2800 is 93% and on our RX2660 the % used is 23%.

I tried rebooting in case something large was open but even after the reboot the / filesystem was still at 93% used.

I will download and try the "lsof" to see if it shows anything I am not seeing. Thanks for the suggestion!

[Patrick Wallek]

How large are the root file systems?

If the '/' mount point on the rx2800 is 1.5 GB and you're using 1.37 GB, then your usage will be in the 90+% range.

If the '/' mount point on the rx2660  is 4.0 GB in size and you're using 1 GB, then you will be in the 20% range.

You cannot go just on % utilized.  You have to look at how large the file system is versus how much is used.

If you can show the complete 'bdf' output from both servers that will tell us a whole lot more.

[Bill Hassell]

When troubleshooting file space issues. it is always better to look for the big directories first.
This is the easiest way to find big directories:

# du -kx / | sort -rn | head -20

You can replace / with /tmp or /var to analyze space on other mountpoints.
The / directory is often misused by inexperienced root administrators.
It should never be the $HOME directory for root and should not be a default drop box for files. Use /var/tmp and /tmp.

Run the above command and paste the results. We can then look at what is occupying your / filesystem.

[Darrell Albee]

Patrick, attached is a wordpad document showing the bdf command on both systems. The newer system (rx2800) has a larger root filesystem and yet the space used is 70% higher than our older system (rx2660).

I'm at a total loss here. We have had HP-UX systems here for over 20 years now and I have never run into this before.

[Darrell Albee]

Thanks for that info Bill. I will run the "du" command as suggested and compare.

[Bill Hassell]

Unfortunately, the newest incarnation of the forums prevents attaching virtually any file.

The output from bdf and du are small enough to copy/paste.
To retain the formatting, use the forum Insert Code tool </> to paste into.

[Bill Hassell]

And once you find an unusually big or unrecognized directory, use this to sort the files by size:

# ls -la | sort -rnk5 | head -20

There may be a big file lurking in the / directory.

[Bertram Stoeckler]

If you still cannot find files with the suggested commands, then those files maybe located under one of the mount-points. (e.g. /var, or /tmp), and not visible/accessible.

There are 2 ways to check/find them.

1. Boot into single-user mode.

In single-user-mode only the root-filesystem is mounted and  you should check if there are files in /usr, /var, /tmp, /opt etc...

Note that in this mode you only have a limited set of commands that can be run as /usr is not mounted.

2. The second way does not require a reboot. You simply NFS-export the root-filesystem and then nfs-mount it as a loopback-nfs mount :

In the following example i assume that your system-hostname is "rx2800", if not change the hostname...

# mkdir /newroot

#share -o root=rx2800 /

#mount -o vers=3 rx2800:/ /newroot

Now you can "cd" into /newroot directory and search from there.  

There should not be any files in /newroot/var, /newroot/tmp etc....

After the check you can use the following commands to umount/unshare:

umount /newroot

unshare /

rmdir /newroot

Hope that helps!