topic ROOT file system almost full in Operating System - HP-UX

ROOT file system almost full

Darrell Albee — Thu, 31 Jan 2019 15:17:32 GMT

We have a new RX2800 system with HP-UX 11i v3 and I noticed the root file system is at 93% full. We also have an RX2660 with 11i v3 and when I compare all the files/directories that make up the root file system with the RX2800 the RX2660 has more used space but the root file system on the RX2660 is only at 23% full. I can't seem to find what is filling up the root file system on our RX2800.

Any ideas or suggestions?

Re: ROOT file system almost full

Steven Schweda — Fri, 01 Feb 2019 05:06:41 GMT

> [...] when I compare all the files/directories that make up the root
> file system [...]

   "compare" how, exactly? How big are these file systems? "93%" of
not much might easily be smaller than "23%" of some big thing.

> [...] I can't seem to find what is filling up the root file system on
> our RX2800.

   "find" how, exactly?

   "lsof" might be useful if there's some large, open file.

      http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.89/

Re: ROOT file system almost full

Darrell Albee — Fri, 01 Feb 2019 18:50:46 GMT

By compare I mean I went through each directory in the mount point and did a "du -sk" to get the total size for the directory and in most cases the exact same directory on our RX2660 was the same size or slightly larger. On our RX2800 the current size of / is 1.37GB and on our RX2660 the current size of / is 1.04GB. The % used on our RX2800 is 93% and on our RX2660 the % used is 23%.

I tried rebooting in case something large was open but even after the reboot the / filesystem was still at 93% used.

I will download and try the "lsof" to see if it shows anything I am not seeing. Thanks for the suggestion!

Re: ROOT file system almost full

Patrick Wallek — Fri, 01 Feb 2019 19:37:16 GMT

How large are the root file systems?

If the '/' mount point on the rx2800 is 1.5 GB and you're using 1.37 GB, then your usage will be in the 90+% range.

If the '/' mount point on the rx2660 is 4.0 GB in size and you're using 1 GB, then you will be in the 20% range.

You cannot go just on % utilized. You have to look at how large the file system is versus how much is used.

If you can show the complete 'bdf' output from both servers that will tell us a whole lot more.

Re: ROOT file system almost full

Bill Hassell — Sun, 03 Feb 2019 06:47:24 GMT

When troubleshooting file space issues. it is always better to look for the big directories first.
This is the easiest way to find big directories:

# du -kx / | sort -rn | head -20

You can replace / with /tmp or /var to analyze space on other mountpoints.
The / directory is often misused by inexperienced root administrators.
It should never be the $HOME directory for root and should not be a default drop box for files. Use /var/tmp and /tmp.

Run the above command and paste the results. We can then look at what is occupying your / filesystem.

Re: ROOT file system almost full

Darrell Albee — Mon, 04 Feb 2019 18:58:46 GMT

Patrick, attached is a wordpad document showing the bdf command on both systems. The newer system (rx2800) has a larger root filesystem and yet the space used is 70% higher than our older system (rx2660).

I'm at a total loss here. We have had HP-UX systems here for over 20 years now and I have never run into this before.

Re: ROOT file system almost full

Darrell Albee — Mon, 04 Feb 2019 19:00:44 GMT

Thanks for that info Bill. I will run the "du" command as suggested and compare.

Re: ROOT file system almost full

Bill Hassell — Mon, 04 Feb 2019 19:40:55 GMT

Unfortunately, the newest incarnation of the forums prevents attaching virtually any file.

The output from bdf and du are small enough to copy/paste.
To retain the formatting, use the forum Insert Code tool </> to paste into.

Re: ROOT file system almost full

Bill Hassell — Mon, 04 Feb 2019 19:55:36 GMT

And once you find an unusually big or unrecognized directory, use this to sort the files by size:

# ls -la | sort -rnk5 | head -20

There may be a big file lurking in the / directory.

Re: ROOT file system almost full

Bertram Stoeckler — Tue, 05 Feb 2019 08:59:39 GMT

If you still cannot find files with the suggested commands, then those files maybe located under one of the mount-points. (e.g. /var, or /tmp), and not visible/accessible.

There are 2 ways to check/find them.

1. Boot into single-user mode.

In single-user-mode only the root-filesystem is mounted and you should check if there are files in /usr, /var, /tmp, /opt etc...

Note that in this mode you only have a limited set of commands that can be run as /usr is not mounted.

2. The second way does not require a reboot. You simply NFS-export the root-filesystem and then nfs-mount it as a loopback-nfs mount :

In the following example i assume that your system-hostname is "rx2800", if not change the hostname...

# mkdir /newroot

#share -o root=rx2800 /

#mount -o vers=3 rx2800:/ /newroot

Now you can "cd" into /newroot directory and search from there.

There should not be any files in /newroot/var, /newroot/tmp etc....

After the check you can use the following commands to umount/unshare:

umount /newroot

unshare /

rmdir /newroot

Hope that helps!