Hidden files in Root filesystem?

[Chet Woods]

Hi all,
My root filesystem is currently at 92% full, and I can't seem to find what is filling it up. (It is the same size as 85 other servers that are around 40% full) Is there some way that I can find hidden files here. Or anywhere I can look that might be filling this up? (Running HP-UX 10.20)

Thanks in advance,
Chet

[Santosh Nair_1]

A couple of usual suspects are /etc and /dev, they should be fairly small...if not, then check for newly created/accessed files (ls -ltr /etc /dev). Other possibilities are filesystems that didn't get mounted and someone created files in these directories (effectively writing on the root filesystem).

You can do a ls -lad /* and compare that to bdf /* to find all the filesystem that are mounted. Eliminate those and then check the other directories.

Hope this helps.

-Santosh

[Alan Riggs]

du -sk /* | sort -n

This will show disk usage of directory roots in ascending order. Ignore any directories that are on separate filesystems.

[Sridhar Bhaskarla]

Use du -x / to find out all the files in only root file system and their sizes. It will help you determine the large files easily.

-Sri

[A. Clay Stephenson]

Hi Chet:

This is very commonly caused by 1) core files or 2) regular files that accidently got written to /dev. e.g. /dev/rmt/Om (letter 'O') rather than the device /dev/rmt/0m.

I like to use du -s to narrow down where to look. Man du for details.

The other technique is to use find to locate large files like this:

from /
find . -xdev -size +1000000c

This will find all finds > 1000000 bytes in size and not cross filesystem boundaries.

By the way, root should essentially not grow at all. Yolu need to nail this down.

Clay

[Lou Zirko_1]

As was mentioned checkout the /dev directory, especially the rmt subdirectory for a file there. It has happened more than once that someone mistypes a device name and ends up with a file in that directory instead.

Lou Zirko

[James R. Ferguson]

Hi Chet:

Another common problem which causes the root filesystem to fill up is an erroneous specification of a tape device file -- typing the letter 'o' instead of the number '0', as '/dev/rmt/om' instead of '/dev/rmt/0m'. Look for non-special files in /dev.

Regards!

...JRF...

[Darrell Allen]

Hi Chet,

If you can't readily tell which directory has the growing file, try this:

find / -type d -xdev | xargs du -sk >file1
sort -k2,2 file1 >sorted1

Wait awhile (30 minutes?)

find / -type d -xdev | xargs du -sk >file2
sort -k2,2 file2 >sorted2
join -j 2 sorted1 sorted2 >file3

file3 will be formatted as:
directory firstsize secondsize

Simply look for large discreprencies between the first and second sizes to narrow down to a directory. Then you can hopefully identify the file.

Darrell

[Mladen Despic]

Hi Chet,

Perhaps you are not seeing a "hidden" file or a directory whose name begins with '.' (the dot).

If you try this command:

du -axk / | sort -nr | more

you will see them all in a sorted list.

If you only want the list of (sub)directories,
try this:

du -xk / | sort -nr | more

Mladen

[Wodisch]

Hello Chet,

a very bad way of "hidden" files is caused by sysadmins working in single-user mode and writing to some directory which later is used as a mount-point. Then those files "below" the mount-point will not be visible (=hidden).
The only chance of seeing those (in addition to go single-user mode) could be using "ncheck" on the root file system and comparing the output to the list of "visible" files in the root file system...

Just my $0.02,
Wodisch

[Volker Borowski]

Hi Chet,

my tip:
/etc/utmp
/etc/wtmp (if not linked to /var!)

In addition to Wodisch assumption, keep in mind, that also NFS can be a way to write "under" a mounted directory !

If your mount order is incorrect, you might be trapped. See this:

# bdf | grep top
/dev/vg05/test 79701 10 71720 0% /top
localhost:/top 79696 8 71720 0% /nfs_mount
# # NOW THEY ARE THE SAME
# umount /nfs_mount
# umount /top
# mount localhost:/top /nfs_mount
# mount /dev/vg05/test /top
# bdf | grep top
localhost:/top 99664 43456 46240 48% /nfs_mount
/dev/vg05/test 79701 10 71720 0% /top
# # NOW THEY ARE DIFFRENT !!!!!

After the remount, the /nfs_mount filesystem writes in fact to the root-filesystem !

Hope this helps
Volker

[John Payne_2]

Chet,

I agree with Wodish that it may be files sitting under a mount point, but my theory is that it is caused by programmers, not Sys admins. :) We see this all the time, the application guys want a new file system, and someone throws a huge tar file in the mount point just before you mount the new filesystem. They seem to think that as soon as they see the directory, it is ready to go. If all else fails, try unmounting the filesystem, and look at the mount points to see if there is crap sitting there. (You may have to boot to single user mode to quiet the system down in order to unmount...)

John