Root partition is 96% but unable to find any large file.

[Mohammad Kaleem]

Hi,
Root partition is 96% on a L-Class HP-UX 11.0 system. Only following files I can see on the system. Can I delet them?
Omansms2:root> pwd
/etc/opt/resmon/log
Omansms2:root> ls -al
total 17174
drwxrwxrwx 2 bin bin 1024 Jan 13 03:06 .
dr-xr-xr-x 10 bin bin 1024 Feb 2 14:57 ..
-rw-r--r-- 1 root root 371753 Jan 26 08:09 api.log
-rw-r--r-- 1 root root 500530 Nov 24 08:28 api.log.old
-rw-r--r-- 1 root root 667431 Feb 5 12:24 armmon.log
-rw-r--r-- 1 root root 1420124 Jan 13 02:23 armmon.log.old
-rw-r--r-- 1 root root 91578 Jan 13 03:07 client.log
-rw-r--r-- 1 root sys 500395 Nov 24 05:53 client.log.old
-rw-r--r-- 1 root sys 120880 Jan 13 03:04 emsagent.log
-rw-r--r-- 1 root root 667979 Feb 5 12:25 fc60mon.log
-rw-r--r-- 1 root root 1420903 Jan 13 02:23 fc60mon.log.old
-rw-r--r-- 1 root root 321204 Jan 13 03:09 registrar.log
-rw-r--r-- 1 root root 500031 Sep 22 2002 registrar.log.old
-rw-r--r-- 1 root root 861312 Jan 28 04:16 reslog.html
-rw-r--r-- 1 root root 1001735 Mar 20 2002 reslog.old.html

Best regards,
Mohammad

[Leif Halvarsson_2]

Hi,
It should at least not be any problem to tremove the .old logs. Instead of removing the active log you can truncate them with (for ex.):

>armmon.log

[Mohammad Kaleem]

Thanks,

Any other idea to see large file pm root partition. Unfortunitly lsof is not installed on the system and system is at remote.
Regards,
Mohammad

[Steve Lewis]

Have you tried checking /dev/rmt ?

A common error is a spelling mistake when specifying the backup device. This will start copying the whole backup to a new (real) file in that directory, filling up your root filesystem.

[MarkSyder]

Have you checked for large core dumps?

find / -name core -print

You can safely delete these.

You can check which directories are using the most space by:

du -k|sort -k1,1nr|pg

will give you the directories in size order, starting with the largest.

Mark Syder (like the drink but spelt different)

[Elmar P. Kolkman]

Don't forget the -xdev option when doing a find.

Also, check the sum of du -xk / and compare it with the used column of bdf. Probably your files have been deleted but are still open. In that case, without lsof, a reboot is the best remedy, because you need to stop the proces keeping the file open, which is very hard without lsof.

And of course, do a 'find /dev -type f' to find wrong entries in /dev, caused by typing errors or wrong scripts.
I once encountered a script that created a logfile by starting with removing it. The name of the logfile was in a variable. The developers then stopped looking at the logfile, so they changed the variable content to /dev/null. Result: /dev/null was removed and re-created as file. Which got VERY big...

[Hoefnix]

If you have mounted file systems mounted to a directory thats on the rootvolume and it also contains data in the root volume. It's hard to find out where all the usage come from, while the other filesystem is still mounted on it.

Try to find out if this is the case at your system (unmount file systems that are not from vg00 and have a special look at NFS filesystems)

Regards,
Peter

[Mohammad Kaleem]

Hi all,

I have tried every thing mentioned above but still couldn't find any clue. There is now any core, /dev/rmt also doesn't contain any large file.

I believe I have to schedule a system reboot?

Cheers,
Mohammad

[Clemens van Everdingen]

Hi,

Did you check in /var/stm/logs/os

Most of the times there are big files as well.

These are your diagnostics logs.
Just remove the old logs, be sure to leave the *.raw.cur file, because this is the current file.

If you want to remove this one, you will have to stop stm -> remove file and start stm.

Or just do a switch log from the stm menu.
Then remove the old file.

C.

[Elmar P. Kolkman]

Mohammed,

a reboot will only help if there are open files without i-node or if you have files that are removed/replaced during the boot process.

Have you looked at possible differences between du and bdf?

[Paula J Frazer-Campbell]

Hi

Use SAM and go to the log file option and look for large files. These can be trimmed in sam.

Paula

[Mohammad Kaleem]

Hi Clemens van,

/ partition is full not /var. Will it make difference if I will remove files from var?

Regards,
Mohammad

[Pete Randall]

Mohammad,

Typically /var is a separate filesystem. You can tell from the output of bdf. If you see a separate line for /var, then it is a separate filesystem and removing things from it will not help the root filesystem problem.


Pete

[Mohammad Kaleem]

Hi Pete,
Yes it is a seperate file system and it got sufficient space.

Thanks,
Mohammad

[Robert-Jan Goossens]

Hi Mohammed,

Could you next command,

# find / -size +1000 -xdev -exec ls -l {} \;

Kind regards,
Robert-Jan

[Mohammad Kaleem]

Hi Robert,
Here is the output.

Omansms2:root> find / -size +1000 -xdev -exec ls -l {} \;
-r-xr-xr-x 1 bin bin 2547712 Apr 6 2000 /etc/opt/resmon/lbin/m
ibmond
-r-xr-xr-x 1 bin bin 2560000 Apr 6 2000 /etc/opt/resmon/lbin/r
dbmsmond
-r-xr-xr-x 1 bin bin 532480 Apr 6 2000 /etc/opt/resmon/lib/li
bresmon.1
-rw-r--r-- 1 root root 672446 Feb 5 16:39 /etc/opt/resmon/log/ar
mmon.log
-rw-r--r-- 1 root root 861312 Jan 28 04:16 /etc/opt/resmon/log/re
slog.html
-rw-r--r-- 1 root root 672994 Feb 5 16:40 /etc/opt/resmon/log/fc
60mon.log
-rw-r--r-- 1 root sys 579664 Jul 14 2001 /etc/opt/HP-AIN/config
/typeDB/typeDB.0.1
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/lvchange
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/lvcreate
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/lvdisplay
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/lvextend
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/lvlnboot
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/lvreduce
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/lvremove
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/lvrmboot
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/pvchange
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/pvcreate
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/pvdisplay
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/pvmove
-r-sr-xr-x 1 root sys 778240 Apr 29 1999 /sbin/sdstolvm
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/vgcfgbackup
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/vgcfgrestore
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/vgchange
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/vgcreate
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/vgdisplay
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/vgexport
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/vgextend
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/vgimport
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/vgreduce
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/vgremove
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/vgscan
-r-sr-xr-x 4 root sys 811008 Aug 14 1999 /sbin/pvck
-r-sr-xr-x 4 root sys 811008 Aug 14 1999 /sbin/lvchange.run
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/lvmerge
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/lvsplit
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/lvsync
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/vgsync
-r-sr-xr-x 28 root sys 737280 Nov 6 1997 /sbin/nomwcsyncd
-r-sr-xr-x 4 root sys 811008 Aug 14 1999 /sbin/pvremove
-r-sr-xr-x 4 root sys 811008 Aug 14 1999 /sbin/vgchgid
-rw-rw-rw- 1 root sys 861727 Feb 3 16:43 /filelist.Z

[Joakim Lemström]

Mohammad

How small/large are your root filesystem?

Maybe you could resize the root volume?

//Jocke

[Kent Ostby]

Mohammad --

Since the usual tricks haven't worked (such as find -size ), there are a couple of other possible scenarios.

1) Large temporary file caused by a process using too much CPU/not doing the right thing.

2) File that was put on root and then had a file system mounted over the top of it.

----------------------

With regards to #1) do:

ps -ef | cut -c44-80 | sort -nr | head

This will show you the top 10 processes on your box as far as CPU usage.

Caution should be used as far as killing any of these off, but if you see something you wouldn't expect to see there, stopping and restarting it could fix your problem.

With regards to #2), the only way to troubleshoot this would be to reboot into single user mode and do the following:

WITHOUT MOUNTING any filesystems,
type:

ll /usr
ll /tmp
ll /var
ll (any other filesystems that you mount from root).

See if files appear there at a mountpoint from a mounted filesystem (see /etc/fstab for a list of what directories you use as mountpoints).

Best regards,

Kent M. Ostby

[sinhass]

U may check /usr/preserve /lost+found /dev
-sinhass

[Cheryl Griffin]

Look for open files that may have been removed (unlinked). For instance:
# /opt/lsof/bin/lsof +aL1 /var

[Steven E. Protter]

If you have had trouble mounting var or:

If you have accidentally copied files into any filesystem mount folder while its unmounted.

The files are not visible after mount.

You should do this:

boot at console

interupt at the 10 second prompt

hpux -is

N Don't interact

In single user mode, with no mounts check out the folders you normally mount in your /etc/fstab file.

You may find stuff that does not belong.

SEP