Full root FS

[Troyan Krastev]

Hi,

I have a HP-UX 11.0 box with root file system 150MB.
Today I see it is 98% full and I can't find nothing wrong. All FS are mounted, no large files found using "du". I try lsof but I still
can not find the process that keep this large file open, I think.

Help!

[A. Clay Stephenson]

Hi Troyan,

Try this find command; you are less likely to miss a file than with du:

find / -xdev -size +2000

There should be very few files > 2000 blks in /

Clay

[Bill Hassell]

It's probably not a large file at all, but instead, a directory that doesn't belong in /. NOTHING goes in /except /etc /dev /sbin and perhaps /root for root's $HOME. Read the atttachment about root filesystem space.

[James R. Ferguson]

Hi:

Check for files that don't belong in /dev . If you have done something lika a 'tar' but mis-spelled "/dev/rmt/0m", for instance, then you didn't reference the device file but rather created a file and therefore added substantial disk space.

...JRF...

[Patrick Wallek]

I agree with James. Look for something in /dev/rmt/ directory that is called om, omn, Om,Omn, lm,lmn, where the first character of the file is the letter O (NOT a zero [0]) or the letter L (not the number 1). This is probably the most common mistake when filling your / (root) file system.

[Shannon Petry]

Some other goodies can also be found in the following locations,

/var/adm/crash
Look for everything
/var/adm
look for large acct files
/var/mail
look for big mail files...
/var/spool
look for dead print jobs that are huge!
check /var/dt for unused configurations, old error logs, etc...

check the dot files under /, especially if your running netscape! The cache directory defaults to 10MB. Also, check the /.dt directory. If there are problems, logs and junk will be there.

The best tool to use is "/usr/bin/du".
I.E.
> cd /
> du -sk *

This will show the use of all the directories!

Regards,
Shannon

[A. Clay Stephenson]

Hi:

I've thought of one other place your space may be hiding - beneath a mountpoint. You may need to unmount a filesystem (e.g. /home) and then do an ls /home to see if files and directories appear.

Just one more idea, Clay

[Troyan Krastev]

Hi A. Clay Stephenson,
silver:/ # find / -xdev -size +2000
/etc/rc.log.old
/etc/bgs/SD/bgssd
rc.log.old you already now what is, bgssd is a daemon for Best1 product.

Hi James and Patrick,
silver:/dev # du .
2 ./vg00
4 ./dsk
4 ./rdsk
0 ./diag/rmt
2 ./diag
2 ./rmt
0 ./rscsi
8 ./ptym
8 ./pty
8 ./pts
0 ./screen
0 ./telnet
0 ./krm
0 ./swap
2 ./symmvg01
2 ./symmvg02
2 ./symmvg03
2 ./symmvg04
4 ./symmvg05
0 ./symmvg06
58 .
Nothing wrong with /dev!?!

Hi Shannon,
You talk about /var, it si mount point:
lver:/ # ls -di *
677 bin 4 etc 3 lost+found
2 tmp 1303 cdrom 2 home 2 opt 2 usr 64 dev
502 lib 13 sbin 2 var
2 download 2 local 2 stand
/cdrom - empty
/bin - link to /usr/bin
/lib - link to /usr/lib
silver:/sbin # du .
3100 ./fs/hfs
3722 ./fs/vxfs
448 ./fs/cdfs
136 ./fs/nfs
72 ./fs/autofs
296 ./fs/lofs
8554 ./fs/vxfs3.3
16330 ./fs
12 ./lib/mfsconfig.d
60 ./lib/eisa
1028 ./lib
750 ./init.d
4 ./rc1.d
4 ./rc2.d
0 ./rc3.d
2 ./SnmpAgtStart.d
2 ./rc0.d
0 ./rc4.d
210 ./set_parms.d
47460 .

Hi again again A. Clay Stephenson,

Good idea, If the problem doesn't go away after reboot,
I have to check it!

Thanks all, but I still need help.
If you need some more information just let me know!!!

[Vincenzo Restuccia]

You may have had a file opened by a process which has been removed while it was still open.
The normal way to cure this is find the process using something lik efuser, or reboot the system and see if that frees up the space.
If it does, monitor var closely and try to discover which process is writing to it.

[Asad Malik]

Hi
Check the /dev directory. Sometime a failed backup is dumped there. look for a non character file.
du -ka /dev |sort -rn|more will list the large file in ascending order.

[Troyan Krastev]

Hi again,

Some aditional info:
silver:/ # bdf /
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 143360 141770 1512 99% /

Output of "silver:/ # du -kx / | sort -rn " is attached!

Thanks for your time!

[Patrick Wallek]

What is roots home directory? Is it still / ? I did your du command on my / directory and my largest directory is my .netscape directory. Do you have any . directories on this machine? Here is the output of my du for your reference.

[Troyan Krastev]

Hi Patrick,

I attach output of "ll /". I still believe that it is a opened file that is removed, but I can't find the process that holds this file.

[Patrick Wallek]

You might try installing and using lsof to see what open files you have on the system. lsof will also tell you what process is using the file.

http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.55/

[boley janowski]

check your crontab as well, this may be a good place to identify specifics that are running. make sure you are not running any scripts that my be looping while tring to write to your somewhere under /

[Troyan Krastev]

Hi Patrick,

I installed lsof as a depot, but it was compiled for 32bit and I had to build it myself. The output of # lsof -a +L1 / didn't produce enything. When I tried #lsof -a +L / in the output in column NLINK I see some big numbers. In HP-UX 11.0 32bit this column is correct???? Check the attached file.

Nothing in cron, Boley!

[Troyan Krastev]

Hi all,

One good and one bad news. The good news is that after reboot the "/" is 25% full :-). The bad news is that I couldn't find what was wrong :-(.

Bill Hassell, send me a message to give you some points. I like your attachmants.

[Ovidiu D. Raita]

It happened to me once. Long time ago ... It was a cron job. I killed cron and I didn't have to reboot.

If it happens again you might want to see if it's cron.

Ovidiu