topic Re: Full root FS in Operating System - HP-UX

Full root FS

Troyan Krastev — Mon, 21 May 2001 20:27:04 GMT

Hi,

I have a HP-UX 11.0 box with root file system 150MB.
Today I see it is 98% full and I can't find nothing wrong. All FS are mounted, no large files found using "du". I try lsof but I still
can not find the process that keep this large file open, I think.

Help!

Re: Full root FS

A. Clay Stephenson — Mon, 21 May 2001 20:42:46 GMT

Hi Troyan,

Try this find command; you are less likely to miss a file than with du:

find / -xdev -size +2000

There should be very few files > 2000 blks in /

Clay

Re: Full root FS

Bill Hassell — Mon, 21 May 2001 20:44:53 GMT

It's probably not a large file at all, but instead, a directory that doesn't belong in /. NOTHING goes in /except /etc /dev /sbin and perhaps /root for root's $HOME. Read the atttachment about root filesystem space.

Re: Full root FS

James R. Ferguson — Mon, 21 May 2001 20:44:56 GMT

Hi:

Check for files that don't belong in /dev . If you have done something lika a 'tar' but mis-spelled "/dev/rmt/0m", for instance, then you didn't reference the device file but rather created a file and therefore added substantial disk space.

...JRF...

Re: Full root FS

Patrick Wallek — Mon, 21 May 2001 22:50:18 GMT

I agree with James. Look for something in /dev/rmt/ directory that is called om, omn, Om,Omn, lm,lmn, where the first character of the file is the letter O (NOT a zero [0]) or the letter L (not the number 1). This is probably the most common mistake when filling your / (root) file system.

Re: Full root FS

Shannon Petry — Mon, 21 May 2001 23:56:14 GMT

Some other goodies can also be found in the following locations,

/var/adm/crash
Look for everything
/var/adm
look for large acct files
/var/mail
look for big mail files...
/var/spool
look for dead print jobs that are huge!
check /var/dt for unused configurations, old error logs, etc...

check the dot files under /, especially if your running netscape! The cache directory defaults to 10MB. Also, check the /.dt directory. If there are problems, logs and junk will be there.

The best tool to use is "/usr/bin/du".
I.E.
> cd /
> du -sk *

This will show the use of all the directories!

Regards,
Shannon

Re: Full root FS

A. Clay Stephenson — Tue, 22 May 2001 00:28:07 GMT

Hi:

I've thought of one other place your space may be hiding - beneath a mountpoint. You may need to unmount a filesystem (e.g. /home) and then do an ls /home to see if files and directories appear.

Just one more idea, Clay

Re: Full root FS

Troyan Krastev — Tue, 22 May 2001 13:01:06 GMT

Hi A. Clay Stephenson,
silver:/ # find / -xdev -size +2000
/etc/rc.log.old
/etc/bgs/SD/bgssd
rc.log.old you already now what is, bgssd is a daemon for Best1 product.

Hi James and Patrick,
silver:/dev # du .
2 ./vg00
4 ./dsk
4 ./rdsk
0 ./diag/rmt
2 ./diag
2 ./rmt
0 ./rscsi
8 ./ptym
8 ./pty
8 ./pts
0 ./screen
0 ./telnet
0 ./krm
0 ./swap
2 ./symmvg01
2 ./symmvg02
2 ./symmvg03
2 ./symmvg04
4 ./symmvg05
0 ./symmvg06
58 .
Nothing wrong with /dev!?!

Hi Shannon,
You talk about /var, it si mount point:
lver:/ # ls -di *
677 bin 4 etc 3 lost+found
2 tmp 1303 cdrom 2 home 2 opt 2 usr 64 dev
502 lib 13 sbin 2 var
2 download 2 local 2 stand
/cdrom - empty
/bin - link to /usr/bin
/lib - link to /usr/lib
silver:/sbin # du .
3100 ./fs/hfs
3722 ./fs/vxfs
448 ./fs/cdfs
136 ./fs/nfs
72 ./fs/autofs
296 ./fs/lofs
8554 ./fs/vxfs3.3
16330 ./fs
12 ./lib/mfsconfig.d
60 ./lib/eisa
1028 ./lib
750 ./init.d
4 ./rc1.d
4 ./rc2.d
0 ./rc3.d
2 ./SnmpAgtStart.d
2 ./rc0.d
0 ./rc4.d
210 ./set_parms.d
47460 .

Hi again again A. Clay Stephenson,

Good idea, If the problem doesn't go away after reboot,
I have to check it!

Thanks all, but I still need help.
If you need some more information just let me know!!!

Re: Full root FS

Vincenzo Restuccia — Tue, 22 May 2001 13:49:25 GMT

You may have had a file opened by a process which has been removed while it was still open.
The normal way to cure this is find the process using something lik efuser, or reboot the system and see if that frees up the space.
If it does, monitor var closely and try to discover which process is writing to it.

Re: Full root FS

Asad Malik — Tue, 22 May 2001 14:01:24 GMT

Hi
Check the /dev directory. Sometime a failed backup is dumped there. look for a non character file.
du -ka /dev |sort -rn|more will list the large file in ascending order.

Re: Full root FS

Troyan Krastev — Tue, 22 May 2001 14:19:08 GMT

Hi again,

Some aditional info:
silver:/ # bdf /
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 143360 141770 1512 99% /

Output of "silver:/ # du -kx / | sort -rn " is attached!

Thanks for your time!

Re: Full root FS

Patrick Wallek — Tue, 22 May 2001 16:38:16 GMT

What is roots home directory? Is it still / ? I did your du command on my / directory and my largest directory is my .netscape directory. Do you have any . directories on this machine? Here is the output of my du for your reference.

Re: Full root FS

Troyan Krastev — Tue, 22 May 2001 16:55:21 GMT

Hi Patrick,

I attach output of "ll /". I still believe that it is a opened file that is removed, but I can't find the process that holds this file.

Re: Full root FS

Patrick Wallek — Tue, 22 May 2001 19:18:28 GMT

You might try installing and using lsof to see what open files you have on the system. lsof will also tell you what process is using the file.

http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.55/

Re: Full root FS

boley janowski — Tue, 22 May 2001 19:47:28 GMT

check your crontab as well, this may be a good place to identify specifics that are running. make sure you are not running any scripts that my be looping while tring to write to your somewhere under /

Re: Full root FS

Troyan Krastev — Tue, 22 May 2001 20:39:11 GMT

Hi Patrick,

I installed lsof as a depot, but it was compiled for 32bit and I had to build it myself. The output of # lsof -a +L1 / didn't produce enything. When I tried #lsof -a +L / in the output in column NLINK I see some big numbers. In HP-UX 11.0 32bit this column is correct???? Check the attached file.

Nothing in cron, Boley!

Re: Full root FS

Troyan Krastev — Thu, 24 May 2001 15:47:39 GMT

Hi all,

One good and one bad news. The good news is that after reboot the "/" is 25% full :-). The bad news is that I couldn't find what was wrong :-(.

Bill Hassell, send me a message to give you some points. I like your attachmants.

Re: Full root FS

Ovidiu D. Raita — Thu, 24 May 2001 17:18:47 GMT

It happened to me once. Long time ago ... It was a cron job. I killed cron and I didn't have to reboot.

If it happens again you might want to see if it's cron.

Ovidiu