topic Re: / 100% full - HELP ! in Operating System - HP-UX

/ 100% full - HELP !

Tan Shirley — Sun, 28 Sep 2003 20:04:58 GMT

Hi,

I've got a K-class machine running on HP-UX 10.20. About two weeks ago, I noticed that the root filesystem was 100% full. Dmesg reflected the following:

vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)

Did a du to catch the culprit, but to no avail as most of them are mounted onto their own mount points. Tried clearing root's email in elm, and trimmed all logfiles but still does not help. Rebooted the system as well and its' still 100% full. tried remove /etc/mnttab and then perform mount -a again and still does not help. Any guidelines on what to check ? What is taking up the root filesystem ?

Many Thanks,
Shirley

Re: / 100% full - HELP !

Michael Tully — Sun, 28 Sep 2003 20:10:56 GMT

Hi,

In many cases this is caused by a an incorrect backup being started to a flat file instead of a device.

# du /dev

# du /etc

Regards
Michael

Re: / 100% full - HELP !

Tan Shirley — Sun, 28 Sep 2003 20:15:01 GMT

Hi Michael,

Thanks for your prompt response !
My du /dev/ shows the following:

#du /dev |sort -rn
24 /dev
4 /dev/ptym
4 /dev/pty
2 /dev/vg01
2 /dev/vg00
2 /dev/rmt
2 /dev/pts
2 /dev/diag
0 /dev/rscsi
0 /dev/rdsk
0 /dev/dsk
0 /dev/diag/rmt

And my du /etc shows the following:
#du /etc |sort -rn |more

5504 /etc
1762 /etc/lvmconf
908 /etc/hpC2400
674 /etc/X11
488 /etc/UPS.d
228 /etc/vue
196 /etc/opt
188 /etc/rc.config.d
188 /etc/mail
184 /etc/opt/dce
166 /etc/vue/config
118 /etc/SnmpAgent.d
108 /etc/sam

Any clues what has gone wrong ?

Re: / 100% full - HELP !

Michael Tully — Sun, 28 Sep 2003 20:27:42 GMT

Going by the output, we have picked the wrong directory. Let's try and find some large files in /

# find / -xdev -size 100000c -print
This will only check / and not any other filesystems.

Re: / 100% full - HELP !

Bill Hassell — Sun, 28 Sep 2003 20:31:33 GMT

You need to run du on just the / directory. Do this:

du -kx / | sort -rn > /var/tmp/du.root

Now the file /var/tmp will have all the directories in / sorted by size. The largest directories are:

31056 /
18979 /sbin
9331 /etc
6680 /etc/opt
3676 /sbin/fs

No other directories should show up in this list more than a few Kbytes. Growth in / is caused by sysadmin mistakes or intallers that fail to ask the sysadmin where files should be stored.

/dev should NOT be larger than about 20-40 Kbytes (the -k option for du reports in Lb) If /dev is large, there are ordinary files in /dev (there should never be any). Most common are files like /dev/rmt/om or /dev/null. Find all the files in /dev with:

find /dev -type f -exec ll {} \;

There should be nothing shown. If /dev/null shows up as an ordinary file, do this:

rm /dev/null
mknod /dev/null c 3 2

If the root user accidently removes /dev/null, then it will become an ordinary file, possibly megs in size. If the / directory is the largest in the du.root file, then sort all the / files with:

ll / | sort -rnk5 | head

There should be *no* ordinary files in /, just directories. Unfortunately, HP-UX (and a few other Unix flavors) defauot root's $HOME directory to /, probably the worst possible location. Note that logfiles, email, spool files, etc are not in the / directory so this isn't the problem.

Re: / 100% full - HELP !

Michael Steele_2 — Sun, 28 Sep 2003 22:33:18 GMT

All of these should be run:

a) du -k | sort -rn | more

b) find /dev -type f

c) quot /

d) find / -xdev -ctime 0 -exec ll {} \'

NOTE: curly brackets for d.)

Re: / 100% full - HELP !

Steven E. Protter — Sun, 28 Sep 2003 22:34:16 GMT

Something else that gets by a lot of folks, the old hidden file trick.

You create a folder on root called /steve

You copy in 100 MB.

You forget and decide to mount a filesystem called /steve on it. You set it up with lvm and mount it. Never remove the 100 MB.

The files are still there and you can't see them, because there is a filesytem mounted on it.

If all other measures faile the way to go is to bring the system into single user mode and start looking around.

Interupt a boot at the console at the 10 second prompt.

bo

Y Interact ...
hpux -is

A more realistic scenario is when /var failed to mount once during a boot. Operations missed it while I was out of town. Logs got written to the /var folder until the / filesystem got near full, then they booted the system. I got home to a very full / filesystem with no explanation. The only way to find it was single user mode.

SEP
Happy 5764
<

Re: / 100% full - HELP !

Tan Shirley — Sun, 28 Sep 2003 23:16:24 GMT

Hi,
Here's the results for the following commands:

#find /dev -type f
/dev/null2

#quot /
/dev/vg00/rlvol3 (/):
66335 oracle
13084 bin
3581 root
11 lp

#du -kx / | sort -rn
17093 /
14163 /sbin
3039 /sbin/fs
2752 /etc
1297 /sbin/fs/vxfs
1110 /sbin/fs/hfs
881 /etc/lvmconf
510 /sbin/lib
454 /etc/hpC2400
388 /sbin/fs/nfs
337 /etc/X11
269 /sbin/init.d
244 /etc/UPS.d
132 /sbin/fs/cdfs
122 /sbin/set_parms.d

Any idea what could have gone wrong and eating up the space ? Thanks !

Re: / 100% full - HELP !

Con O'Kelly — Sun, 28 Sep 2003 23:23:30 GMT

Hi Shirley

Can you post:
# bdf /

Cheers
Con

Re: / 100% full - HELP !

Ian Kidd_1 — Sun, 28 Sep 2003 23:59:38 GMT

Shirley,

I use the `find` command for situations like this. There's a couple of options that will help:
-xdev This keeps the find command from crossing to a different mount point
-size Allows you to select the size of the files you're searching for
-exec executes a command on the found files

I'd do something like:
find / -xdex -size +20000 -exec ls -l {} \;

This will give a long listing (`ls -l`) on the file that you found.

The "\;" is necessary on the find command when you're using the exec option. If this finds too many files, bump up the "-size" from +20000 to +25000. If it doesn't find anything, start using smaller numbers( +15000, then +10000, etc).

Re: / 100% full - HELP !

Tan Shirley — Mon, 29 Sep 2003 00:44:12 GMT

Hi,
When issued with the following find command, here's the results...

#find / -xdev -size +1200 -exec ls -l {} \;

-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvchange
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvcreate
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvmerge
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvdisplay
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvextend
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvlnboot
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvreduce
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvremove
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvrmboot
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/pvchange
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/pvcreate
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/pvdisplay
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/pvmove
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgcfgbackup
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgcfgrestore
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgchange
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgcreate
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgdisplay
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgexport
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgextend
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgimport
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgreduce
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgscan

And my bdf / output is as below:

#bdf /
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 86016 85104 888 99% /
Many thanks !

Re: / 100% full - HELP !

Mark Grant — Mon, 29 Sep 2003 00:51:39 GMT

Your root filesystem is a bit small. I would consider making it bigger if I were you. I think a complete "bdf" output for vg00 would be useful here. If, for example, you don't have a separate file system for /tmp or /var you are going to have some trouble.

Re: / 100% full - HELP !

Tan Shirley — Mon, 29 Sep 2003 01:07:50 GMT

Hi Mark,

Here's the bdf output for vg00.

#bdf |grep vg00
/dev/vg00/lvol3 86016 85104 888 99% /
/dev/vg00/lvol1 47829 18230 24816 42% /stand
/dev/vg00/lvol8 327680 144480 173419 45% /var
/dev/vg00/lvol7 417792 313482 97823 76% /usr
/dev/vg00/lvol4 32768 4591 27760 14% /tmp
/dev/vg00/lvol10 658197 478014 114363 81% /oradata2
/dev/vg00/lvol9 1024000 290235 687928 30% /oracle
/dev/vg00/lvol6 368640 286149 77508 79% /opt
/dev/vg00/lvol5 40960 5485 33395 14% /home

Any clue what could have caused the root filesystem to be suddenly full? This machine has been running for years and the root filesystem had never reached near 90% before.
Thanks !

Re: / 100% full - HELP !

T G Manikandan — Mon, 29 Sep 2003 01:13:35 GMT

#ls /
stand
var
usr
tmp
oradata2
oracle
opt
home

Do you have anyother dir's inside root where some files are residing and not mounted probably that is using the space

Revert

Re: / 100% full - HELP !

Mark Grant — Mon, 29 Sep 2003 01:21:15 GMT

Perhaps worth looking for "core" files.

find / -name core -ok rm {} \;

This will find any core files and prompt you to remove them. You probably don't need them.

Re: / 100% full - HELP !

Thierry Poels_1 — Mon, 29 Sep 2003 01:28:31 GMT

Hi Shirley,

(as already mentioned 80MB for / ain't really big)

As no large files could be found on /, there probably must be many small files, not?
.
Try:
find / -xdev -exec ll -d {} \;
Have a quick browse through this long list, and (re)move non-system files.
.
find /dev -size +20 -exec ll -d {} \;
should not produce any output.

good luck,
Thierry.

Re: / 100% full - HELP !

Con O'Kelly — Mon, 29 Sep 2003 01:36:15 GMT

Hi

The main issue here is that du reports around 30MB for / while bdf shows its using 85MB.

Normally this indicates that you have an open file that has been removed but hasn't released space in /. However a reboot fixes this problem.

If you have lsof installed run the following command:
# lsof / | sort -rnk 7 |more.
(This shows all open files for / fs, sorted by order of size.)

Often its possible from this output to figure out the offending file and then kill the process that has the file open, thus releasing the space.

The other possibility is what SEP & TG mentioned. Have you mounted any new filesystems recently (run mount command to see date of filesystem mounts).

Cheers
Con

Re: / 100% full - HELP !

Tan Shirley — Mon, 29 Sep 2003 02:10:55 GMT

Hi Mark,
Using the find / -name core command manage to find one core file. Have removed it but still does not help.

Hi Con,
I agree with you. The strange thing is, du reports about 30MB, but why does a root filesystem of 85MB not sufficient ? That's strange.
Unfortunately I do not have "lsof" installed. Anyway to install it? I dont think I have any new filesystems recently created and mounted wrongly.
Normally I'll use the "SAM" method if I want to create a new lvol. I'll never create a directory if I want a new filesystem.

Any clues?

Troubled Shirley.. :-(

Re: / 100% full - HELP !

Stanimir — Mon, 29 Sep 2003 02:16:15 GMT

Try:
#find / -name core -exec rm {} \;

Also you could try to remove log-files
/with regards to its content/:
# > logfile;

Regards

Re: / 100% full - HELP !

T G Manikandan — Mon, 29 Sep 2003 02:42:22 GMT

get lsof from here

https://www.beepz.com/personal/merijn/