1826332 Members
3482 Online
109692 Solutions
New Discussion

Re: / 100% full - HELP !

 
SOLVED
Go to solution
Tan Shirley
Frequent Advisor

/ 100% full - HELP !

Hi,

I've got a K-class machine running on HP-UX 10.20. About two weeks ago, I noticed that the root filesystem was 100% full. Dmesg reflected the following:

vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)

Did a du to catch the culprit, but to no avail as most of them are mounted onto their own mount points. Tried clearing root's email in elm, and trimmed all logfiles but still does not help. Rebooted the system as well and its' still 100% full. tried remove /etc/mnttab and then perform mount -a again and still does not help. Any guidelines on what to check ? What is taking up the root filesystem ?

Many Thanks,
Shirley
34 REPLIES 34
Michael Tully
Honored Contributor

Re: / 100% full - HELP !

Hi,

In many cases this is caused by a an incorrect backup being started to a flat file instead of a device.

# du /dev

# du /etc

Regards
Michael
Anyone for a Mutiny ?
Tan Shirley
Frequent Advisor

Re: / 100% full - HELP !

Hi Michael,

Thanks for your prompt response !
My du /dev/ shows the following:

#du /dev |sort -rn
24 /dev
4 /dev/ptym
4 /dev/pty
2 /dev/vg01
2 /dev/vg00
2 /dev/rmt
2 /dev/pts
2 /dev/diag
0 /dev/rscsi
0 /dev/rdsk
0 /dev/dsk
0 /dev/diag/rmt


And my du /etc shows the following:
#du /etc |sort -rn |more

5504 /etc
1762 /etc/lvmconf
908 /etc/hpC2400
674 /etc/X11
488 /etc/UPS.d
228 /etc/vue
196 /etc/opt
188 /etc/rc.config.d
188 /etc/mail
184 /etc/opt/dce
166 /etc/vue/config
118 /etc/SnmpAgent.d
108 /etc/sam

Any clues what has gone wrong ?
Michael Tully
Honored Contributor

Re: / 100% full - HELP !

Going by the output, we have picked the wrong directory. Let's try and find some large files in /

# find / -xdev -size 100000c -print
This will only check / and not any other filesystems.
Anyone for a Mutiny ?
Bill Hassell
Honored Contributor

Re: / 100% full - HELP !

You need to run du on just the / directory. Do this:

du -kx / | sort -rn > /var/tmp/du.root

Now the file /var/tmp will have all the directories in / sorted by size. The largest directories are:

31056 /
18979 /sbin
9331 /etc
6680 /etc/opt
3676 /sbin/fs

No other directories should show up in this list more than a few Kbytes. Growth in / is caused by sysadmin mistakes or intallers that fail to ask the sysadmin where files should be stored.

/dev should NOT be larger than about 20-40 Kbytes (the -k option for du reports in Lb) If /dev is large, there are ordinary files in /dev (there should never be any). Most common are files like /dev/rmt/om or /dev/null. Find all the files in /dev with:

find /dev -type f -exec ll {} \;

There should be nothing shown. If /dev/null shows up as an ordinary file, do this:

rm /dev/null
mknod /dev/null c 3 2

If the root user accidently removes /dev/null, then it will become an ordinary file, possibly megs in size. If the / directory is the largest in the du.root file, then sort all the / files with:

ll / | sort -rnk5 | head

There should be *no* ordinary files in /, just directories. Unfortunately, HP-UX (and a few other Unix flavors) defauot root's $HOME directory to /, probably the worst possible location. Note that logfiles, email, spool files, etc are not in the / directory so this isn't the problem.


Bill Hassell, sysadmin
Michael Steele_2
Honored Contributor
Solution

Re: / 100% full - HELP !

All of these should be run:

a) du -k | sort -rn | more

b) find /dev -type f

c) quot /

d) find / -xdev -ctime 0 -exec ll {} \'

NOTE: curly brackets for d.)
Support Fatherhood - Stop Family Law
Steven E. Protter
Exalted Contributor

Re: / 100% full - HELP !

Something else that gets by a lot of folks, the old hidden file trick.

You create a folder on root called /steve

You copy in 100 MB.

You forget and decide to mount a filesystem called /steve on it. You set it up with lvm and mount it. Never remove the 100 MB.

The files are still there and you can't see them, because there is a filesytem mounted on it.

If all other measures faile the way to go is to bring the system into single user mode and start looking around.

Interupt a boot at the console at the 10 second prompt.

bo

Y Interact ...
hpux -is


A more realistic scenario is when /var failed to mount once during a boot. Operations missed it while I was out of town. Logs got written to the /var folder until the / filesystem got near full, then they booted the system. I got home to a very full / filesystem with no explanation. The only way to find it was single user mode.


SEP
Happy 5764
<
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Tan Shirley
Frequent Advisor

Re: / 100% full - HELP !

Hi,
Here's the results for the following commands:

#find /dev -type f
/dev/null2

#quot /
/dev/vg00/rlvol3 (/):
66335 oracle
13084 bin
3581 root
11 lp

#du -kx / | sort -rn
17093 /
14163 /sbin
3039 /sbin/fs
2752 /etc
1297 /sbin/fs/vxfs
1110 /sbin/fs/hfs
881 /etc/lvmconf
510 /sbin/lib
454 /etc/hpC2400
388 /sbin/fs/nfs
337 /etc/X11
269 /sbin/init.d
244 /etc/UPS.d
132 /sbin/fs/cdfs
122 /sbin/set_parms.d


Any idea what could have gone wrong and eating up the space ? Thanks !
Con O'Kelly
Honored Contributor

Re: / 100% full - HELP !

Hi Shirley

Can you post:
# bdf /

Cheers
Con
Ian Kidd_1
Trusted Contributor

Re: / 100% full - HELP !

Shirley,

I use the `find` command for situations like this. There's a couple of options that will help:
-xdev This keeps the find command from crossing to a different mount point
-size Allows you to select the size of the files you're searching for
-exec executes a command on the found files

I'd do something like:
find / -xdex -size +20000 -exec ls -l {} \;

This will give a long listing (`ls -l`) on the file that you found.

The "\;" is necessary on the find command when you're using the exec option. If this finds too many files, bump up the "-size" from +20000 to +25000. If it doesn't find anything, start using smaller numbers( +15000, then +10000, etc).
If at first you don't succeed, go to the ITRC
Tan Shirley
Frequent Advisor

Re: / 100% full - HELP !

Hi,
When issued with the following find command, here's the results...

#find / -xdev -size +1200 -exec ls -l {} \;

-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvchange
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvcreate
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvmerge
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvdisplay
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvextend
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvlnboot
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvreduce
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvremove
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvrmboot
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/pvchange
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/pvcreate
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/pvdisplay
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/pvmove
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgcfgbackup
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgcfgrestore
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgchange
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgcreate
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgdisplay
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgexport
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgextend
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgimport
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgreduce
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgscan



And my bdf / output is as below:

#bdf /
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 86016 85104 888 99% /
Many thanks !
Mark Grant
Honored Contributor

Re: / 100% full - HELP !

Your root filesystem is a bit small. I would consider making it bigger if I were you. I think a complete "bdf" output for vg00 would be useful here. If, for example, you don't have a separate file system for /tmp or /var you are going to have some trouble.
Never preceed any demonstration with anything more predictive than "watch this"
Tan Shirley
Frequent Advisor

Re: / 100% full - HELP !

Hi Mark,


Here's the bdf output for vg00.

#bdf |grep vg00
/dev/vg00/lvol3 86016 85104 888 99% /
/dev/vg00/lvol1 47829 18230 24816 42% /stand
/dev/vg00/lvol8 327680 144480 173419 45% /var
/dev/vg00/lvol7 417792 313482 97823 76% /usr
/dev/vg00/lvol4 32768 4591 27760 14% /tmp
/dev/vg00/lvol10 658197 478014 114363 81% /oradata2
/dev/vg00/lvol9 1024000 290235 687928 30% /oracle
/dev/vg00/lvol6 368640 286149 77508 79% /opt
/dev/vg00/lvol5 40960 5485 33395 14% /home


Any clue what could have caused the root filesystem to be suddenly full? This machine has been running for years and the root filesystem had never reached near 90% before.
Thanks !
T G Manikandan
Honored Contributor

Re: / 100% full - HELP !

#ls /
stand
var
usr
tmp
oradata2
oracle
opt
home

Do you have anyother dir's inside root where some files are residing and not mounted probably that is using the space

Revert
Mark Grant
Honored Contributor

Re: / 100% full - HELP !

Perhaps worth looking for "core" files.

find / -name core -ok rm {} \;

This will find any core files and prompt you to remove them. You probably don't need them.
Never preceed any demonstration with anything more predictive than "watch this"
Thierry Poels_1
Honored Contributor

Re: / 100% full - HELP !

Hi Shirley,

(as already mentioned 80MB for / ain't really big)

As no large files could be found on /, there probably must be many small files, not?
.
Try:
find / -xdev -exec ll -d {} \;
Have a quick browse through this long list, and (re)move non-system files.
.
find /dev -size +20 -exec ll -d {} \;
should not produce any output.


good luck,
Thierry.
All unix flavours are exactly the same . . . . . . . . . . for end users anyway.
Con O'Kelly
Honored Contributor

Re: / 100% full - HELP !

Hi

The main issue here is that du reports around 30MB for / while bdf shows its using 85MB.

Normally this indicates that you have an open file that has been removed but hasn't released space in /. However a reboot fixes this problem.

If you have lsof installed run the following command:
# lsof / | sort -rnk 7 |more.
(This shows all open files for / fs, sorted by order of size.)

Often its possible from this output to figure out the offending file and then kill the process that has the file open, thus releasing the space.

The other possibility is what SEP & TG mentioned. Have you mounted any new filesystems recently (run mount command to see date of filesystem mounts).

Cheers
Con


Tan Shirley
Frequent Advisor

Re: / 100% full - HELP !

Hi Mark,
Using the find / -name core command manage to find one core file. Have removed it but still does not help.

Hi Con,
I agree with you. The strange thing is, du reports about 30MB, but why does a root filesystem of 85MB not sufficient ? That's strange.
Unfortunately I do not have "lsof" installed. Anyway to install it? I dont think I have any new filesystems recently created and mounted wrongly.
Normally I'll use the "SAM" method if I want to create a new lvol. I'll never create a directory if I want a new filesystem.

Any clues?

Troubled Shirley.. :-(
Stanimir
Trusted Contributor

Re: / 100% full - HELP !

Try:
#find / -name core -exec rm {} \;

Also you could try to remove log-files
/with regards to its content/:
# > logfile;

Regards
T G Manikandan
Honored Contributor

Re: / 100% full - HELP !

Jakes Louw
Trusted Contributor

Re: / 100% full - HELP !

Here's a long shot, Shirley, but maybe I'm right: your problem is caused by a program that starts after a boot (/sbin/rcx.d/Sxxxx type script), and that incorrectly reserves space for a file in root.
Have a look at which scripts in /sbin/init.d have changed recently, or have a look in /etc/rc.log for anything suspicious.
To test this theory, boot up in single-user mode, and see if / is still full. Then run the other advized stuff (the du and find ideas) without anything else mounted. You might have to mount /usr manually, can't remember if single-user mounts it or not, so make sure which LVOL is for /usr (default is /dev/vg00/lvol7).
Good luck.
Trying is the first step to failure - Homer Simpson
Nagarathinam
Advisor

Re: / 100% full - HELP !

Have you checked your /usr/adm/crash directory for already generated crash dump files ?

Check for Big files in the root file system using SAM routine tasks

if you are using Oracle database, check for listener log may be wrongly placed in root volume

Hope this helps

A.Nagarathinam
Bill Hassell
Honored Contributor

Re: / 100% full - HELP !

The output from du -kx / shows a normal amount of space usage and normal directories. As mentioned, the wide discrepancy is due to some strange process or script that has created a file during bootup or cron, but failed to properly remove a temp file. Logfiles are not the issue since they are all in /var. It might have been useful to know what caused the core file (hint: file core).

Since bdf and du have a very bad discrepancy, the problem is likely caused during startup. Check /var/adm/syslog.log, and /etc/rc.log for possible error messages. You may have to go reboot into single user mode to see usage without any mounted filesystems. NOTE: in single user mode, you'll have to issue: mount /usr to get access to most commands.


Bill Hassell, sysadmin
Jakes Louw
Trusted Contributor

Re: / 100% full - HELP !

Thanks, Bill. I knew I was on the right track.
Trying is the first step to failure - Homer Simpson
Michael Steele_2
Honored Contributor

Re: / 100% full - HELP !

Didn't see this command. -ctime will list all new files, -xdev will stay with /, -exec ll will list bytes size, time stamp, etc. along with files.

d) find / -xdev -ctime 0 -exec ll {} \'

Regarding: "...#find /dev -type f
/dev/null2..."

'/dev/null2' can be deleted.

Regarding "...#quot /
/dev/vg00/rlvol3 (/):
66335 oracle
13084 bin
3581 root
11 lp..."

Shouldn't have oracle in /. So this is where to concentrate. In the long run make a new logical volume and copy all the oracle files in / into it. Get with oracle for help. For now, run this command:

find / -name -user oracle | xargs ll, or,

find / -name -user oracle -exec ll {} \;

Regarding: "...#du -kx / | sort -rn
17093 /
14163 /sbin
3039 /sbin/fs
2752 /etc
1297 /sbin/fs/vxfs
1110 /sbin/fs/hfs
881 /etc/lvmconf
510 /sbin/lib
454 /etc/hpC2400..."

This list indicates / sub directories are OK. /sbin, /etc, for example. But '...17093..." is too, too many files in /. Concentrate here. This command will provide a cronological list for /.

ls -lrt /

Use it also.
Support Fatherhood - Stop Family Law