topic Re: / at 91% in Operating System - HP-UX

/ at 91%

Claude Fortin — Tue, 11 Sep 2001 12:03:18 GMT

Hello,

When I make a bdf, my / was at 91%
But yesterday, it was at 66%

Nothing change on the server, no new file?
Bad statistics?

Re: / at 91%

Ron Irving — Tue, 11 Sep 2001 12:07:54 GMT

this happened to me once. Look in the /dev directory for large 'rmt' devices. Did you attempt a backup recently? It may have written it out to the file, instead of the tape.

Hope this helps.

rri

Re: / at 91%

Santosh Nair_1 — Tue, 11 Sep 2001 12:09:39 GMT

Typically this happens to me when some process is trying to write to where it thinks that a filesystem exists...but for some reason or other, there is no filesystem. I would start by doing a bdf on /* and then du -sk on all local directories to find which is taking up the space.

Hope this helps.

-Santosh

Re: / at 91%

Volker Borowski — Tue, 11 Sep 2001 12:10:19 GMT

Hi,

most likely a coredump has been written due to a died application.

Go with "fine / -name core -print" to locate it.

Check if your /var/adm/syslog/syslog.log contains information.

In addition, it would be helpfull, if you could qualify the amount of MB you lost.

Best regards
Volker

Re: / at 91%

Sachin Patel — Tue, 11 Sep 2001 12:32:41 GMT

Hi Claude,
Either big cron file or you might have perform backup and it has created file in /dev/rmt/

Sachin

Re: / at 91%

James R. Ferguson — Tue, 11 Sep 2001 12:41:00 GMT

Hi Claude:

Look for 'core' files and look for files in /dev that don't belong there (i.e. regular files as opposed to non-character-special files beginning with "c" in the mode bits field (the first) as output by 'ls'.

It is likely that a tape backup device file specification was mis-keyed and instead of "0m" (zero) the letter "o" was substituted.

Regards!

...JRF...

Re: / at 91%

Jim Turner — Tue, 11 Sep 2001 13:59:48 GMT

Hi Claude,

To find the location of your offender:

# for a dir-by-dir summary
du -kxs /* | sort -rn | more

# for the whole list
du -kx / | sort -rn | more

All the best,
Jim

Re: / at 91%

Bill McNAMARA_1 — Tue, 11 Sep 2001 14:10:09 GMT

Start with this:

# find / -local -size +30000 -print

Bill

Re: / at 91%

Shannon Petry — Tue, 11 Sep 2001 16:09:59 GMT

Another thing to remember is that the % of disk used means squat! 91% of 100MB and 91% of 1GB are quite different.

While the system reports and enforces writes based on the percentage, the default size for / in HP-UX 11.0 is about 86MB. A simple core file can make up the 30% difference between days.

Regards,
Shannon

Re: / at 91%

Claude Fortin — Wed, 12 Sep 2001 18:22:11 GMT

Thanks for suggestions, did not find cause, decided to reboot when used increased to 92% and " / " went back to 33% ??

Re: / at 91%

Jim Turner — Wed, 12 Sep 2001 19:23:04 GMT

Hi (again) Claude,

What your observation tells me is this:

1. Someone or something created an enormous file (one or more) under /.

2. One or more processes had the file open.

3. The file was deleted.

4. bdf would still show the space as used until every last proc using that file was killed.

5. Rebooting killed said proc(s).

6. bdf now reflects reality.

All the best,
Jim

Re: / at 91%

Roger Baptiste — Wed, 12 Sep 2001 19:46:05 GMT

Claude,

The suggestions so far have been comprehensive covering all possibilities.
Since you say, only a reboot helped clearing
the space, a runaway process writing into
the / filesystem could have been the culprit.
To figure out such processes in the future:
* Make use of top or glance and see which
process is consuming a lot of CPU time
rapidly.
* look at ps listing and see the CPU times
of processes which are changing.

Reboot may have solved the problem , but
it could be a temporary lull, before it
occurs again. (rest of the things being
the same).

From a File check point of view:
find / -xdev -print >/tmp/filelist
and then a
du -sk on the files listed in /tmp/filelist
would give you a clue on the largest files
under root. (-xdev stops scanning below
root filesystem).

HTH
raj

Re: / at 91%

Darrell Allen — Wed, 12 Sep 2001 20:18:54 GMT

Hello Claude,

Just a few additions to what others have already said. It could be a log file that was growing and was "rolled" when you booted. If it was /var growing I'd think something in /var/adm and perhaps something was writing much to /var/adm/syslog/syslog.log. But since this is /...

To help identify the culprit, you can narrow it down to suspect directories like this:

find / -type d -xdev | xargs du -sk >/file1

Wait awhile (30 minutes?)

find / -type d -xdev | xargs du -sk >/file2
join -j 2 file1 file2 >file3

file3 will be formatted as:
directory firstsize secondsize

Simply look for large discreprencies between the first and second sizes to narrow down to a directory. Then you can hopefully identify the file.

Darrell

Re: / at 91%

Darrell Allen — Wed, 12 Sep 2001 20:34:07 GMT

Major omission! file1 and file2 must be sorted on their 2nd field before join will perform correctly. Use this:

find / -type d -xdev | xargs du -sk >/file1
sort -k2,2 file1 >sorted1

Wait awhile (30 minutes?)

find / -type d -xdev | xargs du -sk >/file2
sort -k2,2 file2 >sorted2
join -j 2 sorted1 sorted2 >file3

file3 will be formatted as:
directory firstsize secondsize

Sorry for the boo-boo!

Darrell