var full only at shutdown

[Fred Martin_1]

The /var file system is at 56% (377425 available) but when I do a "shutdown -h 0" this is what I get:

System shutdown time has arrived
vxfs: mesg 001: vx_nospace - /dev/vg00/lvol9 file system full (1 block extent)

System halted.

The vxfs message appears six times.
I don't see any file system full during operating times, only at shutdown. Any idea what that's about?

[harry d brown jr]

Check your shutdown scripts. Sounds like you have a "rogue" process doing a lot of logging!

Look at /var/tmp, which is most likely the directory under /var being "filled"

live free or die
harry

[John Bolene]

something in the shutdown that writes a bunch to the syslog?

have you checked /var/adm/syslog/OLDsyslog.log for any indications?

[Paula J Frazer-Campbell]

Hi Fred

/var/tmp is used at shutdown time to store info that the system would like after restart, so this is most likly the reason var is being reported full.

Clean the var/adm and tmp dir

wtmp and btmp can get a bit big.

Hth

Paula

[harry d brown jr]

Fred, after the shutdown, boot into single user mode and mount /var then check the filesystem /var and /var/tmp.

live free or die
harry

[James R. Ferguson]

Hi Fred:

Examine (and/or post here) '/etc/rc.log.old'. This will show you the *shutdown* script output.

Do you automatically clean 'var/tmp' during startup? You will if 'CLEAR_TMP=1' in '/etc/rc.config.d/clean_tmps'.

Regards!

...JRF...

[Fred Martin_1]

I don't clean out /var/tmp normally because an application (acckk) puts important files in there. But I do manually clean it up. As I said, there was 370K bytes in there at shutdown, which produced this at the end of the log:

Unmount file systems
Output from "/sbin/rc0.d/K900localmount stop":
----------------------------
umountall: umount : has failed.
umountall: diagnostics from umount
umount: cannot unmount /home : Device busy
umountall: umount : has failed.
umountall: diagnostics from umount
umount: cannot unmount /usr : Device busy
umountall: umount : has failed.
umountall: diagnostics from umount
umount: cannot unmount /var : Device busy

*******************************************
HP-UX run-level transition completed
Thu Sep 19 17:31:26 EDT 2002
*******************************************

[James R. Ferguson]

Hi Fred:

It sounds like the application that writes to 'var/tmp' is the culprit. Does it monitor other resources and log their health into a file in 'var/tmp/'? Do you have kill scripts to gracefully stop this application?

...JRF...

[Fred Martin_1]

I don't think the app is causing this. The files written to /var/tmp are created by users through the application (a Progress database). For the above shutdown, all users were out and the database was shut down before the system shut down.

I checked to verify that I was the only user logged in (at console).

Obviously though, something was in those file systems. I'll poke around to see if I can find it.

[harry d brown jr]

Fred,

If you don't already, install lsof, and verify that no application has any files in /var/tmp open.

Have you tried the boot into single user mode??

btw, you could have a clean up script at boot time cleaning up /var/tmp!

live free or die
harry

[Fred Martin_1]

I'll have to wait for a weekend or overnight to try the single-user mode thing. I'll post the result....thanks.