Had /var filesystem full. Where to look for after effects?

[Keith Jahn]

HP UK 11.11 , LVM, online JFS. Had /var full for a few hours due to a rogue overnight print job. One very strange effect was that UID-to-name lookups by applications and system utilities returned blank usernames. Even "sendmail" was unable to verify a username and junked each mail item in turn.
We have restarted "sendmail", "syslogd" and "lpsched". Also repaired /usr/adm/wtmp which was badly corrupted by blank usernames in the first field!
The system appears healthy, but is there anywhere else to look for after-effects?

[Bill Hassell · Accepted solution]

As you have seen, /var is the most critical filesystem in HP-UX because so many processes depend on it. You can look in syslog and run dmesg to see if there are any other errors. But because so many logfiles are in /var/adm, it is possible that one or more daemons or programs quit because they could not write to the logs. A reboot is about the only guaranteed way to restart any stopped processes.

As you can see, notifications about full filesystems is critical, but preventing total failures is also important. I would str4ongly suggest creating new logical volumes for the following directories:

/var/spool/lp/request
/var/mail
/var/adm/sw
/var/tmp
/var/adm/crash

Then move the files from their current /var directories to the new volumes. Now, a rogue printer job, a flood of emails, installing new patches, users playing with the system or a system crash won't fill up /var, but instead a single subsystem (like email or printing) may be stopped but the rest of the system goes on.

[Keith Jahn]

Thanks. Yery good advice. We'll schedule a reboot and review the filesystem layout. We'll also reconsider using Openview to monitor filesystem space (and syslog etc.) because it failed too and compounded the problem.