Strange sendmail /var problem.

[David Heath_1]

I have a user whose workstation has filled up /var very rapidly.
He recieved this error message

NOQUEUE: low on space (have 0, SMTP-DAEMON needs 101 in /var/spool/mqueue)

I thought this was due to him receiveing an email which was too large for the filesystem and so it filled it up and timed out. However, there is only the root mail account set up on the machine and this is not full.

Doing a mailq command I receive, no mail in queue.

And when I do a find for large files and compare two different find jobs I find no differences in the size of the files found even though a bdf has told me /var has grown 5% in between finds.

I am stumped and dont know what could be filling up /Var so rapidly.

Any ideas?

[Stefan Farrelly]

Are there lots of files in /var/spool/mqueue ? you could have a looping email problem which causes hundreds of files to accumulate in this dir. Check whats in /var/tmp (all temp files - can be removed).

cd /var
du -sk * | sort -n

To see which dirs under /var are big.
How big is /var/adm/sw/save ? If its big you can clean it up by doing;
swmodify -x patch_commit=true \*

[David Heath_1]

Nope.

/var/spool/mqueue is empty.

I did the swmodify and received this error:

ERROR:
The selected software to be committed was not modified. This is probably the result of none of the selected software having a patch fileset to commit

Thanks for your help though.

Thanks,

David.

[David Burgess]

If /var is growing, check /var/adm/syslog/syslog.log. If this is growing rapidly you've probably got a problem.

tail -f /var/adm/syslog/syslog.log to see what's going on.

Also go into SAM, routine tasks, system logfiles and clear down any large logfiles.

HTH,

Dave.

[David Heath_1]

Nothing in syslog only it telling me that lvol8 is full and the No queue smtp message - which i know.

Ive cleared all large log files.
But still no joy.

Thanks for your help.

[David Heath_1]

Actually something else has happened.

I deleted the syslogs maually, but just ran sam to see if I could see what was happening anywhere. I exported the display to my workstation, types sam, the sam logo appeared but then it closed and I received the following error message:

sam
cpp: error 12: Unable to access output file (check for disk error).
sam: Error: The cpp(1) command failed on file: /usr/sam/lib/C/fal.ui.

Now whats going on?
Could the machine have a bad disk?

Thanks.

[David Burgess]

ps -ef | grep syslogd and kill -HUP the pid to make sure it's writing to the syslog file. Then all should be ok with syslog. If not kill it and run /usr/sbin/syslogd -D

You might have a SAM patch the isn't configured. Check this with :-

swlist -l fileset -a state

If you find filesets in the installed state, run :-

swmodify \*

Check it worked :-

swlist -l fileset -a state

Then try running sam again.

Regards,

Dave.

[David Burgess]

How about :-

find /var -name core

Regards,

Dave.

[David Heath_1]

Cheers David, I think you miss understood me.

Syslog is being written too fine, I meant there was no useful information in there explaining the problem further, not there was nothing in there at all!!

I did the swlist command and all patches have been configured correctly. None are in the installed state.

no core files either.

Cheers for you help.

[Robin Wakefield]

Hi David,

Are you seeing any pertinent messages in /var/adm/syslog/mail.log (assuming that's where it is).

Also see:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x0be7c6af36b7d5118ff10090279cd0f9,00.html

and

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xe4b814a24fd1d4118fef0090279cd0f9,00.html

Rgds, Robin.

[David Heath_1]

Cheers Robin, but unfortuantely that doesn't help.

It doesn't seem tobe a mail problem. It just seems that /var is filling up from something but I cannot find what is causing it.

[Bill Hassell]

/var is the most critical filesystem in HP-UX (and most other Unices). The problem is that there are many different subsystems that can fill it up (and fill it quickly). You should have several hundred megs of free space in /var most of the time. First, see how full all your filesystems are:

bdf

and for just /var:

bdf /var

So the task is to locate the big directories (NEVER start looking for big files, always look for big DIRECTORIES, then look inside). Here's the way to do that:

du -kx | sort -rn > /tmp/du.var

Then look at the top of the du.var file to see what is occupying most of the space. /var should be 750 to 2500 megs in size but this depends very much on how you use your system. Once you find a big directory, use:

ll /var/what_ever | sort -rnk5 | head

to see the largest files in that directory. Note that a parent directory may be large but most of the space is in a subdirectory.

Note that sendmail will shutdown when temp space in /var is too low. It doesn't matter if the message is large or small.

The most notorious directories that fill up are:

/var/tmp
/var/mail
/var/spool
/var/adm
/var/adm/sw

Each of the directories is a good candidate for a separate mount point. By keeping these directories separate, a problem with a printer won't cause everything to stop, just spooling. And /var/adm is the logfile directory...you need to actively truncate logfiles every week.

[David Heath_1]

Thanks Bill for your help, however still no joy.

/var is set at 800 mb.

At the moment it is full.

Output from :

du -kx | sort -rn > /tmp/du.var

showed that /var/adm and /var/adm/sw as the largest directories.

In /var/adm, ll /var/adm | sort -rmk5 | head showed that ps_data was the largest file.

In /var/adm/sw, ll /var/adm/sw | sort -rnk5 | head, showed that the swagent.log was the largest file at 208 kb!! Not very big!!

Basically, I cannot find a reason as to why the filesystem is filling up or where the data is comeing from/going to. Im going to reboot the machine to try and stop ithe process and then try and free some space and see what that does.

Thanks for all your help.