Server dead whilst /.secure is full!!!!

[mpua]

Hi,

Some weird thing happend at my company two days ago. 

We have HPUX auditing enabled in this server (HPUX 11.23), writing the data in the a filesystem called /.secure.

We've automated the rotation of audifile so the filesystem seldom fills past 70%-80%....

Thing is two nights ago some application went out of control and began to fill the audfile with thousands of entries about mpctl syscalls (which was being audited).

The consequence of that was the FS /.secure reaching a 100% occupation, which shouldn't be that important, since a filesystem like that shouldn't  affect the others...

Well, what happened when /.secure was full was that it was impossible to log in or su to any other user other than root and that all the applications in the server (its devoted mainly to host WebLogic applications) ceased working.... when i emptied the filesystem everything started to work again.

Is there any way to avoid this??? I'm interesting in keeping the auditing enabled to monitor things but i dont want to endanger a critical server normal functioning...

[Patrick Wallek]

The best bet is to have some sort of monitoring on this file system so that you get notified when it reaches a certain utilization percentage. 

[Bill Hassell]

NEVER use / to store logs. As you have found out, filling / is a catastrophe -- it really does matter if / is full, for a lot of different reasons.

The / directory is static and should not change significantly over time. Auditing on the other hand can very easily get out of control and you want to limit the damage caused by runaway file growth by keeping logs in a non-critical directory. The first choice is to change the (very bad default) /.secure to a big, dynamic mountpoint such as /var. The man page for audsys even gives the warning. Edit the /etc/rc.config.d/auditing file to change the destination directories.