System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Server dead whilst /.secure is full!!!!

mpua
Frequent Advisor

Server dead whilst /.secure is full!!!!

Hi,

 

Some weird thing happend at my company two days ago. 

We have HPUX auditing enabled in this server (HPUX 11.23), writing the data in the a filesystem called /.secure.

 

We've automated the rotation of audifile so the filesystem seldom fills past 70%-80%....

Thing is two nights ago some application went out of control and began to fill the audfile with thousands of entries about mpctl syscalls (which was being audited).

The consequence of that was the FS /.secure reaching a 100% occupation, which shouldn't be that important, since a filesystem like that shouldn't  affect the others...

 

Well, what happened when /.secure was full was that it was impossible to log in or su to any other user other than root and that all the applications in the server (its devoted mainly to host WebLogic applications) ceased working.... when i emptied the filesystem everything started to work again.

 

Is there any way to avoid this??? I'm interesting in keeping the auditing enabled to monitor things but i dont want to endanger a critical server normal functioning...

2 REPLIES
Patrick Wallek
Honored Contributor

Re: Server dead whilst /.secure is full!!!!

The best bet is to have some sort of monitoring on this file system so that you get notified when it reaches a certain utilization percentage. 

 

 

Bill Hassell
Honored Contributor

Re: Server dead whilst /.secure is full!!!!

NEVER use / to store logs. As you have found out, filling / is a catastrophe -- it really does matter if / is full, for a lot of different reasons.

 

The / directory is static and should not change significantly over time. Auditing on the other hand can very easily get out of control and you want to limit the damage caused by runaway file growth by keeping logs in a non-critical directory. The first choice is to change the (very bad default) /.secure to a big, dynamic mountpoint such as /var. The man page for audsys even gives the warning. Edit the /etc/rc.config.d/auditing file to change the destination directories.



Bill Hassell, sysadmin