audomon problems

[Glenn S. Davidson]

It appears as though audomon is spawning a new process every 2 minutes until it fills up the process table and then I have bigger problems.

I have 2 servers doing this. I unconverted them both. I made sure .secure was gone. I removed the .ataids and .cronaids directories. I checked the password file (just to be sure) and modified /etc/rc.config.d/auditing to make sure it wouldn't restart when the server was rebooted.
I then rebooted them. One has stopped spawning the audomon processes but one still spawns then. I went back thru the server to see if I missed something but I don't think I have.
I don't like trusted systems but it's mandated that I do it so I'm stuck. I'm going to be checking for patches and have someone else check the server to see what I missed.

Thanks for your assistance.

Glenn

[doug hosking]

I don't recall ever seeing such a thing. What HP-UX release are you running, and what is the output from 'what /usr/sbin/audomon' ? The only signicant changes to audomon in recent memory had to do with making it happy with file systems > 4 GB in size, but that was several years ago.

It appears that the only time audomon will fork is when it first starts. As part of becoming a daemon, if will fork twice. The children immediately exit, so that shouldn't be a problem unless they're not being properly wait()ed for.

Can you tell anything interesting from ps output re who the parent process is? Is there anything useful on the console, in the syslog file or in the sam log files?

'chmod -x /usr/sbin/audomon' and a reboot should be enough to stop the problem, but it's not clear why it would happen in the first place.

[Keith Buck]

I have seen a similar problem (though not identical). The problem I saw was related to audomon not stopping when the stop script was called.

That problem was fixed by PHKL_26059 (11.00 only)

Without this patch the kernel may stop sending signals under high stress conditions, which can cause audomon processes to not be killed.
(This patch requires a reboot)

The "high stress conditions" we experienced were part of HP-UX Bastille stress testing where we converted to a trusted system and back again hundreds of times.

As Doug says, this doesn't explain the cause of the problem or how the processes are spawning automatically.

Note also that it's safer to use SAM or Bastille to enable auditing, as there are some additional checks which may not be in place if you edit the rc.config.d file directly.

If it's still happening, try to get some ps output as Doug suggests.

-Keith

[Glenn S. Davidson]

OK, I tried several things as I included in the original post but HP said that if I convert a server using SAM then I should also unconvert the server using SAM.
For the server that continued to spawn processes I had to re-convert it to a trusted host with SAM then unconvert it with SAM. I didn't even turn it on. I just converted then unconverted it. This seems to have fixed the problem. I have left them off for now to see if I can find some more patches regarding this process.

To answer some of your questions:
L2000 running HP 11.0
audomon Ver 82.1
all the additional processes were owned by init
The only thing that showed up in the syslog was an inetd registrar/tcp connection and vmunix: kthread: table is full

Reply if something here jars your memory or if there is something in addition I could look at.

Glenn

[doug hosking]

PHCO_25796 *might* be related to this, but it's a bit of a long shot. (If init hangs, it
can't reap orphans of other processes, which
could eventually lead to process/thread table full problems, etc.)

82.1 won't be happy when run on a configuration where the file system on which your audit files reside is > 4 GB in size. That indirectly affects some of the internal sleep logic, but I don't yet see how that would cause the problem you are seeing. HP-UX 11.11 has fixes for the 4 GB file system problem.

Can you reproduce this problem when the audit file system is < 4 GB in size?

[Glenn S. Davidson]

OK, so I'm not so lucky as to have figured this out. I just had one of the servers go through this again. How disheartening!

Doug, I haven't forgotten you! I want to see if the patch solves this before I assign you points. I don't want to prematurely discount the possibility that the patch could do some good and not assign you the proper due.

If anyone else has any ideas on this that would be great!

I would like to add that I have created a script to stop and restart the auditing process using audsys -f and audsys -n in order to move the audit files off before they screw up the server. I've also changed the default startup arguments to "-p 20 -t 1 -w 100 -o /var/adm/syslog/syslog.log"

Hopefully that isn't what is causing this.

Glenn

[Darren Prior]

Hi Glenn,

I'm not sure if it's a good idea to use -o ...syslog.log. There is potential for audomon to write to the file at the same time as syslogd. If you want the messages to go to syslog I'd suggest perhaps piping through logger.

regards,

Darren.

[Glenn S. Davidson]

OK, I figured out the problem and it had nothing to do with the auditing system or the audomon daemon. I found out that when you invoke audomon from the command line it spawns another process leaving the current one in tact. I also found out that our VPO administrator was having problems with event storms and used an old version of our script in an automatic action to try to stop the storms. So that's the story! It's fixed now!

Darren, exactly how would I pipe audomon through logger? I have used the logger command in other situations but I'm not sure how I would go about this.

Thanks again for all your assistance.

Glenn