Audit trim causing swap problems

[Alexey Borchev]

System GS1280, TRU64 5.1b2, C2 security, auditing on.

I've got following problem:
- System responce time increased, i've started investigations.

1) Someone consumed 15 GB of swap space:
# swapon -s
Swap partition /dev/disk/dsk0b:
Allocated space: 3724854 pages (28.42GB)
In-use space: 2197029 pages ( 58%)
Free space: 1527825 pages ( 41%)
(Normally workload fits in memory!)

2) I see paging out, in spite while system still have 326K pages( 2 GB!) mem free.
See attachment.

3) I see unusual processes about audit:
# ps xu
USER PID %CPU %MEM VSZ RSS TTY S STARTED TIME COMMAND
root 276249 1.5 3.9 1.64G 1.3G ?? U Dec 01 1-17:12:42 /usr/sbin/audit_tool -e truncate /var/audit/aud

I suspecting this audit stuff erated my resources.
It seems it's a periodic (every 3 mths) trimming of audit logs.

Can I kill the process? Any consequences?
Can I optimize this someway?

I afraid this process will consume all swap on weekends -> system crash.

[Alexey Borchev]

I've got two newses.
Good one:
-The process of truncating auditlogs is completed during weekends.
Bad one:
- Swap space is not released back. I still
have 60 % of swap busy.

Can I see who the hell is using it?
Can I reclaim my swapspace somehow?
(Of course reboot will help, but it's not easy to get downtime.)

[Ann Majeske · Accepted solution]

auditlogtrim doesn't work right. It takes a long time and doesn't delete all of the old audit log files. Since it doesn't delete all of the files, it will take longer the next time (and even longer the time after that...). There will be a patch available in the next patch kit for V5.1b (actually two patches, one for auditlogtrim and the other for audit_tool to properly support auditlogtrim). There are no bad consequences of killing the audit_tool commands generated from the auditlogtrim cron job, especially since they're not doing what they should anyway. I've never seen this take excessive swap, though. So I don't know what to tell you about that.

I would recommend that you not try to run auditlogtrim until you get a working version. You could either wait for the next patch kit (I'm not sure when it will be available) or if you have a support contract you could contact your support person to open a case with engineering to get the patch. But, personally, I'd not bother with auditlogtrim at all. Making your own procedure would give you the flexibility to do your backups and delete old audit log files on your own schedule, according to your security policy.

How long and how you keep audit log data depends on your security policy, but a basic procedure for purging old audit data to keep your system from running out of disk space is very simple:
1) dump current audit data and start a new auditlog by running "auditd -dx"
2) back up any old audit log files you're going to delete.
3) delete the old audit log files.

One of the things that really bothers me about auditlogtrim is that it automates one simple part of the process, deleting old audit log files, but basically ignores the other parts of the process, setting up and following your security policy and backing up your audit log data to maintain the audit trail. I think you should be setting up and implementing the security policy that is best for you, not using tools that limit the possibilities.

Ann

[Mario Stargard]

We don't use the auditlogtrim utility because of this problem - rather we use our own script. We use a script called "daily" that has a pile of admin stuff in it. Attached is the snippet that does the audit logs.

Mario