topic Re: HIDS and System Audit Log in Operating System - HP-UX

HIDS and System Audit Log

Bruce Wheeler_1 — Wed, 18 Jun 2008 11:27:32 GMT

Hello,

I have nine IA servers running HP-UX 11.23 and am having a problem with HIDS v4.1 filling up the system audit logs (most of the 800mb file generated each day per server) with an 'Event=open' on '/usr/lib/nls/msg/C/strerror.cat'. Doing a grep for 'strerror.cat' piped to 'wc -l' shows a count of 4,712,468 for one days log. This event is primarily responsible for the nearly 135GB of audit logs, for all nine servers, generated in a month.

If I stop the 'idsagent' the event stops. I created a schedule that only tracks failed logins and activated it on just one server to test. After a days worth of activity, no alerts have been generated in HIDS but the audit log is filling up with the same event.

Does anyone know what the connection is between HIDS and strerror.cat?

Thanks

Re: HIDS and System Audit Log

TTr — Wed, 18 Jun 2008 12:56:41 GMT

If I understand this correctly, HIDS thinks that file opens on "strerror.cat are intrusion attempts and logs an event everytime a process opens that file. Somehow HIDS needs to be trained to ignore the "file opens" on the strerror.cat file. I don't know if this requires a change in some config file for HIDS or an HIDS upgrade to fix it.

Re: HIDS and System Audit Log

Bruce Wheeler_1 — Wed, 18 Jun 2008 13:50:42 GMT

TTr,

I am not sure if this is what you meant by your response, but HIDS is not logging the event as an "Alert", HIDS is responsible for the event and it is going to the system audit log. I have attached some tusc and audit log output.

Thanks

Re: HIDS and System Audit Log

TTr — Wed, 18 Jun 2008 16:25:59 GMT

You mentioned in your attachment that the "open" to "msgrcv" text blocks repeat. Is there anything else before the "open" statement? Something that would trigger the idsagent to go and open the strerror.cat file? If not, it looks like the idsagent has a bug and is stuck in a loop. Look for any upgrades or fixes.

If you do "strings usr/lib/nls/msg/C/strerror.cat" you will see all the error messages that are available and idsagent is "seeking and reading" each time. Do you see in the tusc output the idsagent going through each and every message in the strerror.cat file? And then repeating it for each error message? Check in the tusc listing (I know it is lenghty) but if idsagent is repeatedly reading each and every message from the strerror.cat file for no reason?

There is also the case that a valid event causes idsagent to go and open the strerror file to get the error message. Can you correlate these error messages to any events/alerts?. But it would be very inefficient to do it this way, it would be easier to read the entire file in and use each error message accordingly. That's why I suggested the possibility of needing a patch.

Re: HIDS and System Audit Log

TTr — Wed, 18 Jun 2008 16:38:40 GMT

On another thought, the strerror file is opend by just about every process in the system for picking up the error strings. And maybe this logging is normal based on the debugging and verbosity level of idsagent. Check what they are set to.

Re: HIDS and System Audit Log

Dennis Handly — Wed, 18 Jun 2008 17:06:25 GMT

>TTr: the strerror.cat file is opened by just about every process in the system for picking up the error strings.

Right. It is used if you call perror(3) and strerror(3).

Re: HIDS and System Audit Log

Bruce Wheeler_1 — Wed, 18 Jun 2008 19:08:02 GMT

>TTr: And maybe this logging is normal based on the debugging and verbosity level of idsagent. Check what they are set to.

Can you explain this a little better?

Thanks

Re: HIDS and System Audit Log

TTr — Wed, 18 Jun 2008 20:43:27 GMT

Check with what parameters idsagent is started.

For example

idsagent -c 2 -d

"-c 2" gives you error messages, and high level and verbose log messages.

"-d" enables debug messages.

I couldn't find any settings in /etc/rc.config.d/ so if the verbosity and debugging are changed, they are in /sbin/init.d/idsagent. Look towards the bottom, the default line to start the idsagent should be

su - ids -c "cd /opt/ids/bin ; ./idsagent -a" 2>&1

Also check if somebody has manually restarted the idsagent with the debugging paramaters on.

Re: HIDS and System Audit Log

Bruce Wheeler_1 — Mon, 23 Jun 2008 18:32:23 GMT

TTr: Attached is some short output with debugging enabled by starting the idsagent per the example you gave "idsagent -c2 -d -e -l /tmp/ids_debug_logfile.txt". The idsagent normally starts with 'su - ids -c "cd /opt/ids/bin ; ./idsagent -a" 2'. It does appear that it may be stuck in a loop. I already have PHKL_34798 installed and there are no other new patches available for HIDS.

Re: HIDS and System Audit Log

Dennis Handly — Tue, 24 Jun 2008 04:15:32 GMT

>It does appear that it may be stuck in a loop.
rcm_receive_message: nonblocking read returned no message.

Well, it seems to not be getting any messages. But there should be some type of sleep between the reads.

Re: HIDS and System Audit Log

TTr — Tue, 24 Jun 2008 11:53:58 GMT

I still think the logging is normal based on the frequency of the strerror.cat file. The issue naow is to suppress the logging of that event. I had not done HIDS for a few years and I just installed HIDS 4.01 on my personal workstation but it needs java5 and am downloading now.

Re: HIDS and System Audit Log

TTr — Fri, 27 Jun 2008 11:31:22 GMT

I finally got IDS going between two HP-UX 11.00 boxes, one managment station, one idsagent. I did not observe the error yes, maybe it is still early.

However I noticed something else that was very interesting. On the management station and many other servers I checked, the access date "ll -u /usr/lib/nls/C/strerror.cat" was as old as the uptime (reboot) of the server.

On the ids agent box, the access date "ll -u /usr/lib/nls/C/strerror.cat" was always current even when nothing was happening. I was the only one logged in on this box. When I stopped the idsagent the access date also stopped being current.

So does idsagent have a bug and is doing itself in by first keeping the strerror.cat file current and then reporting it or is that normal for ids? If it is normal then there must be a way to turn it off. Look at each schedule and click on the details panel and check the TEMPLATE details in the listing to see if the strerror.cat or /usr/lib has been added in the rules.

Re: HIDS and System Audit Log

Pierre Pasturel — Fri, 27 Jun 2008 20:44:23 GMT

Hi Bruce -

Does your HIDS agent's error.log file (by default, at /var/opt/ids/error.log on the agent server) contain any errors?

I checked one of my IA 11.23 systems and the date on strerror.cat is from 2003 and I know I've run the agent on that system since then :)

Pierre

Re: HIDS and System Audit Log

Pierre Pasturel — Fri, 27 Jun 2008 20:55:17 GMT

Actually, yes, I do see the access date change when idsagent is running. Let me investigate further.

Pierre

Re: HIDS and System Audit Log

Pierre Pasturel — Fri, 27 Jun 2008 21:38:56 GMT

Ok, I believe I have found the call to strerror() by idsagent that is generating all these open audit records in your System Audit Log.

AudFilter is an add-on product that allows you to filter out any open(2) events invoked by user "ids." User "ids" is the uid under which HIDS agent processes run. Unfortunately, AudFilter is available starting with 11.31, not 11.23.

Please go through your regular support channel if a fix to this problem is critical. We normally address fixes in the next HIDS release that is tentatively slated for the end of this year.

Pierre

Re: HIDS and System Audit Log

TTr — Fri, 27 Jun 2008 22:43:30 GMT

Pierre, can you elaborate why idsagent goes out to open the strerror.cat file in 11.23 and not in 11.00?

As Dennis pointed out regular processes make a perror(3) and strerror(3) calls and aparently the kernel does not open the strerror.cat file, it has cached the error (or doesn't it?). Does idsagent interpret the calls as file opens and reports them as events? Is it how idsagent sits in the kernel in 11.23 as opposed to 11.00 or was there a change in the code?

Re: HIDS and System Audit Log

Dennis Handly — Fri, 27 Jun 2008 23:40:09 GMT

>TTr: can you elaborate why idsagent goes out to open the strerror.cat file in 11.23 and not in 11.00?

It may want to produce more informative messages?

>regular processes make a perror(3) and strerror(3) calls and aparently the kernel

The kernel isn't involved here, only libc.

>does not open the strerror.cat file, it has cached the error (or doesn't it?).

This may depend on the locale? It doesn't seem like it since the path has "C/".

>Does idsagent interpret the calls as file opens and reports them as events?

Well, there are file opens. It seems to keep reporting these errno values:
EAGAIN Resource temporarily unavailable
ENOMSG No message of the desired type

Re: HIDS and System Audit Log

Pierre Pasturel — Sat, 28 Jun 2008 04:58:40 GMT

idsagent is invoking strerror(ENOMSG) to build a debug string every time it checks for messages from its subprocesses and there are no messages. Unfortunately, strerror() is called even when idsagent is not run in debug mode. We can address this in the next release of HIDS.

> can you elaborate why idsagent goes out to open the strerror.cat file in 11.23 and not in 11.00?

It is possible that the HIDS code was different in 11.0. BTW, HIDS is no longer supported on 11.0 and has not been delivered for 11.0 since HIDS v3.0 (released Dec 2004).

>Does idsagent interpret the calls as file opens and reports them as events?

HIDS does not process audit records that are triggered by a process running as user "ids." For processes running under different users, an open(2) for modification, say, on a read-only file, can trigger a HIDS alert.

Pierre

Re: HIDS and System Audit Log

TTr — Sat, 28 Jun 2008 08:48:37 GMT

Pierre and Dennis, good info!

Bruce, it looks like you are stuck with it until the next release.

Re: HIDS and System Audit Log

Bruce Wheeler_1 — Wed, 02 Jul 2008 10:36:51 GMT

Pierre,

Sorry for the delay. Can you tell me why the audit log shows 'root' as the user instead of 'ids'? A 'ps -ef |grep ids' shows 'ids' as the user of all HIDS processes. I could then turn off auditing for 'ids' using 'audusr' (yes we are in trusted mode).

Actual Audit Log sample (same as attachment above):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ Event=open; User=root; Real Grp-ids; Eff.Grp=ids; ]
RETURN_VALUE 1 = 8;
PARAM #1 (file path) = 0 (cnode) ;
0x4000000a (dev) ;
75 (inode) ;
(path) = /usr/lib/nls/msg/C/strerror.cat
PARAM #2 (int) = 0
PARAM #3 (int) = 12
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Thanks,
Bruce