HIDS and System Audit Log

[Bruce Wheeler_1]

Hello,

I have nine IA servers running HP-UX 11.23 and am having a problem with HIDS v4.1 filling up the system audit logs (most of the 800mb file generated each day per server) with an 'Event=open' on '/usr/lib/nls/msg/C/strerror.cat'. Doing a grep for 'strerror.cat' piped to 'wc -l' shows a count of 4,712,468 for one days log. This event is primarily responsible for the nearly 135GB of audit logs, for all nine servers, generated in a month.

If I stop the 'idsagent' the event stops. I created a schedule that only tracks failed logins and activated it on just one server to test. After a days worth of activity, no alerts have been generated in HIDS but the audit log is filling up with the same event.

Does anyone know what the connection is between HIDS and strerror.cat?

Thanks

[TTr]

If I understand this correctly, HIDS thinks that file opens on "strerror.cat are intrusion attempts and logs an event everytime a process opens that file. Somehow HIDS needs to be trained to ignore the "file opens" on the strerror.cat file. I don't know if this requires a change in some config file for HIDS or an HIDS upgrade to fix it.

[Bruce Wheeler_1]

TTr,

I am not sure if this is what you meant by your response, but HIDS is not logging the event as an "Alert", HIDS is responsible for the event and it is going to the system audit log. I have attached some tusc and audit log output.

Thanks

[TTr]

You mentioned in your attachment that the "open" to "msgrcv" text blocks repeat. Is there anything else before the "open" statement? Something that would trigger the idsagent to go and open the strerror.cat file? If not, it looks like the idsagent has a bug and is stuck in a loop. Look for any upgrades or fixes.

If you do "strings usr/lib/nls/msg/C/strerror.cat" you will see all the error messages that are available and idsagent is "seeking and reading" each time. Do you see in the tusc output the idsagent going through each and every message in the strerror.cat file? And then repeating it for each error message? Check in the tusc listing (I know it is lenghty) but if idsagent is repeatedly reading each and every message from the strerror.cat file for no reason?

There is also the case that a valid event causes idsagent to go and open the strerror file to get the error message. Can you correlate these error messages to any events/alerts?. But it would be very inefficient to do it this way, it would be easier to read the entire file in and use each error message accordingly. That's why I suggested the possibility of needing a patch.

[TTr]

On another thought, the strerror file is opend by just about every process in the system for picking up the error strings. And maybe this logging is normal based on the debugging and verbosity level of idsagent. Check what they are set to.

[Dennis Handly]

>TTr: the strerror.cat file is opened by just about every process in the system for picking up the error strings.

Right. It is used if you call perror(3) and strerror(3).

[Bruce Wheeler_1]

>TTr: And maybe this logging is normal based on the debugging and verbosity level of idsagent. Check what they are set to.

Can you explain this a little better?

Thanks

[TTr]

Check with what parameters idsagent is started.

For example

idsagent -c 2 -d

"-c 2" gives you error messages, and high level and verbose log messages.

"-d" enables debug messages.

I couldn't find any settings in /etc/rc.config.d/ so if the verbosity and debugging are changed, they are in /sbin/init.d/idsagent. Look towards the bottom, the default line to start the idsagent should be

su - ids -c "cd /opt/ids/bin ; ./idsagent -a" 2>&1

Also check if somebody has manually restarted the idsagent with the debugging paramaters on.

[Bruce Wheeler_1]

TTr: Attached is some short output with debugging enabled by starting the idsagent per the example you gave "idsagent -c2 -d -e -l /tmp/ids_debug_logfile.txt". The idsagent normally starts with 'su - ids -c "cd /opt/ids/bin ; ./idsagent -a" 2'. It does appear that it may be stuck in a loop. I already have PHKL_34798 installed and there are no other new patches available for HIDS.

[Dennis Handly]

>It does appear that it may be stuck in a loop.
rcm_receive_message: nonblocking read returned no message.

Well, it seems to not be getting any messages. But there should be some type of sleep between the reads.

[TTr]

I still think the logging is normal based on the frequency of the strerror.cat file. The issue naow is to suppress the logging of that event. I had not done HIDS for a few years and I just installed HIDS 4.01 on my personal workstation but it needs java5 and am downloading now.

[TTr]

I finally got IDS going between two HP-UX 11.00 boxes, one managment station, one idsagent. I did not observe the error yes, maybe it is still early.

However I noticed something else that was very interesting. On the management station and many other servers I checked, the access date "ll -u /usr/lib/nls/C/strerror.cat" was as old as the uptime (reboot) of the server.

On the ids agent box, the access date "ll -u /usr/lib/nls/C/strerror.cat" was always current even when nothing was happening. I was the only one logged in on this box. When I stopped the idsagent the access date also stopped being current.

So does idsagent have a bug and is doing itself in by first keeping the strerror.cat file current and then reporting it or is that normal for ids? If it is normal then there must be a way to turn it off. Look at each schedule and click on the details panel and check the TEMPLATE details in the listing to see if the strerror.cat or /usr/lib has been added in the rules.

[Pierre Pasturel]

Hi Bruce -

Does your HIDS agent's error.log file (by default, at /var/opt/ids/error.log on the agent server) contain any errors?

I checked one of my IA 11.23 systems and the date on strerror.cat is from 2003 and I know I've run the agent on that system since then :)

Pierre

[Pierre Pasturel]

Actually, yes, I do see the access date change when idsagent is running. Let me investigate further.

Pierre

[Pierre Pasturel]

Ok, I believe I have found the call to strerror() by idsagent that is generating all these open audit records in your System Audit Log.

AudFilter is an add-on product that allows you to filter out any open(2) events invoked by user "ids." User "ids" is the uid under which HIDS agent processes run. Unfortunately, AudFilter is available starting with 11.31, not 11.23.

Please go through your regular support channel if a fix to this problem is critical. We normally address fixes in the next HIDS release that is tentatively slated for the end of this year.

Pierre

[TTr]

Pierre, can you elaborate why idsagent goes out to open the strerror.cat file in 11.23 and not in 11.00?

As Dennis pointed out regular processes make a perror(3) and strerror(3) calls and aparently the kernel does not open the strerror.cat file, it has cached the error (or doesn't it?). Does idsagent interpret the calls as file opens and reports them as events? Is it how idsagent sits in the kernel in 11.23 as opposed to 11.00 or was there a change in the code?

[Dennis Handly]

>TTr: can you elaborate why idsagent goes out to open the strerror.cat file in 11.23 and not in 11.00?

It may want to produce more informative messages?

>regular processes make a perror(3) and strerror(3) calls and aparently the kernel

The kernel isn't involved here, only libc.

>does not open the strerror.cat file, it has cached the error (or doesn't it?).

This may depend on the locale? It doesn't seem like it since the path has "C/".

>Does idsagent interpret the calls as file opens and reports them as events?

Well, there are file opens. It seems to keep reporting these errno values:
EAGAIN Resource temporarily unavailable
ENOMSG No message of the desired type

[Pierre Pasturel]

idsagent is invoking strerror(ENOMSG) to build a debug string every time it checks for messages from its subprocesses and there are no messages. Unfortunately, strerror() is called even when idsagent is not run in debug mode. We can address this in the next release of HIDS.

> can you elaborate why idsagent goes out to open the strerror.cat file in 11.23 and not in 11.00?

It is possible that the HIDS code was different in 11.0. BTW, HIDS is no longer supported on 11.0 and has not been delivered for 11.0 since HIDS v3.0 (released Dec 2004).

>Does idsagent interpret the calls as file opens and reports them as events?

HIDS does not process audit records that are triggered by a process running as user "ids." For processes running under different users, an open(2) for modification, say, on a read-only file, can trigger a HIDS alert.

Pierre

[TTr]

Pierre and Dennis, good info!

Bruce, it looks like you are stuck with it until the next release.

[Bruce Wheeler_1]

Pierre,

Sorry for the delay. Can you tell me why the audit log shows 'root' as the user instead of 'ids'? A 'ps -ef |grep ids' shows 'ids' as the user of all HIDS processes. I could then turn off auditing for 'ids' using 'audusr' (yes we are in trusted mode).

Actual Audit Log sample (same as attachment above):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ Event=open; User=root; Real Grp-ids; Eff.Grp=ids; ]
RETURN_VALUE 1 = 8;
PARAM #1 (file path) = 0 (cnode) ;
0x4000000a (dev) ;
75 (inode) ;
(path) = /usr/lib/nls/msg/C/strerror.cat
PARAM #2 (int) = 0
PARAM #3 (int) = 12
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Thanks,
Bruce

[Dennis Handly]

>Can you tell me why the audit log shows root as the user instead of ids? A 'ps -ef |grep ids' shows ids as the user of all HIDS

It may be related to real vs effective user.
Note: instead of grep use:
ps -fu ids
Then:
UNIX95=1 ps -fu ids

[Bruce Wheeler_1]

Dennis,

That might be the reason.
ps -fu ids shows 'idssysdsp' as user 'ids' just like 'idscor', 'idskerndsp' and 'idsagent'.
UNIX=95 ps -ef |grep ids shows that 'idssysdsp' is user 'root' while the others are 'ids'.
The HIDS v4.1 Admin Guide does state that 'idssysdsp' must be user:root group:ids and is readable and executable by root. I am guessing that 'idssysdsp' must be tied to the 'open' event in the audit log. If this is the case then it looks like I am out of luck, for now.

Thanks,
Bruce

[Pierre Pasturel · Accepted solution]

> Can you tell me why the audit log shows 'root' as the user instead of 'ids'?
> [ Event=open; User=root; Real Grp-ids; Eff.Grp=ids; ]

The "User=" entry in the system audit record is the login user for that session. If you login as root to start the agent, this entry will be set to root and not user ids. Unfortunately there is no way to filter out records based on effective or real uid or gid.

There is a workaround that might work for you. The HIDS Admin Guide recommends that you first login as root and then su to user ids in order to avoid making the ids account a login account with a potentially weak password. However, you can decide to make the "ids" account a login account (with a password as strong as root's or perhaps even the same as root's password). If so, then login directly as user "ids" instead and start the agent by running "/opt/ids/bin/idsagent -a" (see p. 60 of the V4.1 Admin Guide). The other option is to run "/sbin/init.d/idsagent start" as user "ids" but you will be prompted 3 times for a password due to the three su's in that script, twice for root's password and once for ids' password.

You should then be able to use "audusr -d ids" to filter out the "User=ids" system audit records.

FYI, the idssysdsp is a setuid-root program that needs to have an effective uid of root when accessing root owned files; during all other times, it runs with effective uid of "ids."

Pierre

[Bruce Wheeler_1]

Pierre,

Logging in as user 'ids' seems to be the ticket. It has dramatically reduced the growth rate of my audit logs. Still, it would be nice to have this remedied in the next release as I will need to remove startup from /sbin/rc3.d dir and start the agents manually after a reboot. Also, I am not auditing user 'ids' which may be a problem.

Thanks everyone for your help,
Bruce

[Bruce Wheeler_1]

See my final comments above. Points have been assigned. Thanks.