IDS9000 java CPU hog

[Kurt Renner]

I am evaluating IDS9000. My IDS management server is running HP-UX 11.11 and is on a 2-way K570 with 1.2GB RAM. I have 2 clients configured with IDS9000 on them. One is the management server itself, and the other is a test workstation. The version of IDS9000 I am running is: "J5083AA B.02.01.32 HP IDS 9000 B.02.01" I have java 1.3.01 installed (1.4.x is not supported)

When I run "idsgui" as user "ids", the application takes a long time to start up. When it does finally start up (> 4 minutes), and load all the log entries, the "java" process on my machine consumes > 90% of the CPU with the gui sitting idle.

Is this typical, or is there something I can do to improve the performance of Java/IDS9000?

Thanks in advance!
Kurt

[Kent Ostby]

Rummaging around, I found the following in an internal document:

The 3 main reasons to fall in this situation are:

1- enable "buffer overflow" template

2- enable "Race condition" template

3- use the "blockig mode" (IDDS_MODE = 2 in idf.cf).

There is no rules to determine which of the 3 above conditions because
it depends of the system activity that may be very different from system
to system.
Nevertheless by experience, it appears that items 1 & 2 are generaly
the cause of high CPU usage.

IDDS_MODE is also to be taken in account: The ids.cf file gives some
explanations on how to determine the IDDS_MODE value.

# Recommended Values:
# IDDS_MODE 2 the default setting. Provides greater security over
# performance
# IDDS_MODE 3 Provides performance at the expense of lost audit
# data which may lead to missed intrusion attempts.

Hope this helps,

Kent M. Ostby

[Steven E. Protter]

When I configured IDS9000 servers in Internet Security Class(Feb 2002, Mountain View Ed Center D220), there were several complaints from IDS-9000 users about CPU pigginess.

We stayed after class and matches the server settings of one of the clients and found we pretty much locked up the test center box.

If you collect too much information on a slower box, you will slow it down to a crawl.

The advice I saw was to dedicate a box to security checking so the CPU slowdown doesn't matter. Every other machine in the shop was a client.

I have good reports on this technique. Our boxes are fast enough to handle IDS-9000 server.

SEP

[Kurt Renner]

Kent,
Thanks for the incredibly quick response!

I had seen in the documentation the fact that race conditions and buffer overflows add significant cpu overhead, and have not been tracking those conditions. I was not however aware of the IDSS_MODE options. The problem I have with implementing mode 3 is that in the event of an intrusion, Murphy's law is sure to take effect.

It appears to me that it is the gui/java itself consuming CPU. I say that because the idssysds and idsagent processes are only consuming 4 and 3% of the cpu respectively. During experimentation, I enabled buffer overflows and race conditions, and found that causes these process to consume significantly more CPU resources.

... but of course, I could be all wet....

Anyone know of any Java-specific tuning that can be done, or any patches that might help relieve the CPU consumption?

Thanks,
Kurt

[Kurt Renner]

Stephen,
I was afraid that was going to be the answer. We have a serious consolidation effort under way. A dedicated server will not be received positively. If there are other suggestions for performance improvement so a separate server is not required, I would love to hear it.

[Kent Ostby]

Kurt --

Three things.

1) One patch I found that relates to performance improvement on 11.11 with JFS is: PHKL_29110.

2) I found a reference to a known performance bug in the 1.3 version convertors (Sun Bug id: 4311984) that is fixed in version 1.4 (and yes I see your note about it not being supported).

3)I found a note that some users testing shows that threads run as root run faster then threads .

A workaround to this would be to grant RTSCHED privledge to the non-root user (or more specifically the group of the non-root user by using the setprivgrp command:

setprivgrp groupname RTSCHED

This command seems to raise the non-root user to the level of root in terms of performance at least in the java application
that was tested.

Best regards,

Kent M. Ostby

[Kurt Renner]

Kent,
I tried each of your suggestions. Before I posted the original, I tried the latest java version, that's how I found out it was not supported (a message was displayed stating anything greater than 1.3.0 WAS, but then the next line said 1.4.x was not)

Unfortunately, the patch, and the RTPRIO privilege appears to have had little or no effect. CPU usage while the IDS gui sits idle:

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 pts/ta 3360 ids 152 20 250M 70700K run 4:29 131.88 131.65 java

Thanks for your attempts. I appreciate it.

I will likely put out a separate posting on this sometime, but does anyone have any idea as to how widely IDS9000 is used? In other words, If we were to implement this, would we likely be one of a dozen companies using this product, or would we more likely be one of hundreds or thousands? My gut feel from research I've done trying to locate IDS questions/answers, etc is that there are not a lot of companies using it... at least not in production. Anyone have other thoughts?

[Pierre Pasturel]

Kurt -

Have you installed all the HP-UX patches required for Java *before* installing Java 1.3.1?

You can find the patches you need by going to
http://www.hp.com/products1/unix/java/patches/index.html

Pierre

[Kurt Renner]

I think we may have a bug in the IDS/9000 - System Manager GUI.

What I have done:

1. Installed all patches as Pierre suggested
2. Un-installed "B9788AA 1.3.1.01.00release Java 2 SDK 1.3 for HP-UX (700/800), PA1.1 + PA2.0 Add On"
3. Re-installed "B9788AA 1.3.1.01.00release Java 2 SDK 1.3 for HP-UX (700/800), PA1.1 + PA2.0 Add On"
4. Applied Kernel parameter changes suggested by the "HPjconfig" utility http://www.hp.com/products1/unix/java/java2/hpjconfig/index.html

What I observed:
1. When starting up "idsgui", top still reported over 100% CPU utilization for the "java" process. I thought maybe I had too many things for IDS9000 to check (everything but buffer overflows and race conditions), so I de-selected everything but "Monitor logins/logouts", and applied the modified schedule to my test servers. I watched the CPU consumption for the "java" process gradually to down to a utilization level of 3-4%.

2. I quit the idsgui application and re-started it to see if I had indeed found the source of the high CPU consumption. When I re-started idsgui, I found the "java" process again consumed over 100% CPU during startup, and continued even after the event population for each client was complete. I have found that by hitting the "Status" button, and forcing the gui to re-populate the entries per client, the cpu utilization of the "java" process over a period of about a minute will decrease to 3-5% utilization.

3. I then re-enabled all the checks in the original schedule, and saw the same scenario play out (extremely high CPU consumption). After hitting the "Status" button on the gui, java ran as a normal process should again.

In each of my test scenarios, both CPU's were consumed to a point that the total %idle metric of the machine was 0% until either a new schedule was activated, nor the "Status" button was hit.

I will open a call with HP to report this as a possible bug.

Thanks to all for the many useful suggestions!

Kurt

[Pierre Pasturel]

Kurt -

Is your GUI configured to do an auto-status and/or auto-sync of your agents on startup? If so, will turning either or both of them off (in the preferences menu) and then restarting the GUI result in the GUI consuming low CPU?

Also, how many agents are you managing in your GUI?

The templates you have configured in your schedule should have nothing to do with the CPU usage by the gui.

Please do submit this problem to HP for review.

thanks,
Pierre

[Kurt Renner]

Both "Automatic Startup Status Poll" and "Automatic Startup Alert Synchronization" were enabled.

I have only 2 clients defined at this time... more to come.... soon.

I agree the problem is not with the templates. Java only appears to be used to provide the gui front-end, and java is the one process that has been hogging the CPU.

I think I have a resolution to the problem now. I did open a call with HP today, and have been working on this the biggest part of the day. I found that I was not running the latest version of IDS 9000, so I upgraded from B.02.01.32 to B.02.02.16. That in and of itself did not solve the problem. I then upgraded Java from 1.3.1.01.00 to 1.4.2.00.00 (though only 1.3.x is officially supported). The Java upgrade did not solve the problem either. Next I messed around with the schedules some, and now the CPU consumption issue is gone. It still consumes 100% upon startup, but gradually decreases to 1-5% after about a minute after the application is fully initialized.

I'm not sure EXACTLY what caused the symptom to go away. HP indicated that a similar issue was reported a couple of months ago, though not identical to this one. The labs are supposed to be looking into it, and trying to replicate it.

One thing for certain is that the combination of the newer version of IDS9000 and Java 1.4 performs better than the older combination.


Again, Thanks to all for your help.
Kurt Renner

[Pierre Pasturel]

Kurt -

fyi, I experimented a bit with IDS V2.2 using JDK 1.3.1 on HP-UX 11.0, and found that the CPU utilization goes to 100% on startup, regardless of auto-startup/auto-sync modes, and gradually decreases to < 1% after a couple of minutes.

When using Java 1.3.1, did you allow the GUI plenty of time (> 4 minutes) to lower its CPU utilization? Or did it stay at 100% even after waiting a few minutes and the CPU utilization did not decrease unless you did a an agent status?

Pierre

[Kurt Renner]

I'm seeing the same thing on my system now. That I can live with and expect.

I had let it run for 10 minutes or more at one point AFTER it appeared to be fully initialized. Java appeared to be stuck in a tight loop until after it was told to do something to refresh the clients' status (such as hitting the status button, or re-loading a schedule). Then it would go down to a normal consumption percentage over a period of 30 seconds or more, and would work normally from there.

Thanks for your attetion to this Pierre. If you want the call ID # let me know, and I can send it to you so you can review.