Delayed Event Entries

[pedro-chicago]

Recently I have noticed that we are receiving events from servers much later than when they actually occur.

For example, over the past few days, a server has sent out disk space threshold alerts. These events are recorded in the local System Event Log on the server. The dates and times of these events are below.

10/26/2005 2:46:46 AM Rising Threshold Passed.
10/25/2005 7:49:38 PM Critical Rising Threshold Passed.
10/25/2005 2:46:39 AM Rising Threshold Passed.
10/24/2005 7:47:23 PM Critical Rising Threshold Passed.
10/24/2005 2:46:20 AM Rising Threshold Passed.
10/24/2005 2:46:20 AM Rising Threshold Passed.

These same events are in the SIM database (HP SIM 5.0 with MIB Update and TrustMX.jar fix on Windows Server 2003/MS SQL 2000), but often the dates and times will be different. The dates and times of the events in SIM are below.

10/26/05 7:59 AM Rising Threshold Passed
10/26/05 5:09 AM Critical Rising Threshold Passed
10/25/05 4:20 AM Rising Threshold Passed
10/25/05 12:34 AM Critical Rising Threshold Passed
10/24/05 2:46 AM Rising Threshold Passed
10/24/05 2:46 AM Rising Threshold Passed

Comparing these dates and times:

--Server Time-- --SIM Time-- --Difference--
10/26/05 02:46 10/26/05 07:59 5 hr, 12 min
10/25/05 19:49 10/26/05 05:09 9 hr, 19 min
10/25/05 02:46 10/25/05 04:20 1 hr, 33 min
10/24/05 19:47 10/25/05 00:34 4 hr, 46 min
10/24/05 02:46 10/24/05 02:46 0 hr, 0 min
10/24/05 02:46 10/24/05 02:46 0 hr, 0 min

I checked the notices table in SQL, and the times (in UTC) correspond the the time reported by SIM, not the time the event occurred on the server.

I did some testing by sending test traps while doing a network trace. When the test trap was sent, I saw the SNMP packets receieved by the SIM server, but the event did not show up in SIM or the notices table until several hour later. Below is an alert email that shows this:

Event Name: Generic trap (11003)
Event originator: *****
Event Severity: Major
Event received: 26-Oct-2005, 02:46:30
Event description: Generic trap.
Text: Management Agents Test Trap sent - Tuesday, October 25, 2005 4:44:20 PM

The event was not received until over 10 hours after it was sent, but the SNMP packets were received at the SIM server immediately after it was sent.

Initially I though it might be a time zone issue, but the time differneces are not full full hours and are not consistent. Additionally, it does not happen all the time and on all systems. I can always reproduce the problem when sending a test trap.

I have check the system logs, SIM logs, task schedules, SQL logs, etc., but have not found anything that could have caused this.

Has anyone else seen this or have any ideas what may be causing it?

Thanks,
Pete

[Richard Litchfield]

Have you noticed any traps that never show up at all?

If you can restart the box, send some test traps as soon as SIM is available again and see if they are delayed or not. Then do the same test again after a few hours and see if the delay has come back.

Richard

[FS2006]

Hello Pete,

we happen to have the *exact same issue*. My gut feeling is that it could be some kind of resource leak but I have no real evidence to support this idea. For us, the only thing we can do to eliminate the delay is to reboot the OS.

By any chance, did you manage to solve this problem in the meantime?

Thanks

FS

[pedro-chicago]

When this did occur, the only solution was to restart/reboot. It hasn't happened in quite a while, but I don't know if that can be attributed to any upgrades (OS patches, Agent updates, or HP SIM updates).

Ptee

[FS2006]

Many thanks for your reply, I wasn't even sure you would see my post so that's cool.

Could you tell me how many systems are monitored on your HP sim (in my case, 380 systems) and if you use MSDE or MSSQL (i currently use MSDE, running on the same server as hp sim)?

And by any chance, did you defrag your DB, or re-build it from scratch at some point? Just to see if it's worth investigating that route before other things...

Thx.

[pedro-chicago]

We have almost 550 devices (480 servers, the rest are networking or other). We are using MS SQL 2000 locally on Windows Server 2003. We have not done anything with SQL (optimizations, integrity checks, etc.) since we had the problem that we were not already doing before the problem.

Pete

[FS2006]

I think I'm on to something... Pete,
would you have any IP phones or VC devices in your environment?

[pedro-chicago]

I believe we do, but they are not monitored via HP SIM.