topic Re: New SIM Memory Leak in Server Management - Systems Insight Manager

New SIM Memory Leak

Todd Lewis_1 — Tue, 06 Jan 2004 08:45:35 GMT

We are having problems with the system eating up memory and crashing the web service. Even with discovery turned off this is still happening. Anybody else experiencing the same issues?

Re: New SIM Memory Leak

Tim McGue — Tue, 13 Jan 2004 12:40:45 GMT

It looks like I am seeing the same issue to. I just started monitoring the memory today after reading your post. I see the memory usage in mxdomainmgr increasing steadily throughout the day regardless of what the system is actually doing. Are you seeing the leak in the mxdomainmgr process too or something else?

Attention HP:
Can someone take a look into this?

Re: New SIM Memory Leak

Tim McGue — Tue, 13 Jan 2004 14:57:39 GMT

More information: I have watched the mxdomainmgr.exe over the past three hours. The memory has gone from 448,000K usage to 1,000,000K usage. It continues to steadily climb even though I am physically not doing anything to the SIM server. In fact I can't do anything to the server because when I attempt to logon I receive the error:
java.lang.OutOfMemoryError

Re: New SIM Memory Leak

Jadrice Toussaint — Tue, 13 Jan 2004 15:01:51 GMT

Tim -

You may be right on this one. I'm currently running perfmon on this as well as strings to find out if this is leaking.

Re: New SIM Memory Leak

Tim McGue — Tue, 13 Jan 2004 15:30:08 GMT

Thanks for looking at this too Jadrice. I've opened up a call with HP through our support agreement. I'll post here when/if I hear anything.

Re: New SIM Memory Leak

David Claypool — Tue, 13 Jan 2004 16:03:57 GMT

If the growing memory is from Java (as opposed to one of the MXxxxxx processes), the following post from the IM7 forum may be useful (I post the answer itself because it was a long thread and the answer was at the end):

Compaq engineers and I are 99.9% sure that we have found the root cause of the java memory leak. We believe that it is caused from servers that are/aren't in the database having incorrect snmp settings assigned in the trap and security tabs. With the incorrect settings and without 127.0.0.1 (local loopback adapter) set in the snmp properties, the server will send authentication failure traps to the console; which would flood IM7 and make java.exe rise with memory consumption. After running a debug tool such as network monitor to monitor the snmp traps coming in; we realized which servers were sending all the authentication failures as i described and fixed the snmp settings to the correct values.

All in all make sure that (ALL) servers in your environment have the correct SNMP settings. This would be the correct name/ip address of the IM7 server and the (read-only) community string for the trap tab. Also you will need the name/ip address and the local loopback adapter address (127.0.0.1) along w/ the read-only and read-write community strings on the security tab set correctly. Until these items were updated on each server in our environment, the memory leak still existed.

Re: New SIM Memory Leak

Tim McGue — Tue, 13 Jan 2004 16:29:28 GMT

Thank you David for digging in; however, the leak I am seeing is in the mxdomainmgr.exe.

Re: New SIM Memory Leak

Rob Buxton — Tue, 13 Jan 2004 17:09:07 GMT

Out of interest, how many devices are you monitoring?

I've looked at our mxdomainmgr and it's cruising along at about 193,500 K.

We're monitoring just 124 devices, the majority are Proliant Servers with a smattering of Sun, some VMS and a few Switches.

Re: New SIM Memory Leak

Tim McGue — Tue, 13 Jan 2004 17:26:51 GMT

We don't have much more. Last time I was able to logon it was around 211. Most of them Windows with a few others such as yourself.

Re: New SIM Memory Leak

Tim McGue — Tue, 13 Jan 2004 17:37:01 GMT

A question the HP tech asked me was about the database back end. I'm using SQL Server 2000 SP3. Anyone using something different that is/is not seeing the problem?

Re: New SIM Memory Leak

Jadrice Toussaint — Tue, 13 Jan 2004 17:41:06 GMT

Tim -

The backend database should not be an issue especialy if it is run on a seperate machine.

Re: New SIM Memory Leak

Ray Tuholski — Tue, 20 Jan 2004 12:49:52 GMT

I had the same problem after upgrading the Insight agents on the SIM server from 6.3 to 7.0 . I reinstalled the JRE and Win2k SP4 and the problem was resolved after a reboot.

Re: New SIM Memory Leak

Tim McGue — Tue, 20 Jan 2004 12:52:44 GMT

HP has been working this case, but no solution as of yet.

Can someone test something for me? I am performing a SQL profiler against the database. One commands appears to run about 120 times per second. Can someone else run profiler and see if this is also the case for a non-leaking machine?

Here is the command I am seeing:
SELECT trapItems.trap_id, trap_name, trap_enterprise, trap_generic_id, trap_specific_id, trap_severity_id, enabled_flag FROM trapItems,trapItemsEx WHERE trapItems.trap_id = trapItemsEx.trap_id AND trap_generic_id = 4 AND trap_specific_id = 0

Re: New SIM Memory Leak

Tim McGue — Tue, 20 Jan 2004 14:03:50 GMT

Ray,

Thanks for the response. This server is still on 6.4 of the agents; however, it is still on SP3 of Windows 2000. I'm investigating now getting it up to SP3 and reinstalling JRE.

Thanks,
Tim

Re: New SIM Memory Leak

Ray Tuholski — Tue, 20 Jan 2004 14:31:33 GMT

Even more distressing than the memory and cpu usage was that mxdomainmgr.exe had 1.7 million handles and 500 threads going when it's usage had peaked in task manager. after the sp4 reinstall it settled back down to 2300 handles and 180 threads, the memory usage is about the same but the cpu hasn't gone above 30% total usage since.

Re: New SIM Memory Leak

Rob Buxton — Tue, 20 Jan 2004 14:58:06 GMT

In answer to an earlier question, our SQL Backend is on W2003, running SQL Server 2000 SP3.

Re: New SIM Memory Leak

Mike Ludwig — Fri, 23 Jan 2004 12:01:14 GMT

My memory isn't a problem but my CPU has been pegged for about a week. The system is still operating fine and response time dosn't even seem to be slow.

Re: New SIM Memory Leak

Tim McGue — Mon, 02 Feb 2004 13:35:00 GMT

OK, this is where we are at so far with the case from HP. The problem appears to be related to the HPSIM handling authentication traps from discovered servers. Authentication traps are sent when:

1) On a Windows box you have the "Send Authentication Trap" box checked on the Security tab of the SNMP service.
2) That box receives SNMP communication from a host that is not on its acceptable packets host.

In my opinion, this is a good thing to have checked so you know if you are getting unauthorized SNMP attempts against your systems.

There are some situations in which authentication traps can occur because of the HPSIM server. Say you have the wrong community defined on HPSIM or on your server, that would throw an alert. Say you have HPSIM server as a destination but not as a trusted source, that would throw an alert. In those cases it gets into a cyclic affect of the HPSIM server receiving failure traps, sending receipt notices, and getting another failure. You can easily view this by running a network monitoring tool on either system. You can also see a load of entries in the HPSIM audit log as well.

So, to help reduce authentication traps, ensure your SNMP is setup correctly on all your managed systems.

Now, with all that said, HP definitely has something to fix here. They should not be leaking memory when handling authentication traps. It is real easy to recreate the memory leak. This is what I am pushing HP for with the case I have open.

Tim

Re: New SIM Memory Leak

Moe_5 — Thu, 05 Feb 2004 10:40:24 GMT

Hey Tim,
all makes sense but im seeing the same problem with the memory leak and I havent even added any servers to the SIM database! The server is running on its own with nothing to monitor but itself and im seeing major memory leak on the mxdxxxxxx process... I guess that pushes back my upgrade to the new cim as i figure my live environment monitors about 2500 servers and if im seeing memory leak when the server isnt even monitoring anything i cant even imagine what would happen if i was monitoring 2500+ servers!!
Man you would think compaq/hp would do a little more testing...

Re: New SIM Memory Leak

Tim McGue — Thu, 05 Feb 2004 11:47:50 GMT

If I remeber correctly, SIM tries to perform discovery out of the box on its own subnet. You could run a netmon and see a bunch of SNMP chatter or run a SQL trace to see the same command ran multiple times. You could be in a memory grabbing loop with just one system (i.e. SIM talks to the member, the member doesn't like SIM, it reports to SIM the failure, and so on). It would be worth a look before bailing on it 100%. I do agree that HP needed to do more homework here definitely.

Tim