New SIM Memory Leak

[Todd Lewis_1]

We are having problems with the system eating up memory and crashing the web service. Even with discovery turned off this is still happening. Anybody else experiencing the same issues?

[Tim McGue]

It looks like I am seeing the same issue to. I just started monitoring the memory today after reading your post. I see the memory usage in mxdomainmgr increasing steadily throughout the day regardless of what the system is actually doing. Are you seeing the leak in the mxdomainmgr process too or something else?

Attention HP:
Can someone take a look into this?

[Tim McGue]

More information: I have watched the mxdomainmgr.exe over the past three hours. The memory has gone from 448,000K usage to 1,000,000K usage. It continues to steadily climb even though I am physically not doing anything to the SIM server. In fact I can't do anything to the server because when I attempt to logon I receive the error:
java.lang.OutOfMemoryError

[Jadrice Toussaint]

Tim -


You may be right on this one. I'm currently running perfmon on this as well as strings to find out if this is leaking.

[Tim McGue]

Thanks for looking at this too Jadrice. I've opened up a call with HP through our support agreement. I'll post here when/if I hear anything.

[David Claypool]

If the growing memory is from Java (as opposed to one of the MXxxxxx processes), the following post from the IM7 forum may be useful (I post the answer itself because it was a long thread and the answer was at the end):

Compaq engineers and I are 99.9% sure that we have found the root cause of the java memory leak. We believe that it is caused from servers that are/aren't in the database having incorrect snmp settings assigned in the trap and security tabs. With the incorrect settings and without 127.0.0.1 (local loopback adapter) set in the snmp properties, the server will send authentication failure traps to the console; which would flood IM7 and make java.exe rise with memory consumption. After running a debug tool such as network monitor to monitor the snmp traps coming in; we realized which servers were sending all the authentication failures as i described and fixed the snmp settings to the correct values.

All in all make sure that (ALL) servers in your environment have the correct SNMP settings. This would be the correct name/ip address of the IM7 server and the (read-only) community string for the trap tab. Also you will need the name/ip address and the local loopback adapter address (127.0.0.1) along w/ the read-only and read-write community strings on the security tab set correctly. Until these items were updated on each server in our environment, the memory leak still existed.

[Tim McGue]

Thank you David for digging in; however, the leak I am seeing is in the mxdomainmgr.exe.

[Rob Buxton]

Out of interest, how many devices are you monitoring?

I've looked at our mxdomainmgr and it's cruising along at about 193,500 K.

We're monitoring just 124 devices, the majority are Proliant Servers with a smattering of Sun, some VMS and a few Switches.

[Tim McGue]

We don't have much more. Last time I was able to logon it was around 211. Most of them Windows with a few others such as yourself.

[Tim McGue]

A question the HP tech asked me was about the database back end. I'm using SQL Server 2000 SP3. Anyone using something different that is/is not seeing the problem?

[Jadrice Toussaint]

Tim -

The backend database should not be an issue especialy if it is run on a seperate machine.

[Ray Tuholski]

I had the same problem after upgrading the Insight agents on the SIM server from 6.3 to 7.0 . I reinstalled the JRE and Win2k SP4 and the problem was resolved after a reboot.

[Tim McGue]

HP has been working this case, but no solution as of yet.

Can someone test something for me? I am performing a SQL profiler against the database. One commands appears to run about 120 times per second. Can someone else run profiler and see if this is also the case for a non-leaking machine?

Here is the command I am seeing:
SELECT trapItems.trap_id, trap_name, trap_enterprise, trap_generic_id, trap_specific_id, trap_severity_id, enabled_flag FROM trapItems,trapItemsEx WHERE trapItems.trap_id = trapItemsEx.trap_id AND trap_generic_id = 4 AND trap_specific_id = 0

[Tim McGue]

Ray,

Thanks for the response. This server is still on 6.4 of the agents; however, it is still on SP3 of Windows 2000. I'm investigating now getting it up to SP3 and reinstalling JRE.

Thanks,
Tim

[Ray Tuholski]

Even more distressing than the memory and cpu usage was that mxdomainmgr.exe had 1.7 million handles and 500 threads going when it's usage had peaked in task manager. after the sp4 reinstall it settled back down to 2300 handles and 180 threads, the memory usage is about the same but the cpu hasn't gone above 30% total usage since.

[Rob Buxton]

In answer to an earlier question, our SQL Backend is on W2003, running SQL Server 2000 SP3.

[Mike Ludwig]

My memory isn't a problem but my CPU has been pegged for about a week. The system is still operating fine and response time dosn't even seem to be slow.

[Tim McGue]

OK, this is where we are at so far with the case from HP. The problem appears to be related to the HPSIM handling authentication traps from discovered servers. Authentication traps are sent when:

1) On a Windows box you have the "Send Authentication Trap" box checked on the Security tab of the SNMP service.
2) That box receives SNMP communication from a host that is not on its acceptable packets host.

In my opinion, this is a good thing to have checked so you know if you are getting unauthorized SNMP attempts against your systems.

There are some situations in which authentication traps can occur because of the HPSIM server. Say you have the wrong community defined on HPSIM or on your server, that would throw an alert. Say you have HPSIM server as a destination but not as a trusted source, that would throw an alert. In those cases it gets into a cyclic affect of the HPSIM server receiving failure traps, sending receipt notices, and getting another failure. You can easily view this by running a network monitoring tool on either system. You can also see a load of entries in the HPSIM audit log as well.

So, to help reduce authentication traps, ensure your SNMP is setup correctly on all your managed systems.

Now, with all that said, HP definitely has something to fix here. They should not be leaking memory when handling authentication traps. It is real easy to recreate the memory leak. This is what I am pushing HP for with the case I have open.

Tim

[Moe_5]

Hey Tim,
all makes sense but im seeing the same problem with the memory leak and I havent even added any servers to the SIM database! The server is running on its own with nothing to monitor but itself and im seeing major memory leak on the mxdxxxxxx process... I guess that pushes back my upgrade to the new cim as i figure my live environment monitors about 2500 servers and if im seeing memory leak when the server isnt even monitoring anything i cant even imagine what would happen if i was monitoring 2500+ servers!!
Man you would think compaq/hp would do a little more testing...

[Tim McGue]

If I remeber correctly, SIM tries to perform discovery out of the box on its own subnet. You could run a netmon and see a bunch of SNMP chatter or run a SQL trace to see the same command ran multiple times. You could be in a memory grabbing loop with just one system (i.e. SIM talks to the member, the member doesn't like SIM, it reports to SIM the failure, and so on). It would be worth a look before bailing on it 100%. I do agree that HP needed to do more homework here definitely.

Tim

[Steve_500]

did this ever get resolved?

[Darrin Rawls]

You'd be surprised at how much testing we do do with HPSIM. One thing that we have absolutely learned though is that 'no 2 networks are the same.' We try our best to do or simulate all the 'tweaks and tunes', but lets face it; there are millions of permeatations out there. :)

Keep in mind, by default, the IP range of the SIM server (1-254) are prefilled and Discovery is set to kick off once a day (if you haven't turned it off), so it should discover that one subnet. By default, we just put in 'public' for the comm string; so if that's wrong (which I hope it is!), you'll get these "authentication" traps sent. Also, by default, the Discovery Filters are enabled to not discover devices that are not properly id'd as servers and such.

So....if you have the wrong comm string set in SIM, have Discover Filters enabled on the Automatic Discovery page, and the discovery runs every day; you are going to get memory creep. SIM keeps traps in memory for doing notifications and such. We don't want to do anything special to this particular trap b/c this is like the only other 'security item' with SNMP (besides the comm string of course).

Put a real comm string in there and then see if it climbs (you can recycle the SIM service if you want to start 'clean' from a memory POV).

Hope this helps.

[JP_MandG_Please_Remove_]

Hi There,

I have also had the issue of a memory leak on my installation of SIM.

I have found that the numbers in the memory column of all my processes do not add up to the total amount of memory that is being used. In this case the amount being used by is close on 950MB while in the memory column it adds up to about 300-350MB. This is one HUGE memory leak of 600MB or more.

I have had a look through my systems and checked the SNMP trap settings and the string settings and all of them are set correctly.

Even though my SIM server was set up as a test and I had no traps going to that server, the memory usage was getting larger and larger as time goes by without it doing anything, and the only noticible process was mxdomainmgr.exe. If you watch it closely it gets higher and higher, but when when it drops (as if releasing the memory it is using) the total memory used on the server barely drops and only gets bigger every time mxdomainmgr starts processing again.

As a result of all of these issues that SIM 4.2 has I have reverted back to good old CIM 7 for usability, performance and reliability, and informed my manager not to move over to SIM anytime too soon.

Why can't HP admit to the fact that there is an issue with the product (seeing so many people have problems with it) and fix it.

I think there are enough real world examples here of people's postings for HP not to give excuses, that no 2 networks are the same, and get a bigger server spec, and change your log sizes, and SNMP strings, and all of these that are coming out. People are seeing issues with a standalone servers and nothing on them, getting maxed out and leaking memory like the Titanic.

Please HP, please sort it out. The package has great potential....

[David Claypool]

Jason's note brings up an interesting subject that requires a reminder: this forum is for the exchange of ideas and is not an official method of reporting problems to HP. If you look back at the anecdotes on this thread there is nothing that is actionable by a dev team.

Please, please, call your problem in to customer services so that it can be duplicated and escalated.

[Darrin Rawls]

By the way, we did see a little memory creep in the 4.1 product that was addressed in the 4.2 version (that shipped in December). Most of these posts where early in 2004 which was mostly against the 4.0 product (there were a ton of fixes in the 4.1 version around memory handling).

Hope this helps.