Memory issues stopping HP SIM functioning within 10 minutes of the service starting

[Sean Wheeler]

I installed HP SIM 5 SP4 a week ago to monitor around 150 servers. The specs of the CIM server I am using are:

OS Name Microsoft Windows 2000 Server
Version 5.0.2195 Service Pack 4 Build 2195
System Model ProLiant DL380 G3
Processor x86 Family 15 Model 2 Stepping 7 GenuineIntel ~2785 Mhz
Processor x86 Family 15 Model 2 Stepping 7 GenuineIntel ~2785 Mhz
BIOS Version 01/31/03
Total Physical Memory 2,620,948 KB

I have set up an automatic event handler which runs a script every time an event is logged. However within about 10 minutes of the HP SIM service running the:
mxdomainmgr is using 500 Meg of RAM
sqlservr is using 100 Meg of RAM
hpvmmsvcj, mxinventory, mxdtf, and hpsmhd are using about 200 Meg between them. Consequently the automatic event handler task which runs the script hangs on 0%. These processes are also hammering the CPUs.

The only scheduled tasks are the default ones. Is this normal behaviour for the CIM? Has anyone found a way to reduce the amount of memory and CPU time these processes use?

Any help would be appreciated!

[Matthew J Warrick]

"I have set up an automatic event handler which runs a script every time an event is logged."

I think that's your problem right there... running a script for _every_ event seems a little excessive in and of itself.

[Numy]

Hi Sean, strange question but do you currently have any VM Hosts in your SIM environment?

Long story but here's the short of it:

We have a case opened currently with HP on problems where it appears that the VMM Service is causing SIM to reach it's max memory usage quite quickly where it eventualy dies. Almost behaves like it is being kept from doing garbage collection or some type of leak assoicated with the VMM Service when talking to SIM. We have since last Thursday stopped the VMM Service and SIM is holding steady between 425 - 450MB of memory for the mxdomainmanger.exe process.

I would try stopping the VMM Service on your instance and then restart SIM and see if it makes any difference for you just so you can rule it out. If yours is the same I will try to post what I find out to this thread. Thanks.

[Numy]

Matthew has a great point, I totally skimmed right over that crucial data. I would definitly modify that behaviour also. (Or at the very least remove it if you can and see how things behave for a period of time.)Thanks!

[Sean Wheeler]

sorry my script doesnt run for _every_ event, it filters out ones I don't want to know about but it still runs for most of them.

The reason for this is the script writes out the events into a text file that can be picked up by BMC Patrol (another monitoring agent). I realise there is a plug in which will do this but my company would not pay for it.

Thanks for the tip about VMs, I don't have any in the environment at the moment but I'll watch out for them.

[Numy]

Thanks Sean, I would still shut off the VMM Service just in case since I see you have it running(hpvmmsvcj process is running in your list.) Will continue to think through other items as we have a similiar setup with around 2000 nodes including iLOs that is running pretty well(with the exlcusion of when VMM is running).

[Sean Wheeler]

Thanks Numy, I have shut off the VMM Service and disabled the task that runs whenever an event comes in.

However the program is still using a lot of memory. Is there any way I can find out what HP SIM is doing to use so much memory. I have attached the audit log. I can't find a way of working out which tasks are using all the memory.

[Numy]

Sean, are you seeing any errors like this in your mxdomainmgr.(PID).out log files?

java.lang.OutOfMemoryError: Java heap space

Since it is a Java app after all you will see memory get used to a certain point. I do think 500 seems a little high out of the gate but usually my server does jump to around 425MB of usage right away which is pretty normal.

I'm a little concerned about the CPU getting hit pretty hard as I am not seeing that on my system until I start running out of memory. (Usually it is when we get up to the 850 - 890MB range and committed bytes in use is nearing the 2GB range for the user memory space.) Can you give a little more detail on which processes are chewing the memory up?

I would also considering changing some of the polling intervals or stopping them for a while and see if it makes a difference.

Lastly, how many events do you currently have out there? How long are you retaining them for? Did you ever turn on Authentication Traps? (Default is off but thought I would ask since it takes some overhead to keep running with a lot of nodes from what I have heard, so if you don't recall doing that it is probably off still.)

[Numy]

One change for above, I meant to ask for what processes are chewing up CPU not memory. Sorry about the lack of proofreading there.

[Sean Wheeler]

Thanks for your help Numy, I seem to have got the CPU and RAM usage under control now. I have changed the hardware polling from 5 minutes to 15 minutes. I have disabled my e-mail event task. Also I was getting a lot of back up exec events which I have switched off individually on the servers. I had look at the mxdomainmgr.(PID).out log files but couldn't find any OutOfMemoryError errors.
mxdomainmgr is now using about 250 Meg of RAM and the CPU is running at around 10% but with spikes when events come in. Which from what your saying is normal for the system. Most importantly my task to log events to text files is now working :)

[Numy]

Sean, that's great to hear.

I have also had to modify my polling in order to keep things in control. One word of caution and you may already know this but if you change hardware polling and a server goes into a critical state because the server is completely down, I believe it will not notify you until that polling job runs. So in other words if yours is set to 15 minutes it could take that long to find out about a server being down.(Not that 15 minutes is an extreme time but just in case you decided to space it out any farther.)

My location uses OpenView for that part so we weren't really concerned with the polling frequency.

[NEETI TOLIA]

Best way to handle such situations is get a call logged with HP support.

Incase anyone is using VMM for managment of their virtual machine, there is one more workaround that can be tried out for mxdomainmgr hogging issue


1. Stop the HP VMM service , â HP Virtual Machine Management Service â from the Service Console (Programs->Administrative tools ->Services)

2. Open the \HP Virtualization Management Software\bin \hpvmm.conf file and add this property definition:
SimQueryDelaySec = 300

3. Start the VMM service service , â HP Virtual Machine Management Service â from the Service Console (Programs->Administrative tools ->Services)


Neeti