Very slow IMC

[racowi]

Hi guys,
Is there a way to increase performance and make the IMC faster?
I'm getting slow performance while using the web-gui and also sometimes I get an "Server Busy" error.

Most of the time it get so bad that the only workaround is to do a logout and re-login. After doing that, the page is a little bit faster and less buggy.

Any recommendation?

[jguse]

Hello,

Here are a few things you can check when IMC is not performing well:

* Clear your browser cache - especially important after upgrading IMC, but also helpful when you have opened the GUI for a long time in the browser

* Check on the IMC server using Deployment Monitoring Agent - is the CPU or Memory usage very high (90% or more)? Are there any particular processes using a lot of CPU?

* Make sure your IMC server hardware is sized accordingly for your device count and performance monitoring volume - see the following guide for scaling recommendations: https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00075913en_us

Note that if you use optional modules, their hardware requirements should be added to those of the base platform if they are running on the same system. Also note that vCPUs are generally less capable than physical CPUs and those recommendations are for physical CPU, so add more vCPUs if it's a VM.

* Ensure you are running the latest iMC version. Recently saw some issues with Server busy on 7.3 E0703 that were resolved by upgrading to E0705+.

* If you are on a version prior to 7.3 E0703 and can't upgrade yet, make sure your IMC server has enough Java Heap size allocated. You can see the heap size available on the Environment tab of DMA. This is how much memory the "jserver" process of IMC can use, and its memory usage should not be hitting the heap size limit, or such issues with performance can happen. See the scaling guide I linked above for instructions on how to change it. It's no longer relevant on the OpenJDK versions E0703+.

Hope that helps.

[racowi]

Thanks Justin for the reply.

I'm running version E0705P04 and the java heap size is 8Gb. We have around 2000 nodes count and we are using 6 vCPUs.

When I looked at the CPU or Memory RAM usage, both are running below 90%.
Most of the time the CPU are on 18% to 32%, just in sporadic moments it can hit around 70%, when I looked what process was on top of the list, here is what appears:  "imcnetresdm" (not sure what is that)

The Memory RAM is always around 72% and the apps usages winners are for SQL (10GB) and Java & javaw (~5GB).

[jguse]

Hello,

imcnetresdm is the 'network resource daemon' which is used by the Resource Management component, one of the core components of IMC. It is responsible for gathering device information via SNMP polling and uses the config_db database, which could be considered the main IMC database. You can check its log in iMC\server\conf\log\imcnetresdm<date> to see if there's any indication why its CPU usage is spiking. It could be due to too many/big Auto Discovery jobs running on a regular basis, or too much device polling, etc... 

Are you running MSSQL locally? What version and edition is it? Keep in mind that for 2k nodes it is normally recommended to offload the database to a remote DB server rather than a local one. Consider the following recommendations for 2k nodes from the guide as well - they are referring to physical CPUs here:

Check the physical host where the VM is running too - is it burdened by a lot of other VMs? I recall having slow performing IMC systems in my lab when the hosts had a NFS datastore, with a lot of other VMs running on the host... even though its host CPU was not above 50%, and the VM's vCPUs were not indicating any issue, it took several seconds for any click to respond. A vCPU may not show any issues even though the physical CPU core it runs on is experiencing significant load.

It may also be worth upgrading to 7.3 E0705P06, or waiting a few more weeks for the P07 release, though I am not aware of any related fixes. There was an issue with 100% CPU on imcnetresdm fixed already in E0705P02 - 8. IMC's CPU has reached 100% on imcnetresdm randomly - so you should not be running into that.

Just a few thoughts on what you can check. If you are still stuck and can't figure out what's causing it, I'd suggest opening a support case so we can have a proper look at it.

[racowi]

Yes my issue looks like that, any click took several seconds to respond when trying to edit a device or mark it for batch processes.
The MSSQL is running locally on the same server and I think the edition is SQL 2012 Standard.

Based on what you are saying... sounds like I can build the IMC in an physical server and probably leave the SQL database running on the current VM.

What do you think about that? Can it works better?

I just don't want to touch too much the database or have database headeache. But let me know if I can still use this VM for something (database or IMC stuffs), so I can plan for 1 VM and 1 physical. What services to put where? Database on physical and IMC on VM or viceversa? 

Thanks for the help. 

[jguse]

Hello,

I don't think you necessarily need to use physical hardware - I was just trying to point out that vCPUs do not perform quite as well as physical ones. There is a 'decay factor' to keep in mind, described in the Hardware Deployment Schemes doc I linked previously, and if you want to use IMC in a virtual environment, I always recommend going above the requirements listed in the doc for physical CPUs.

I've done IMC installations for 5k-15k devices on virtual machines and they work just fine. It's a matter of scaling the virtual hardware adequately and not just going for the bare minimum requirements - and also using a remote DB in such cases.

As for your migration, I would not recommend using the same system you run currently, as any issues would cause downtime for IMC. My suggestion would be to set up two new VMs, one for IMC and one for the remote DB. Though you would not have issues with one physical and one virtual machine either.

Make sure they are sized appropriately, install DB and iMC (exact same version), and then perform a migration from your old system to the new one. That way your network monitoring system won't be down if you encounter any 'database headaches' or other potential issues during setup.

There should not be any significant issues setting up the DB or migrating it, as there are detailed guides for every edition of MSSQL in relation to installation for iMC. If you run into trouble you can always ask here or open a support case too.

Centralized Deployment Guide with Remote DB: https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00094606en_us

MSSQL 2012 Installation Guide: https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=c05367564

Migration Guide (after the installation is up and running): https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00075552en_us

Keep in mind the requirements listed there, to ensure you migrate successfully:

Hope that helps!