After Upgrade to 5.3 Insight manager service needs restart daily

[Ragarc]

Any updates?

[Darrin Rawls]

No update; we have isolated a couple of items that are showing issues. Still triaging.

[JKytsi]

Do You need more SIM config captures ?

[Darrin Rawls]

No thank you, we have several now.

[JKytsi]

Do you have any with Oracle DB ?

[mark q]

I've got the same problem here - java heap.

I tried to upgrade to sim 5.3 this weekend and I'm going back to 5.1 sp1 (current version)

so far - java heap kills sim weekly but it's not as bad as daily

the jvmmaxheap=1500 doesn't work as hp suggests

I just gave a whole list of the problems out on the forums..

- no cluster identification etc and bunches of memory

I'm running this on a dl585 g2 (4 procs) and 16 gig of memory

- tried using network teaming but that freaked out snmp

- webes blew up after 24 hours of operation
- osem stopped working after 24 hours of operation.
- can't restore nodes with mxnode
- can't restore nodes with hosts file.

[Marc Carriere]

Darrin/Marc,

Uninstalling the DB & SIM will not help.

1) if PMP is installed, contact support to be beta PMP 5.1 RC3

2) if RSP then the issue appears to be triggered by hw polling tasks. To get around this:
a. temporary disable RSP
b. or slow down the polling rate of hw polling task; the default is 10min (or is it 15 ...?) set it 1/2 hr or more c. or disable the default hw polling tasks and create several hw polling task against a small of group of devices and schedule them at different time slots.

So far customers are not willing go with step "c" since they have thousand of devices.

The fix made in 5.3.1 is to limit the number of SOAP requests from plug-ins.

I m instigating what triggers plug-in (RSP) to keep sending SOAP requests to CMS.

hiep





I'm on the phone with SIM Support and they are recommending that I uninstall the Database, SIM program only and then reinstall SIM with new database and reconfigure. Do you agree?





No solution yet. Do you also have customers in that state? Its looking like we can do a workaround in HP SIM that we will include in SIM 5.3 Update 1 that will be launching with the G6 platforms. Ultimately itâ s a plugin causing some haywire with SIM that we didn't see before launch.


Hi Darrin,

Found you on HP Forum for Problem: "After Upgrade to 5.3 Insight manager service needs restart daily".

Is there a Solution to this problem at this point?

Your response would be greatly appreciated.

Thanks

[Russ Todd]

After opening a case with HP, they have had me follow the followign prodedure.

1. Stop and disable the following services:

Note: All of these services may not be listed depending on the current version of RSP.

Desta_Service
HP Asset Remote Execution
HP OVCM MSI Redirector
HP OVCM Notify Daemon
HP OVCM Scheduler Daemon
HP Remote Support Client
HP RSSWM-SIM Context Service
HP Unified Collector Service
ISEE_CMS_SWMAGENT (A.5.10)
ISEE_SWM_COMMANDQUE (A.5.10)
Open Service Event Manager
SQL Server (WEBESDB)
WCC Proxy

2. Do a find for iseectl.exe and rename it. This is a spin off of the RSP service that is event driven.



Q. Can you give may idea of what functionality has been disabled? Will anything be noticeable in the CMS?


1.The functionality of automatically generating HP service requests when traps are received from a Proliant server (under service contract or warranty) indicating some type of hardware error.
2.The ability to monitor Proliant server contract / warranty status and alert you when the warranty on a server has expired.

You should not notice any changes in SIM except perhaps the CW (contract / warranty) column may not display status





So far, I have not had to reboot the system for the past 5 days. Woo Hoo!

[Steven Baillie]

I just stopped all the services mentioned above and renamed iseectl.exe. my server was running constantly at 50% and above. now running in single figures and even getting down to 0%

brilliant workaround.
sad that HP let this out like this.


steven

[trytosolve]

is this a x64 windows CMS system? how about the remote SQL is that to a x64 windows system

[Darrin Rawls]

doesn't matter x64 or not. Its looking like the problem is only with the Remote Services plugin.

[PabloC]

Hi Russ,

Though advised via the initial support calls, HP does not recommend stopping or removing the RSP functionality.
The issue being seen, has been isolated to the interaction between HP-SIM's "discovery" and RSP's "device enablement" processes.

If you rename the iseectl.exe you are pretty much killing the RSP and won't be able to monitor your Proliants and other devices/clients, which is probably why you installed the Remote Support Pack in the first place.

While the investigation continues into a solution, if the scenario is seen at your site, the "discovery" connection between HP SIM and RSP can be removed via the following steps:

i) Open Window Explorer

ii) Change directory to \jboss\server\hpsim\deploy (Path Example: "C:\Program Files\HP\Systems Insight Manager\jboss\server\hpsim\deploy")

iii) Remove the folder "isee-eventnotification.sar"

iv) Open a Command Window

v) Perform the following command: mxconfigrefresh

At this point, HP SIM with "Hardware Status Polling for Servers" and the RSP Client service can be enabled, however newly discovered endpoints within HP SIM will need to be manually enabled into RSP via the "Synchronize RSE" button under the RSP "Entitlement" tab.

Regards.

[Russ Todd]

Thank you pabloc but the instructions I relayed above were directly from the HP email that I received from:

Rubenstein, Joel P (GCC ISS Engineer)
Wednesday, March 11, 2009 6:24 AM.
Case # 3605150587


I suggest that you confir with him as he is troubleshooting the error as far as I know.

(seems like the left hand doesn't know what the right hand is doing)

[Darrin Rawls]

Pablo is correct; the team has been able to figure out how to still have the RSP product installed and running so you still get the warranty information and such, just without any of the problems.

I would suggest doing what Pablo states. We are working on an advisory that will say the same thing too.

[Russ Todd]

Ok, I have performed the steps as noted by Pablo. Although not listed, I presume inferred was the renaming of iseectl.exe back to the original name.

I have run the mxconfigrefresh.

Lat's see how long it stays up now.

[PabloC]

Hi Russ,

Like Darrin said, it's just a way to get the RSP to continue working.

I mean, the initial fix that was provided to you will work on getting rid of the symptom by killing the whole RSP functionality, this new fix just kills the little part that causes that symptom, eventually we should get a formal workaround that will actually fix the issue without having to loose anything, it's just a process.

Regards.

[Russ Todd]

Sounds good to me....I understand.

I'm looking forward to the full fix. As I'm sure others are as well.

[Anky]

What is RSP? and what exactly the issue with it?

I have SIM 5.3 (fresh install) and we got last error on 10th march for DB disconnection after that it stopped, i dont know how. We have also raised a case with HP and same person Joel is looking into the issues, he has kept this case under monitoring.

second issue is VCRM agent updates on CMS server as well as discovered machines. is there any problem with PSP upgrade on DL machines? Joel is also looking inot it and i have send captures/logs to him but seems like he has decided to send this case to third level.

If you can help me would be grate.

[JKytsi]

Great ...this actually seems to help.
mxdomainmgr using CPU now much less, still 1,5GB of memory but that doesn't worry me.

Waiting for official updates.

[WishfulThinking]

PabloC,

Can you elaborate on the file that's deleted?

Is this a system generated file?

If so is it easily recreated so that when an official fix/patch is relased the changes can be easily reverted?

Many thanks.

[Brian Hahne]

Does anyone have an update on a supported/issued fix for this coming out?
I'm getting REALLY tired of having to restart something multiple times a day, that we rely on for monitoring hardware failures in a 500 server intel environment.

I think some kind of technical advisory at least saying what the status is, is warranted at this time..

[ehuang]

My SIM5.3 is doing this to me too. HP, fix?

[Russ Todd]

If you subscribe to the alerts from hp you will see in the March 31st edition a link to the following:


Advisory: HP Systems Insight Manager (HP SIM) Version 5.3 Windows Software - HP SIM Configured With the Remote Support Pack (RSP) Plug-In May Suddenly Shutdown (c01710870)

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01710870&dimid=1018244228&dicid=alr_apr09&jumpid=em_alerts/us/apr09/all/xbu/emailsubid/mrm/mcc/loc/rbu_category/alerts

[Brian Hahne]

OK here's a dumb question. I subscribe to alerts, or so I thought.. but I didn't see or receive this. Where's the signup page again so I can check my settings of what alerts I'm getting?

[Brian Hahne]

This fix does seem to work. My system stayed up through the weekend, and was able to complete the VCA checks, and warranty checks on all systems... still running...

I'd still like to get the alerts though, I can't recall or find where or why I'm not getting advisories emailed.
anyone?