Version Control Agent hangs

[n.price]

have HP taken the QA manager for these software updates and fired them into the sun?

the last few rounds of PSP agents & the VCRM have had terrible problems!

Are HP even testing their software on their own systems?

[Mahesh Shah_3]

I had many problems with PSP v8.30 (VCA. SHM) and HP SIM 5.3 sp1 with hot fix for a fresh install on ProLiant DL385 G2 server running Win2k3 Enterprise SP2 with SQL 2005 Enterprise.

HP SIM console is a prime time tools for Hardware monitoring, since we have over 3300 ProLiant servers configured for Hardware monitoring.

Finally I found the workaround solution as follows:

1.Keep the HP SIM 5.3 on the server.

2.Do not install PSP v8.30, keep PSP v.8.15B

3.Uninstalled VCA, VCRM and SMH, reboot.

4.Install SMH (2.1.15.210 â cp009426.exe), VCA (2.1.10.801 â cp008925.exe) and VCRM (2.2.1.830 - cp010140.exe).

Since then HP SIM very stable

[Stan Oleszkowicz]

Version 2.2.2.235 of VCA has been released and resolves this issue.

[Dirk Frey]

I have updated VCRM and VCs to version 2.2.2.835 - and now it works fine on all of our servers.

Thanks to HP - I have wasted a lot of time for this case.

[JasonWoerner]

I'm running this version on all ProLiants. Temporarily solves issue. But after data gathering every two weeks all of the blue question marks return. Seems like querying multiple ProLiants at the same time makes them all time out.

[Dirk Frey]

@Jason
Did you update the Version Control Repository Manager also?

[JasonWoerner]

I'm running Repository Manager 2.2.0, which comes with the latest release of HP SIM.

[Dirk Frey]

You can download newer versions of VCRM using the ProLiant web sites for drivers and software.

[jbonnema]

I have the same Problem, the VCA is real slow or times out all together.
It seems to have happened after the HPSIM 5.3 SP1 and "MAY" hotfixes were completed, just updated to the New VCRM 6.0.0.840 and no luck. VCA agents are at 2.2.2.835. Some of them do respond back with the right information, but only after a 2-5 min wait. Something is very wrong and I hope HP figures this out soon.

One other note: We have a GPO policy for our SNMP settings. This issue was supposedly fixed in the VCA v2.2.2.830 but I'm not so sure.

Maybe the New SIM 6 will do away with allot of these issues.

[Ralph Frampton]

Grr... stupid HP web page... the Preview/Edit doesn't appear to work properly meaning I've got to re-type everything again!

I have made some observations regarding VCA/VCRM activity.

All versions of VCA will cause the SMH time-out issue if the target VCRM it is set to enters a higher than normal CPU state. The VCA is typically unusable for any purpose during this period of VCRM high CPU activity.

The following items have been observed to cause high VCRM CPU:
- starting VCRM service, catalog verification
- HPSIM Data Collection Task for many servers
- HPSIM Software Status Polling Task for many servers
- auto-update from HP site, catalog verification
- some unknown HPSIM tasks, these do not show up in the Show all Scheduled Tasks but are apparently initiated by mxadmin accounts (we have two HPSIM systems) on a 4 hour cycle - ie our VCRM goes nuts for a good 30 minutes or so every four hours.

The two HPSIM servers are doing tasks similar to:
- - [13/Jan/2010:18:21:42 -0500] "GET /Proxy/STE?TKN=4c8312...&URL=https://:2381/vcrepository/catalog?format=xmltree&version=C.05.03.01.00 HTTP/1.1" 200 8103179

- - [13/Jan/2010:18:24:03 -0500] "POST /Proxy/STE?TKN=0a97111...&URL=https://:2381/vcrepository HTTP/1.1" 500 538

If anyone knows what these HPSIM processes are then we may be one step closer to figuring out how to get the VCRM under control. I don't think it is the normal HPSIM polling tasks as I set our HPSIM server & it's ILO in the Montreal HPSIM to a Suspend Monitoring indefinitely state, however after doing this the VCRM still exhibited high CPU during the next 4 hour cycle of HPSIM requests.

I can adapt the other processes to running in periods where we aren't trying to use the VCA's but at this time I'm not certain what needs to be adjusted to account for the unexpected HPSIM activity.

[Ralph Frampton]

FYI, we are presently using different versions of HPSIM (5.3 SP1 in Montreal, 5.1 SP1 in Toronto - updating the latter is being considered but is on hold pending a decision whether to implement 5.3 sp1 or the soon to arrive 6.0).

The VCA is the latest version on both HPSIM servers & the VCRM on the Toronto HPSIM is also the latest 6.x revision.

[Ralph Frampton]

Before I go, CPU utilization is not normally an issue on our Toronto HPSIM server & during idle periods it's typically running 3% or less.

A normal VCA interaction to our VCRM will increase CPU briefly when SMH started, a bit longer when VCA opened & again a bit longer when deploying a selection of software (ca. 1 - 2 minutes depending on number & size of each).

In the normal state, the VCRM appears to be capable of handling at least 5 or 6 VCA software updates to different servers at the same time from the present host server (dual 2.8Ghz Xeon with 2 GB RAM & SQL2005 Standard Ed.).

[ajc79]

This seems to fit for me too. I can manually run the software polling task against four or five servers and it works fine. If I leave the automated task to run against every server, it goes all sideways and i'm left with a screen full of question marks underneath the software status column in SIM.

[Ralph Frampton]

Further to my last set of posts... we have confirmed that the Montreal HPSIM server was responsible for several long term hits on our Toronto VCRM.

To determine how much impact our local HPSIM was causing compared to the Montreal one we removed the Toronto HPSIM/VCRM host from the Montreal HPSIM server and the long term hits associated with it stopped happening.

This now allowed us to see that our Toronto HPSIM server impact to the VCRM on same server is on the order of 10 - 15 minutes every fours. It's better than before with two HPSIM's pounding on the VCRM but the VCA's still can't do anything during these windows of high CPU usage by cpqsrhmo.exe.

There appears to be something different between HPSIM 5.1 SP1 (our Toronto one) & HPSIM 5.3 SP1 (our Montreal one) in that HPSIM 5.3 SP1 appears to be accessing the VCRM four times per four hour cycle as opposed to three per four hour cycle for HPSIM 5.1 SP1.

Here's an excerpt from our Toronto HPSIM SMH log (as seen from within SMH as opposed to access_log snippets I'd previously mentioned):
Logs » SMH » System Management Homepage Log

INFORMATIONAL 1/20/2010 12:25:59 PM
Secure Task Execution User:TORHPSIM\mxadmin was authorized with 'Administrator' rights to invoke target URL=https://TORHPSIM.OurDomainName:2381/vcrepository

INFORMATIONAL 1/20/2010 12:24:15 PM
Secure Task Execution User:TORHPSIM\mxadmin was authorized with 'Administrator' rights to invoke target URL=https://TORHPSIM.OurDomainName:2381/vcrepository

INFORMATIONAL 1/20/2010 12:24:01 PM
Secure Task Execution User:TORHPSIM\mxadmin was authorized with 'Administrator' rights to invoke target URL=https://TORHPSIM.OurDomainName:2381/vcrepository/catalog?format=xmltree

...

INFORMATIONAL 1/19/2010 11:48:13 AM
Secure Task Execution User:MTLHPSIM\mxadmin was authorized with 'Administrator' rights to invoke target URL=https://TORHPSIM:2381/vcrepository

INFORMATIONAL 1/19/2010 11:45:53 AM
Secure Task Execution User:MTLHPSIM\mxadmin was authorized with 'Administrator' rights to invoke target URL=https://TORHPSIM:2381/vcrepository/catalog?format=xmltree&version=C.05.03.01.00

INFORMATIONAL 1/19/2010 11:28:43 AM
Secure Task Execution User:MTLHPSIM\mxadmin was authorized with 'Administrator' rights to invoke target URL=https://TORHPSIM:2381/vcrepository

INFORMATIONAL 1/19/2010 11:26:43 AM
Secure Task Execution User:MTLHPSIM\mxadmin was authorized with 'Administrator' rights to invoke target URL=https://TORHPSIM:2381/vcrepository/catalog?format=xmltree&version=C.05.03.01.00

The error_log has what appear to be entries that match the end of the high CPU activity by the above:
[Wed Jan 20 12:31:00 2010] [error] [client toripaddress] Premature end of script headers: cgi

[Tue Jan 19 11:53:14 2010] [error] [client mtlipaddress] Premature end of script headers: cgi

So... the question remains... what HPSIM process is generating these hits on the VCRM as they do not appear to be any Scheduled Tasks as viewed from within HPSIM? It would be nice to hear something from HP as I can't find anything documented in their Help and/or web site.

[Ralph Frampton]

Today's observation re Blue Question marks in HPSIM SW column (Software Version Status: Unknown - VC Agent not configured)
- this should be obvious & in hindsight it is, this status is indicated if you delete a PSP in your VCRM while a VCA is set to use it as a Reference Support Pack.

Solution: To avoid the above scenario causing blue question marks, use Replicate Agent settings to distribute a new set of VCA settings that include the new PSP or custom baseline that you wish to set as your Reference Support Pack & this should preferably be done before deleting items in VCRM.

At least I've learned that Replicate VCA agent settings works as long as:
- a trust exists between your HPSIM and target SMH
- the VCA isn't in a non-responsive state (if so & you want to reset remotely without logging on via RDP then I recommend taskkill & psservice tools)

[Jason Lifsey]

This was resolved for me in version 2.2.2.835 however now in version I am finding the VCA agent 6.1.0.841 hung again on several of our servers. At least it restarts cleanly now, instead of having to kill the exe in task manager.

It doesn't appear to be hanging when it is polled by HP SIM.

[Rob Buxton]

The symptoms of this one showed up in the Cache files. When hung there were only 4 files and one was 0 length. When OK there should be 5 files. You can check that.