Version Control Agent hangs

[Dirk Frey]

Hello,

I have some trouble with VC for some time.

The VC service hangs time and again. Then the service can not be stopped by the Net Stop command. Only killing the process vcagent.exe stopps the VC Agent.

As result of the hanging VC the SIM shows "Software Version Status: Unknown - Version VC Agent not configured." for the affected servers, and the SMH starts very slow (often without the VC section).
This error occurs on all of our servers on different times.

I have installed SIM 5.3 with SP1, SMH 3.0.2.77 and VC 2.2.0.820 running on Windows Server 2003/2008.

Any helpful suggestions?
Thanks

[TechGuru]

Dirk Frey,

Did it ever work with older VCA or HPSIM?

check VCA log for any error.

[Dirk Frey]

Hello Logapriyan,

Thanks for reply.

On most servers VCA has been updated continously. But even a complete reinstallation of VCA did not solve the problem.

In the VC Logs are 2 errors numerous reported:

1)
Severity: Error
Task ID: vc_agent_task
Date / Time: 25.08.2009 00:02:10
Message: An error occurred while parsing XML data.
Additional Data:
XSLTTransform::CompileStylesheet - SAX Exception!
File:
Line: 0
Col: 0
Msg: An exception occurred! Type:MalformedURLException, Message:No protocol prefix present

2.
Severity: Error
Task ID: vc_agent_task
Date / Time: 25.08.2009 00:02:10
Message: An unknown exception occurred while processing a request.
Additional Data:
XSLTTransform::applyTransform
File:

I have no idea what does that mean.

[TechGuru]

you can browse to VCA using https://:2381/vcagent even if SMH don't have VC section.

Dose this issue occur only from 820 VCA?

did you try 800 or 810 VCA in failing system?

Any perticular difference between Working VCA & Non-Working VCA systems?

Make sure all the files are present in c:\hp\hpsmh\data\cgi-bin\vcagent\xsl folder.

[Dirk Frey]

In case of hanging VCA process I can browse https://:2381/vcagent but the VC page replies an internal error "The server encountered an internal error or misconfiguration and was unable to complete your request." or "An error occurred while formatting the page for display.".

Once the VC service ist restarted it running well for some time and then at any time the VC service hangs. I don't know why.

We just have VCA 2.2.0.820 in use. I didn't try older versions.
It seems that all of our servers are affected.

[TechGuru]

Try to rename "Discovery" folder,
Install SNMP & HP Insight Agent and configure SNMP with any valid community string.

Above should make VCA work with SNMP inventory source & it might work well for atleast browsing to VCA Home Page.

[Dirk Frey]

... ok, I try that on 2 servers. Let's wait over weekend to see how it works.

[Dirk Frey]

... unfortunately the renaming of the Discovery folder und reinstallation of agents did not have a positive effect.

The VCA hung again after some time.

[Rob Buxton]

Just starting to look at this myself. Seeing it on recent versions. Our servers happen to be blades. Not sure if that's a common theme or not.

[Rob Buxton]

Anyone get an answer to this.
Certainly seems to be an issue with the more recent versions of VCA, including version 2.2.1.830 which I believe is the most recent.
We're seeing similar errors in the log file as noted above.

[Melanie Sang]

We are having the exact same problem on all our servers. We are running the latest PSP packs, and the only way to get the VC working is to kill the process on the server. We are getting the same errors as above. Anyone found out yet whats causing the problems?

[Dirk Frey]

Sorry, I also did not found still a solution but some new facts.

I have set up 3 new servers (2k3 R2, 2k8). It is interesting, on these servers the VCA works fine up to now and also in the SIM the Software Version Status is correctly indicated.

On the other hand a complete uninstallation of all HP Agents and reinstalltion with the newest PSP 8.30 does not eliminate the problem.

So it seems in such a way as if the problem is caused by the updates history of the HP Agents.

[dougjef]

Same issue here on hundreds of DL380's. I started seeing an issue with the Version control agent starting in 8.15. 8.25 had the same issue and now 8.30 does as well. Killing the vc process and restarting the service gets the service to respond again..but then after a while if you go to the system management homepage it will timeout until you kill the version control agent again. Very frustrating. Hopefully they will fix this as its impossible to update these programattically now and they have to be manually updated as vc is not responding on hundreds of hosts.

[ajc79]

If it's affecting all your servers, have you had a look at the actual VCRM server rather than VCagents? In my experience, the vcagent can seem to hang the System Management Homepage if it's waiting for the VCRM server to respond.

[ajc79]

dougjef, are you setting you SNMP strings via group policy?

If so, version: 2.2.1.830 of the VC agent has a fix for "VCA causing SMH to timeout when the SNMP community string is set using Group Policy editor"

[ajc79]

Actually ignore that, looks like 2.2.1.830 is the version that comes with PSP 8.30, so you're probably on that already.

I've got a few boxes with this issue too - killing the vcagent process then restarting the service seems to fix it for a while, then it hangs again (via the both the SMH or the vagent page)

There's another thread with the same problem here:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1370491

[Stan Oleszkowicz]

I just got off the phone with Support on this issue. We appear to have had some success in at least establishing a work-around.

With a server experiencing this problem, stop the Version Control Agent service. This may require killing the vcagent.exe process from within Task Manager if restarting the service within the Services MMC times out. After the service has been stopped (cleanly or forcibly) start the Version Control Agent using the Services MMC (you may need to refresh the window first if the service timed out when stopping).

Once the VCA has started browse to the System Management Homepage of the server exhibiting the problem and click on the link for the Version Control Agent within the SMH. This work-around should work until the server is rebooted or the Version Control Agent service is restarted. If the server is rebooted or VCA restarted you will have to again browse to the Version Control Agent page within the System Management Homepage to re-establish the work-around.

I have had this work-around in place for only an hour so far (give or take), but expectations are high enough I thought it worth sharing at this point.

What appears to be happening is this:

When VCA starts or stops the all files within the C:\hp\hpsmh\data\cgi-bin\vcagent\cache directory are removed (files only, not folders). If HP SIM does a software status poll to the server before a user manually browses to the Version Control Agent Page then the contents of the C:\hp\hpsmh\data\cgi-bin\vcagent\cache directory are created incompletely. Manually browsing to the VCA page causes the cache files to be created correctly and both HP SMH and HP SIM can collect VCA information successfully.

[Steve weeks_2]

That's not really a viable workaround on approx 200 servers i have where VCA is playing up though.

It would take me a week !

[Rob Buxton]

This does seem to be related to the SW Polling Task.
Can anyone else confirm they get the XML errors at around the same time as the SW Polling task is run?

[NJK-Work]

I have actually had this problem for years - it has been happening long before the 8.x versions.

I agree with Rob - it is definitely related to the SW polling job. It got so bad for me that I ended up disabling the job because it was actually causing problems with the SMH (which seems to be closing tied to the VCAgent - SHM hangs if VCAgent is hung).

I also ended up writting a little batch file to kill the process, restart SHM, and then restart VCAgent:

taskkill /f /im vcagent.exe
net stop sysmgmthp
net start sysmgmthp
net start cpqvcagent

Nelson

[Rob Buxton]

I've opened a call with HP.
Unlike Nelson, I've not seen this until recently.
It is easy to generate the condition here.
Restart the VCA, then immediately run the SW Polling task. In the C:\hp\hpsmh\data\cgi-bin\vcagent\cache directory only 4 files are created and one (loclinv.xml) is zero length.
Once done the VCA will hang on requests to it.

To correct, kill / restart the VCA and then immediately browse to the VCA. This time 5 files will be created and loclinv.xml will not be zero length.
I'll update when I get a significant update. Others may want to open calls as it may help in the overall diagnosis. Reference this thread in the call.

[Stan Oleszkowicz]

Rob:

That's the same process I went through to reproduce the problem with the support case I logged. Unfortunately, we were only able to reproduce the problem on the CMS using this method. Are your results also limited to the CMS?

[Rob Buxton]

No, I can reproduce the error at will on any number of different servers.
Mostly on PSP 8.10 and above. Not sure if that makes a difference.
Also I've done this on x86 and x64 variants. All Servers W2K3 SP2 with some R2 variants.

[Rob Buxton]

I've tested a beta version of the next release of the VCA. It resolves this issue, well it did here on the servers I tested. No idea about release dates, just keep a look out for version 2.2.2.835 or later.

[simonwirral]

any news on the new (and hopefully working) version of VCA?

Thanks