7.30 Insight Foundation Agents service terminated unexpectedly..

[Ian Grobler]

Scenario as follows:
Every evening when HP Data Protector created volume snapshots are presented as part of a backup, the HP Insight Foundation Agents service on the servers being backed up terminates unexpectedly. This occurs +-1 minute after the snapshot disk is presented and is a problem on all Windows Server 2003 Standard Edition servers with SP1 and PSP 7.30 installed (Connected to EVA3000 VCS 3020).

After the Server 2003 upgrades to SP1, PSP 7.30 was installed to resolve the 'Unknown SNMP device' problem with SP1 and 7.10 agents and since then the Foundation Agent has been acting up with the Snapshot process. Pre-SP1 the servers were running PSP 7.10 agents without issue.
Currently All Insight agents are active in the control panel 'HP Management agents' section except for 'clustering & rack information'.

I am doing some tests tonight with the 'fibre agent' support disabled completely in Control Panel. Anyone have an idea why the Foundation Agents in 7.30 are terminating when Snapshot disks are presented from the EVA via DP?

From SYSTEM event logs:
Event ID: 7034 - The HP Insight Foundation Agents service terminated unexpectedly.
P1 : cqmghost.exe P2 : 7.30.0.0

If I revert back to 7.10 then obviously I can't manage the servers properly with SP1.. Any feedback welcome.

[Ian Grobler]

So far no luck with this. Adding/Removing the Fibre agent made no difference. Someone at HP must know what changed between the 7.10 and 7.30 agents and possible issues with EVA Snapclone/Snapshots being presented. I am pretty sure I can't be the only one running 2003 SP1 with PSP7.30 using EVA CA technology. Help?

[Ian Grobler]

Upgrated to PSP 7.40 which unfortunately did not resolve the problem. The foundation agents still terminate when a snapclone is presented to the server(s). Will log it with HP methinks!

[Aravindh Rajaram]

Is the HP System Management Agent running properly? Please do check that. SMH is the key for other agents to run properly.

[Ian Grobler]

The system management homepage is running without issue and configured allowing local and anonymous 'read' access etc. This agent does not terminate and functions 100% when the foundation agent has terminated (even though some info is then obviously missing)

In normal operations the agents, version control etc work without issue and the only trigger for the termination is the snapshot presentation from the EVA. As this occurs on all 4 2003SP1 servers that have snapclones presented I don't think it is configuration related but rather a bug in the 7.3/7.4 agents. Any ideas welcome.. :-)

[Didier Perrinjaquet]

I have exactly the same problem on my DataProtect Cell Manager 5.5 on a windows 2003 cluster with PSP 7.30 (since PSP 7.20) - No problem with PSP 7.10

I mount all my snapshot/snapclone with SMIS integration (EVA3000 & EVA5000) on one node of the cluster and the "Insight Foundation Agents service" crash only on that node.

When I mount them on the other cluster node, the "Insight Foundation Agents service" crash on the other node...

[Ian Grobler]

I implemented the new HP System Management Homepage v2.1.3.132 (21 Sep 05) on one of the problem 2003 servers yesterday and surprisingly the foundation agent did not crash during the snapclone presentation last night. I am now going to implement the new SMH on the other servers with the same symptoms ASAP and see if this works for those as well.

http://h18004.www1.hp.com/support/files/server/us/download/23560.html

[Ian Grobler]

Just an update...
Unfortunately the new SMH has not resolved the problem. Even the original box which I installed the updated SMH on and thought was fixed still exhibits the Foundation Agents termination problem.
Back to the drawing board for now :-(

[Richard Drouin]

Interesting, more or less the same problem here on a dozen 2003 servers, Foundation service dies within a minute of starting. They are connected to an EVA8000. Will investigate with my storage group for the configuration of CA, might be related.
For now I disabled the Foundation agent service.
Please update if someone found a fix.
Thanks, Richard

Event Type: Error
Event Source: Application Error
Event Category: (100)
Event ID: 1000
Description:
Faulting application cqmghost.exe, version 7.41.0.0, faulting module unknown, version 0.0.0.0, fault address 0x00000000

[Ian Grobler]

The following advisory "HP Foundation Agent Version 7.40 or 7.41 for Windows 2000/Windows Server 2003 Experiences an Application Fault on an HP ProLiant Server Configured with a PSFS File System" issued last week may have led to some clarity on this problem (http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c00612467&jumpid=reg_R1002_USEN)

This advisory though seemingly unrelated refers to filesystem/volume names with lengths of more then 64 Bytes exceeding a buffer limit.
The workaround for the mentioned problem is to disable the Performance Monitor subagent from the HP Insight Management Agents applet in the Windows Control Panel.

As the symptoms described seemed very similar I have tried disabling the Performance Monitor subagents on two servers.. lo and behold it does seem to have resolved the problem!
The Foundation Agents do not crash when DP snapclones are presented as before. I have now implemented this on a further 2 problem servers and will know if this is successful this evening. Will keep you posted..

[Ian Grobler]

Disabling the Performance Monitor subagent on the other 2 servers has indeed resolved/worked around the problem. As per the advisory I will test when the next version of agents (7.50?) ship.

Thanks for all the comments... Ian