topic Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred. in Server Management - Systems Insight Manager

Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

Pber — Fri, 15 Jun 2012 15:53:38 GMT

I've recently upgraded my servers to use 9.10.00.0 of the server agents. Ever since the upgrade,ALL of my servers are now filling the system event log with Event ID 1182 Informational events indicating that "A Change in the health status of the server has occurred". This occurs every 2 minutes based on the SNMP polling frequency. It also sends traps to HPSim causing an Informational change of status on the Aggregate Status Icons as well. I can turn off the trap handling for the trap

cpqHoMibHealthStatusArrayChangeTrap - 11020 in CPQHOST.MIB in HPSim, but doesn't fix the problem of this alert filling up my system event logs.

Any ideas other than downgrading back to version 8.70.00.0 which wasn't an issue.

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

BERT_1 — Mon, 18 Jun 2012 06:33:36 GMT

Hello,

I watch the same effect since updates this weekend, but only on 64bit servers?

32bit servers seem not to have this behaviour.

Any solution or config advices?

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

Pber — Mon, 18 Jun 2012 18:53:19 GMT

Good call. I didn't look that closely since we only have a handful of machines that are 32bit. But I stand corrected, it is only x64 machines exhibiting this behaviour. My Windows 2003 x64 and 2008 R2 machines with the latest 9.1 agents are all doing this.

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

greg long_4 — Mon, 18 Jun 2012 20:49:51 GMT

I have the same problem, 2008 R2 SP2 64 bit

Also we are using images with sysprep

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

BERT_1 — Tue, 19 Jun 2012 09:22:26 GMT

Had a hp support call and was adviced to downgrade:

Older server models with Windows 2003 are yet only allowed to SP 8.70 !!

I did and got rid of these messages.

With HP-SIM the server according version for the reference support pack version should be looked for.

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

greg long_4 — Tue, 19 Jun 2012 15:55:23 GMT

I have back reved from HP Service Pack for Proliant 2012.06.0 to PSP 9.0

I removed the Version control agent, insight agent and home page and then installed the 9.00 version.

I now no longer get the 1182 error, but I would like to see HP acknowledge that there is an issue!!!!!

This has cost my company 16 to 20 tech hours troubleshooting and reloading machines as this error does not show itself right away or until you look at event log or configure notification.

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

greg long_4 — Tue, 19 Jun 2012 15:56:44 GMT

FYI We are on

DL380 G5 and DL360 G5 running 2008 R2 with SP2

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

MMetzner — Tue, 19 Jun 2012 17:07:26 GMT

hello,

i have the same problem with many servers proliant dl370 g6 and windows server 2003 r2 64 bit. i downgrade only to hp insight agents 8.70 and now i have no more events, i hope hp will fix this very fast.

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

Jockeman — Wed, 20 Jun 2012 07:04:37 GMT

Installed 2012-06 yesterday and I have same problem on DL380 G5 with Windows Server 2003 R2 x64

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

mbrownnyc — Tue, 26 Jun 2012 21:04:59 GMT

Same problem on Windows 2003 R2 32-bit, ML150 G5.

About every one-two minutes the Foundation Agents reports some of the services that are running locally (review the Data section of the event in the event log), then reports that they are unreachable. This cycle continues indefinitely.

Here is some of the data I have pulled from the Event log entry present when I was receieving Email notices stating an snmp trap was received with a Details stating NOT AVAILABLE :

#Domain Name Server.domain 53/udp
#Domain Name Server.bootps 67/udp

dhcps
#Bootstrap Protocol Server.bootpc 68/udp

dhcpc
#Bootstrap Protocol Client.tftp 69/udp
#Trivial File Transfer.gopher 70/tcp
.finger 79/tcp
.http 80/tcp

www www-http
#World Wide Web.hosts2-ns 81/tcp
#HOSTS2 NameServer.hosts2-ns 81/udp
#HOSTS2 NameServer.kerberos 88/tcp

krb5 kerberos-sec
#Kerberos.kerberos 88/udp

krb5 kerberos-sec
#Kerberos.hostname 101/tcp

hostnames
#NIC Host Name Server.iso-tsap 102/tcp
#ISO-TSAP Class0.rtelnet 107/tcp
#Remote Telnet Service.pop2 109/tcp

postoffice
#PostOffice Protocol-Version2.pop3 110/tcp
#Post Office Protocol -Version3.sunrpc 111/tcp

rpcbind portmap
#SUN Remote Procedure Call.sunrpc 111/udp

rpcbind portmap
#SUN R...WV
CPQTHRSH.HOSTALRT

I do not have many of these services running on the local machine.

When I received an alert email with no Details listed, the Foundation Agent event log event data in the System Event Log listed four processes that were and are running on the machine.

Reviewing: %systemroot%\system32\CpqMgmt\cqmghost\cpqmgmt.chm it appears that the Foundation Agent provides things such as process and service monitoring. You may be able to disable these two features and the problem may or may not go away (but this is what I'm looking to do).

After some investigation with process monitor, I see that the log entries correlate with cqmghost.exe (the Foundation Agents service) accessing two keys:

HKLM\SYSTEM\CompaqPermanent\

and

HKLM\System\CompaqHost\

In my monitoring, both of these seem to be accessed only by cqmghost.exe.

These appear to be a cache of some sort.

I wonder if some sort of issue exists when upgrading directly, without uninstallation first.

I will uninstall the HP Insight Management Agents, delete these keys, then reinstall later.

[update 1]
With the agents uninstalled, and the those keys deleted, the web UI is still populating without issue (since it uses WBEM or SNMP, in my case WBEM) [tested by reviewing sectors read and written on a RAID disk].

cp015760.exe is the file to install the component "HP Insight Management Agents for Windows Server 2003/2008" (taken from %systemdrive%\cpqsystem\hp\log\localhost\hpsum_log.txt)

In the summary of the previous install, I do see that SUM detected the older version of the component:

Component File Name: cp015760.exe
Component Name: HP Insight Management Agents for Windows Server 2003/2008
Original Version: 8.70.0.0
New Version: 9.10.0.0
Installation Result: Success

I ran cp015760.exe solo.

After install a window with manage agents ("C:\WINDOWS\system32\rundll32.exe" Shell32.dll,Control_RunDLL cpqmgmt.cpl /setup) pops up. Registry keys that were deleted are added again.

I unchecked Enable SNMP sets, noting that polling was set to 2 minutes by default... hmm... since I'm using the WBEM (wmi), and click OK.

A 1182 event is logged in the System Event Log.

Four minutes pass. No more 1182 events, because there is no more polling... Enabling polling causes events to be logged.

[Conclusion]

So the problem exists with the SNMP sets that are being polled. You can access the Management Agents configuration control panel applet by ruhning cpqmgmt.cpl or by accessing "HP Management Agents" from the Control Panel itself.

After this select your minimum usage. Since I figured out, by reviewing the Data logged in the 1182 events in the System Event Log, that there was at least one of the "Active Agents" that were causing a problem. I would go through trial and error to figure out which one is causing the issue. You will note that there is also a "Process Monitor" tab present, so this might be how the processes above were listed.

I began tested removing the Active Agents one-by-one. Luckily, Host Information (near the top of the list), seemed to be the problem cause.

I removed the Host Information agent from SNMP polling, waited 8 minutes, and only the single "OK" status change was logged (at three minutes).

I then re-added the Host Information agent, waited 3 minutes, saw the "OK" status change again. Then waited another 3 minutes, received the first status change event (1182, citing CPQTHRSH.HOSTALRT). It appeared that some processes were listed in the Event Data. I then waited another 5 minutes and nothing popped up.

The info is sparse, but the issue is clearly related to HOSTALRT, which is found in both events' Data.

For the sake of testing, I unchecked

[tl;dr]

Through the "HP Management Agents" control panel applet (cpqmgmt.cpl):

1) Try to clear all thresholds on the SNMP Settings tab...

2) If that fails, Try to disable Application Exception Traps...

3) If that fails, remove the Host Information agent from the Active Agents list in the "HP Management Agents" control panel applet. This clearly stops some polling from occurring, but I can not figure out which OIDs the Host Information Agent polls, or which data will be missing.

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

BERT_1 — Wed, 27 Jun 2012 07:27:41 GMT

Hello.

Again my earlier reply:

On a hp support call I got the advice, to respect the max allowed support pack, esp. for older server models and os releases. Newer agents may cause problems on this systems!

Within VCA from Insight Homepage you can see on "reference support pack" the last supported version.

I did so and all is running fine.

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

mbrownnyc — Wed, 27 Jun 2012 17:58:03 GMT

BERT_1: Is the "reference support pack" not dynamic according to the version available to the VCA from the Version Control Repository Manager, but has nothing to do with the server model number or the OS version?\

There is no Version Control Repository Manager server here.

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

BERT_1 — Thu, 28 Jun 2012 09:45:26 GMT

We manage about 60 phys servers and use a zentral VC repository for hp-sim.

In this constellation the "reference supportpack" seems to be dynamic according to model and installed os version.

Otherwise I give you the advice to take a look on the hp support matrix or check with a call to hp support.

Getting "only informational" entries in system eventlog is just harmless, but malfunctions on controllers are no fun.

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

MaxSys — Wed, 04 Jul 2012 16:46:00 GMT

Having this same issue on DL360 G4's and ML110 G7's (Running Server 2008 x64).

I'll just try to keep them all on outdated software for a while longer I guess.

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

Dan DeCoursey — Fri, 13 Jul 2012 18:34:32 GMT

I ran SPP 2012 on windows 2003r2 32 bit servers and I too see my logs filled every 2 second with this 1182 event

so its not just 64 bit installs

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

Dan DeCoursey — Mon, 16 Jul 2012 17:54:14 GMT

As I said in the above post I ran SPP 2012

I have checked (as recommendaed in one of these ealrier posts) the Reference support pack it currenlty says 8.70 so when I selectthat version after finishin that it goes back to the installed software page and there are namy itens that have newer versions listed under "latest version" areyou saying ishouls stay clars of those and alwasy just use whats listed in the Support pack Verison column??

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

BERT_1 — Tue, 17 Jul 2012 06:13:39 GMT

HP support says: Check info text of each item if it is according to your server model and os, - or check by a support request.

I am confused for myself what may and what must be done...

Imho i think it is a good plan to update to the allowed level in general and do further (corrective) updates only on verified troubles.

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

MaxSys — Tue, 24 Jul 2012 19:45:39 GMT

Official Response from HP Support:

This issue is reproduced in our internal lab and has already been escalated to our development team.   We are working on a Customer Advisory to document this issue as it is effecting many customers.  A formal fix has been submitted to the 9.20 version of agents that are slated to be released in August 2012.

The workaround is to reinstall the management agents v 9.0.
If you are not willing to downgrade to 9.0 another workaround would be to disable the HOST INFORMATION sub-agent in the Control Panel applet, which will eliminate the three events from occurring on each polling cycle (default polling cycle for snmp is every 2 min).  If you are also running wbem providers this alternative workaround will not disrupt your hardware monitoring.  However, if you are only monitoring these servers by means of snmp this alternative workaround is not recommended as snmp hardware monitoring depends on the host sub-agent.

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

mbrownnyc — Tue, 24 Jul 2012 19:43:58 GMT

thanks.

Re: Event ID 1182 Foundation Agent: A Change in the health status of the server has occurred.

matiasfortinero — Wed, 22 Aug 2012 17:40:25 GMT

Hi, Is there any news about the fix for this issue?

Thanks

Matias