HP-health issue (centos 7 on proliant DL360 G6)

[edg91]

Hello,

I performed a fresh install of centos 7 x64 on a proliant DL360 G6.

At the beginning, HP-health installation seemed to be success, I could use hpasmcli to check hardware, e.g. :
# hpasmcli
hpasmcli> show powersupply
Power supply #1
Present : Yes
Redundant: Yes
Condition: Ok
Hotplug : Supported
Power : 35 Watts
Power supply #2
Present : Yes
Redundant: Yes
Condition: Ok
Hotplug : Supported
Power : 60 Watts

And succeeded to connect to the web gui (hpsmhd) and to get the devices informations through it.

However, now the hp-health service cannot work anymore.
hp-health service  seems to start  but I get a message relative to ipmi timout (/var/log/messages) :
"hpasmxld[8848]: StartIpmiRmqThread
hpasmxld[8848]: StartIpmiRmqThread: Thread created (TID -482785536)
hpasmxld[8848]: IpmiRmqThread: Entry
hpasmxld[8848]: OsKcsExecCmd: Expected Cmd (0x2f), Msg ID (0x2)
hpasmxld[8848]: OsKcsExecCmd: Received Cmd (0x31), Msg ID (0x1)
hpasmxld[8848]: OsKcsExecCmd: IPMICTL_RECEIVE_MSG_TRUNC returned EAGAIN
hpasmxld[8848]: OsKcsExecCmd: Expected Cmd (0x31), Msg ID (0x1)
hpasmxld[8848]: OsKcsExecCmd: Received Cmd (0x2f), Msg ID (0x2)
hpasmxld[8848]: OsKcsExecCmd: IPMICTL_RECEIVE_MSG_TRUNC returned EAGAIN"
. . .
"hpasmxld[8848]: hpIoctl(): IOCTL Timeout -- Waiting on IPMI"

And when I try to stop it, I get this :
"hpasmxld[8388]: check_ilo2: Failed to get BMC Device Id!
hpasmxld[8388]: The Integrated Lights-Out Management Processor is not responding!"
I tried to start hp-health via "systemctl start hp-health" or via the script "/usr/lib/systemd/scripts/hp-health.sh start" ; but I faced to the same issues.

I  also checked status of snmpd service :

# systemctl status snmpd
● snmpd.service - Simple Network Management Protocol (SNMP) Daemon.
Loaded: loaded (/usr/lib/systemd/system/snmpd.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2016-04-08 10:01:44 CEST; 1s ago
Main PID: 8884 (snmpd)
CGroup: /system.slice/snmpd.service
└─8884 /usr/sbin/snmpd -LS0-6d -f

systemd[1]: Starting Simple Network Management Protocol (SNMP) Daemon....
snmpd[8884]: init cmaX.
snmpd[8884]: cmaX: sent ColdStarts on ports 25376 to 25393
snmpd[8884]: NET-SNMP version 5.7.2
systemd[1]: Started Simple Network Management Protocol (SNMP) Daemon..

For information, this is my ilo 2 conf :
# hponcfg -w hostname.xml
HP Lights-Out Online Configuration utility
Version 4.6.0 Date 09/28/2015 (c) Hewlett-Packard Company, 2015
Firmware Revision = 2.05 Device type = iLO 2 Driver name = hpilo

The only thing that I have done between the beginning (the moment where hp-health was working) and now was to change the swap size (extend the partition size for the swap) and to use the command :
# dracut --force --regenerate-all

I try also to uninstall and reinstall hp-health, but got the same issue... I use the hp repos :
http://downloads.linux.hpe.com/SDR/repo/spp/RHEL/7.1/x86_64/current
and
http://downloads.linux.hpe.com/SDR/repo/mcp/centos/7.1/x86_64/current/

It is as if something was broken in the ipmi configuration of the server...

May you help ?

Best regards

[edg91]

I upgrade this morning the firmware version of ILO 2 (from 2.05 to 2.29)

So after the upgrade, I reboot the server and restart hp-health

Suprise, the service started well !

I could use hpasmcli

Unfortunatly, I also restart hp-health : # systemctl restart hp-health

and the same issue came back, with the "hpasmxld[6080]: OsKcsExecCmd:  IPMI NetFN  0x6   CMD: 0x31 has timed out!" in /var/log/messages

Is there a bug between /opt/hp/hp-health/bin/hpasmxld and ipmi ??

[Nico57]

Same here on a pair of DL360 G5's.

The problem showed up after upgrading to kernel 3.10.0-327.13.1.el7.x86_64.
Resolved it by downgrading to 3.10.0-327.10.1.el7.x86_64.

If 10.1 is still installed on your system, you can run

grub2-set-default "CentOS Linux (3.10.0-327.10.1.el7.x86_64) 7 (Core)"

then reboot, and see what happens.

A newer kernel, 3.10.0-327.18.2.el7.x86_64, has been out for a few days now.
I've not tried booting to it yet, and came here looking for someone who did.

[Nico57]

Bug appears to be fixed in 18.2 indeed.
Running it with no error.

[tonioc]

Same issue observed on a ML350-G5 with rhel6.8 - kernel 2.6.32-642.
The problem disappeared after restarting hp-health service.

Any suggestions ?

[maxtdef]

DL360 G6, DL380 G6 

Version 4.6.0 Date 09/28/2015 (c) Hewlett-Packard Company, 2015
Firmware Revision = 2.29

hp-health-10.40-1777.17.rhel7.x86_64.rpm

hp-snmp-agents-10.40-2847.17.rhel7.x86_64.rpm

kernel 3.10.0-327.36.3.el7.x86_64

Same issue observed on a CentOS6.

The previos version 10.30 works fine on CentOS 6,7.