HP ProLiant DL360 Gen10 with SLES11 SP4 : hp-health fail to start: check_ilo2: BMC Returned Error

[GregCabaj]

Hi All, Dear Community, 

I rarely use forums, but finally I hit the problem I cannot find solution to, and after spending a few days of troubleshooting I thought to ask for help here. Hopefully someone went through a similar issue and will be able to share the knowledge. 

After successful installation of SLES11 SP4 on HP DL360 Gen10, I was trying to install hp-health and hp-snmp-agents. Unfortunately hp-health service hangs up indefinitely during the startup. 

The errors in /var/log/messages

hpasmlited[10046]: check_ilo2: BMC Returned Error: ccode 0x0, Req. Len: 15, Resp. Len: 21
hpasmlited[10046]: The Integrated Lights-Out Management Processor is not responding!

System details: 
HP ProLiant DL360 Gen10
OS: SLES11 SP4
Kernel: 3.0.101-107-default
Kernel modules:
lsmod | grep hp
hpwdt 14212 0
hpilo 17496 0
ILO5: version 1.20 (also tested with 1.15 with same result)
HP packages: 
hp-health-10.70-1846.3.sles11
hp-snmp-agents-10.70-2962.5.sles11
(also tested with older versions with the same result)

The ILO5 looks healthy and is responsive. I just cannot access it from OS side. What else should I check? IPMI libraries are there: 

/lib/modules/3.0.101-107-default/kernel/drivers/char/ipmi/

Has anyone came across same problem? 

I would appreciate some hints, hence currently it is not possible to monitor the server with HPE OIDs. 

Many thanks,
Greg

[sergiuc]

Same here

# cat /etc/debian_version
9.3

# dpkg -s hp-health
Package: hp-health
Status: install ok half-configured
Priority: optional
Section: admin
Installed-Size: 2390
Maintainer: bryan gartner <bryan.gartner@hp.com>
Architecture: amd64
Version: 10.61-1-0

Description: hp System Health Application and Command line Utility Package
This package contains the System Health Monitor for all hp Proliant systems
with ASM, ILO, & ILO2 embedded management asics. Also contained are the

Mar 23 11:34:56 systemd[1]: hp-health.service: Control process exited, code=exited status=1
Mar 23 11:34:56 systemd[1]: Failed to start LSB: hpe System Health Monitor and Command line Utility Package..
Mar 23 11:34:56 systemd[1]: hp-health.service: Unit entered failed state.
Mar 23 11:34:56 systemd[1]: hp-health.service: Failed with result 'exit-code'.
Mar 23 11:34:57 hpasmlited[2534]: check_ilo2: BMC Returned Error: ccode 0x0, Req. Len: 15, Resp. Len: 21

I suppose it will not work, as G10 have ILO 5...

[JJSS]

Hi,

You are not alone.

We are also having the same problem using SLES 12 SP3.

hp-health-10.70-1846.2.sles12.x86_64

[Jimmy Vance]

The health/snmp functionality was moved to iLO on HPE ProLiant Gen10 servers. hp-health, hp-snmp-agents, hp-smh* and hp-ams are for Gen9 servers and earlier.

Gen10 ProLiant use a new AMS (Agentless Management Service) service along with SMA (System Managmenet Assistance)

Refer to these documents

HPE Integrated Lights Out (iLO 5) for Gen10 Servers - Configuring iLO Management Settings
https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00026111en_us&docLocale=en_US#N10256

HPE iLO 5 User Guide Starting on page 285
https://support.hpe.com/hpsc/doc/public/display?sp4ts.oid=1008876525&docLocale=en_US&docId=emr_na-a00039434en_us

[GregCabaj]

Hi Jim, 

Many thanks for the update. Appreciate it very much. I will go through the documents and give feedback here. 

Regards.

[GregCabaj]

Hi Jim, 

I have posted here a reply, which I cannot see anymore. 

We used hp-health package to automate some tasks ie. marking IML logs as resolved with hpasmcli. 

Unfortunately, the links you provided do not cover the subject on how IML logs could be marked as "resolved" via ILO's CLI. 

Thanks.

[GregCabaj]

Below is the example, showing that with ILO5 CLI setting severity to "Repaired" is not supported. The hpasmcli command on earlier ProLiant generations was allowing for this. 

mgmtserver:~ # ssh hpgen10ilo -l Administrator "show system1/log1/record25"
Administrator@hpgen10ilo's password:
show system1/log1/record25
status=0
status_tag=COMMAND COMPLETED
Mon Apr 16 13:53:59 2018

/system1/log1/record25
Targets
Properties
number=25
severity=Caution
date=04/16/2018
time=09:12:28
description=IMPORTANT: The Standard Boot Order (IPL) has been detected as corrupted and has been restored to default values.
Verbs
cd version exit show

################### Fix Attempt ######################################

mgmtserver:~ # ssh hpgen10ilo -l Administrator "set system1/log1/record25 severity=Repaired"
Administrator@hpgen10ilo's password:
set system1/log1/record25 severity=Repaired
status=2
status_tag=COMMAND PROCESSING FAILED
error_tag=COMMAND ERROR-UNSPECIFIED
Mon Apr 16 13:56:20 2018

Command is not supported for this property.

[Jimmy Vance]

I don't think you can mark repaired through the iLO CLI, using the CLI about all you can do with IML is clear the entire log. After checking around I have been told the HPE RESTful interface tool 'ilorest' can perform this function.  Here is the link to the tools documetnation. I'm not well versed with the tool so I can't provide and example for your task

https://hewlettpackard.github.io/python-redfish-utility/#overview

[GregCabaj]

The ilorest does not seem to be very helpful here and "clearlog" is not an option. Historical information on hardware is critical for infrastructure teams. 

iLOrest > serverlogs --selectlog=IML --repair
Usage: serverlogs [LOG_SELECTION] [OPTIONS]

Download the AHS logs from the logged in server.
example: serverlogs --selectlog=AHS

Clear the AHS logs from the logged in server.
example: serverlogs --selectlog=AHS --clearlog

Download the IEL logs from the logged in server.
example: serverlogs --selectlog=IEL -f IELlog.txt

Clear the IEL logs from the logged in server.
example: serverlogs --selectlog=IEL --clearlog

Download the IML logs from the logged in server.
example: serverlogs --selectlog=IML -f IMLlog.txt

Clear the IML logs from the logged in server.
example: serverlogs --selectlog=IML --clearlog

(IML LOGS ONLY FEATURE)
Insert entry in the IML logs from the logged in server.
example: serverlogs --selectlog=IML -m "Text message for maintenance"

Download logs from multiple servers
example: serverlogs --mpfile mpfilename.txt -o outputdirectorypath --mplog=IEL,IML

Note: multiple server file format (1 server per new line)
--url <iLO url/hostname> -u admin -p password
--url <iLO url/hostname> -u admin -p password
--url <iLO url/hostname> -u admin -p password

Insert customized string if required for AHS
example: serverlogs --selectlog=AHS --customiseAHS "from=2014-03-01&&to=2014-03-30"

(AHS LOGS ONLY FEATURE)
Insert the location/path of directory where AHS log needs to be saved.
example: serverlogs --selectlog=AHS --directorypath=C:\Python27\DataFiles

ilorest: error: no such option: --repair

Also when trying to use get or list command, it does not seem to be possible to even filter with "repaired" property. 

If this command works fine:

iLOrest > list Message --filter "OemRecordFormat=Hpe-IML"

why below returns error??

iLOrest > list Message --filter "/Oem/Hpe/Repaired=False"
Full data retrieval enabled. You may experience longer download times.
No type currently selected. Please use the 'types' command to
get a list of types, or input your type by using the '--selector' flag.

iLOrest > list Message --filter "/Oem/Hpe/Repaired=False" --selector=log
Full data retrieval enabled. You may experience longer download times.
Unable to locate instance for 'log' and filter '/Oem/Hpe/Repaired=False'

iLOrest > list Message --filter "Members/Oem/Hpe/Repaired=False" --selector=log
Full data retrieval enabled. You may experience longer download times.
Unable to locate instance for 'log' and filter 'Members/Oem/Hpe/Repaired=False'

HPE decisions on software moves and server managment limitations are very disappointing. Documentation for iloRest is at least poor, with examples for windows only??? What world do you guys live in?? 

https://hewlettpackard.github.io/python-redfish-utility/#overview

The vision of ordering another 300 Gen10 DL360s is at least worrying. 

[Jimmy Vance]

As a follow-up., after discussing what you were previously doing with the old tools, and that there is a now a gap in functionality that the older tools have been retired. The RESTful team is working to add the marked repaired option into the RESTful tool. 

[GregCabaj]

Jim , 

There have been two releases of restful api since we spoke last time and I still do not see the IML logs repair feature. 

Could you please update everyone here? 

Thanks. 

[MyriadFrance]

Hello,

We still do not get how to retreive info with amsd. hpasmcli was a simple tool. We just want to know the equivalent with HP G10.

Regards.

[lttch]

The health/snmp functionality was moved to iLO on HPE ProLiant Gen10 servers. hp-health, hp-snmp-agents, hp-smh* and hp-ams are for Gen9 servers and earlier.  Gen10 ProLiant use a new AMS (Agentless Management Service) service along with SMA (System Managmenet Assistance)

Jimmy, thank you so much for this information.  It might seem obvious/simple, but it tripped me up like it did others.  Even years after you posted, it is still helping people

[SysAdminCanada]

Hi there !

Google for ilorest, works just like hpasmcli.

Here are some examples:
ilorest serverinfo --memory
ilorest serverinfo --thermal
ilorest serverinfo --fans
ilorest serverinfo --power

# Download link:
https://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p1545251609/v211652/RPMS/x86_64/ilorest-3.5.1.0-8.x86_64.rpm

# Documentation resources
iLO REST APIs: https://github.com/HewlettPackard/ilo-rest-api-docs
iLO REST API tutorials: https://hewlettpackard.github.io/ilo-rest-api-docs/ilo5/
RESTful Interface Tool APIs: https://github.com/HewlettPackard/python-redfish-utility
RESTful Interface Tool tutorials: https://hewlettpackard.github.io/python-redfish-utility/

Transition to the RESTful Interface Tool (iLOREST): https://support.hpe.com/hpesc/public/docDisplay?docId=a00061427en_us

Have a great day !  =]