CentOS 6.2 + hp-health = fail

[linko22]

Hello!

I have to upgrage my DL160 G6 server from CentOS 6.0 to CentOS 6.2 and my hp-health stop working now.

I have strange messages in my /var/log/messages:

Dec 29 13:13:17 s14 hpasmpld[2918]: ehpsmb_parse_SMBIOS: SMBIOSInitTable was not successful.
Dec 29 13:13:17 s14 kernel: hpasmpld:2918 map pfn expected mapping type uncached-minus for 9e000-a0000, got write-back
Dec 29 13:13:17 s14 kernel: hpasmpld[2918]: segfault at 0 ip 0000000000414008 sp 00007fff08fb8df8 error 4 in hpasmpld[400000+2a000]

I've googled those messages - but found nothing, google say "No results containing all your search terms were found."

I have upgrade BIOS, BMC to latest verrsion, try use BMC version 4.22 4.21 4.20 and still nothing.

IPMI load seccess, /dev/ipmi0 present, dmidecode say me "SMBIOS 2.7 present.",

IPMI Device Information
Interface Type: KCS (Keyboard Control Style)
Specification Version: 2.0
I2C Slave Address: 0x10
NV Storage Device Address: 1
Base Address: 0x0000000000000CA2 (I/O)
Register Spacing: Successive Byte Boundaries

Also i try use

hp-health-8.6.2.2-14.rhel6.x86_64.rpm

hp-health-8.7.0.22-11.rhel5.x86_64.rpm

hp-health-8.7.0.22-17.rhel6.x86_64.rpm

Nothing.

I have another HP DL160 G6 server with CentOS 6.2 - at him all work fine.

What do i wrong?

[fke]

Hi linko22,

the **exact** same problem here. Could you get any further insight?

Thanks, Florian

[Matt-Park]

I have the same problem. 

12x DL160 + RHEL6.2 + hp-health = FAIL segfault

1x DL160 + RHEL5.8 + hp-health = SUCCESS

7x DL385 + RHEL6.2 + hp-health = SUCCESS

It's stuff like this HP.  Also if you have the broken package installed it throttles your processor because it thinks it's overtemp. *sigh*

[fke]

Hi Matt,

thanks for your reply. 

Did you open a bug?

And can you tell me the version of the installed RPMs  you're using where it's working?

Thanks, Florian

[Marco Aroldi]

Hi

same problem here on two DL180 G6 + Centos 6.2 x86_64

There are some news about this issue?

hpasmpld[17567]: ehpsmb_parse_SMBIOS: SMBIOSInitTable was not successful.
kernel: hpasmpld:17567 map pfn expected mapping type uncached-minus for 9e000-a0000, got write-back
kernel: hpasmpld[17567]: segfault at 0 ip 0000000000413fd8 sp 00007fff83116068 error 4 in hpasmpld[400000+2a000]

 Thanks

[SirKliv]

Same problem here...

HP ProLiant DL180 G6  , BIOS O20 01/24/2011

CentOS release 6.2 (Final) x86_64

localhost hpasmpld[31381]: ehpsmb_parse_SMBIOS: SMBIOSInitTable was not successful.
localhost kernel: hpasmpld:31381 map pfn expected mapping type uncached-minus for 9e000-a0000, got write-back
localhost kernel: hpasmpld[31381]: segfault at 0 ip 0000000000413fd8 sp 00007fffefd42668 error 4 in hpasmpld[400000+2a000]

[fke]

Hi SirKliv,

to cut it short: What finally did it for me, was ask HP to replace the mainboard. It's working flawless now.

Florian

[tomtom1234]

I also have just encountered this bug.  The same exact DL160G6 machines that I had running OpenSUSE 11.2 worked witht the SLES11 8.5.0 hp-health.  Reloading those machines with CentOS 6.2 produces the same bug that everyone else here has identified.  I have a hard time believing this is hardware related.  Looks to me it's a bug in 8.6 and beyond as the 8.5 RHEL driver does appear to work.  My issue occurs with the use of the SNMP package which at the 8.5 level doesn't work with the CentOS6.2 snmp library levels. 

[Bunny82]

This one just died on me too. same error. RHEL6.3!

Linux ichigo 2.6.32-279.el6.x86_64 #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

Red Hat Enterprise Linux Server release 6.3 (Santiago)

hpasmpld:6989 map pfn expected mapping type uncached-minus for 9e000-a0000, got write-back
hpasmpld[6989]: segfault at 0 ip 0000000000414918 sp 00007fff43d2bf38 error 4 in hpasmpld[400000+2a000]

hp-health.x86_64 9.1.0.42-54.rhel6 @HP-ProLiantSupportPack
hp-snmp-agents.x86_64 9.1.0.57-51.rhel6 @HP-ProLiantSupportPack
hpdiags.x86_64 9.1.0-301 @HP-ProLiantSupportPack
hponcfg.noarch 4.0.0-0 @HP-ProLiantSupportPack
hpsmh.x86_64 7.1.0-16 @HP-ProLiantSupportPack
kmod-hpsa.x86_64 3.0.0-8.el6 @HP-ProLiantSupportPack

Not sure why it is HP-ProLiantSupportPack since it seems to be configured for SPP

downgrading to 9.0.0 or 8.7.0 does not cure it, the coredump is the same:

hpasmpld:7501 map pfn expected mapping type uncached-minus for 9e000-a0000, got write-back
hpasmpld[7501]: segfault at 0 ip 0000000000414808 sp 00007fff088e1068 error 4 in hpasmpld[400000+2a000]

^^-- 9.0

hpasmpld:7727 map pfn expected mapping type uncached-minus for 9e000-a0000, got write-back
hpasmpld[7727]: segfault at 0 ip 0000000000414008 sp 00007fff2935a748 error 4 in hpasmpld[400000+2a000]

^^-- 8.7

[xonxoff]

We have a DL180G6 with CentOS 6.3 and are getting :

# service hp-health start
  Using Proliant Standard
        IPMI based 1XX System Health Monitor
  Using standard Linux IPMI device driver
Starting ipmi drivers:                                     [  OK  ]
  Starting Proliant Standard
        IPMI based 1XX System Health Monitor (hpasmpld):
hpasmpld -f /dev/ipmi0 xxxx
/etc/init.d/hp-health: line 666: 13690 Segmentation fault      (core dumped) $PNAME $PARGS < /dev/null >> $LOGFILE 2>&1
                                                           [FAILED]

# dmesg | tail

<snip>

hpasmpld:13690 map pfn expected mapping type uncached-minus for 9e000-a0000, got write-back
hpasmpld[13690]: segfault at 0 ip 0000000000414918 sp 00007fff93b30be8 error 4 in hpasmpld[400000+2a000]

# tail /var/log/messages

<snip>

Jul 20 14:44:54 sd0 hpasmpld[13690]: ehpsmb_parse_SMBIOS: SMBIOSInitTable was not successful.
Jul 20 14:44:54 sd0 kernel: hpasmpld:13690 map pfn expected mapping type uncached-minus for 9e000-a0000, got write-back
Jul 20 14:44:54 sd0 kernel: hpasmpld[13690]: segfault at 0 ip 0000000000414918 sp 00007fff93b30be8 error 4 in hpasmpld[400000+2a000]

System was fine before upgrade from 6.0 -> 6.3

# rpm -q hp-health
hp-health-9.1.0.42-54.rhel6.x86_64

Has anyone got any ideas how we can get this fixed ?

TIA

[busche]

Too sad, same issue here with CentOS 6.2 on a DL 180 G6 w/o iLO.

Initial install 3-4 months ago: all tools worked perfectly.

A plain normal OS update last week (and after hot-swapping a malfunctioning disc drive) resulted in the issue every one is reporting. This morning, I tested the following kernels:

2.6.32-279.1.1.el6.x86_64 #1 SMP Tue Jul 10 13:47:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
2.6.32-220.23.1.el6.x86_64 #1 SMP Mon Jun 18 18:58:52 BST 2012 x86_64 x86_64 x86_64 GNU/Linux

each with hp-health 8.7, 9.0 and 9.1, always with reboots - same failures.

I just raised an issue on their hotline... lets see what happens.

Some further remark; the following appears in my /var/log/messages (just tested this one again; thus the strange time information):

Jul 20 16:20:12 host kernel: hpasmpld:22300 map pfn expected mapping type uncached-minus for 9e000-a0000, got write-back
Jul 20 16:20:12 host hpasmpld[22300]: ehpsmb_parse_SMBIOS: SMBIOSInitTable was not successful.
Jul 20 16:20:12 host kernel: hpasmpld[22300]: segfault at 0 ip 0000000000414918 sp 00007fff53881ef8 error 4 in hpasmpld[400000+2a000]
Jul 20 16:20:12 host abrtd: Directory 'ccpp-2012-07-20-16:20:12-22300' creation detected
Jul 20 16:20:12 host abrt[22301]: Saved core dump of pid 22300 (/opt/hp/hp-health/bin/hpasmpld) to /var/spool/abrt/ccpp-2012-07-20-16:20:12-22300 (516096 bytes)
Jul 20 16:20:12 host abrtd: Package 'hp-health' isn't signed with proper key
Jul 20 16:20:12 host abrtd: 'post-create' on '/var/spool/abrt/ccpp-2012-07-20-16:20:12-22300' exited with 1
Jul 20 16:20:12 host abrtd: Corrupted or bad directory /var/spool/abrt/ccpp-2012-07-20-16:20:12-22300, deleting

The problems wrt. the package signing stuff is also reported (for some other package) in the CentOS forums.

However, I doubt that in my case this is related to additional/alternate repositories.

btw: the direct command causing the segfault is

/opt/hp/hp-health/bin/hpasmpld -f /dev/ipmi0

for me ...

Best,

  /b

[busche]

OK, first thing is that I accidentally upgraded my CentOS to version 6.3. I am quite sure that this is - somehow - the root cause of my troubles.

After some discussions with HP we came to the following thoughts:

* the IPMI on DL180G6 should be automatically disabled (since hp health uses the ILO which is present in all G6s.) by hp-health

Unluckily, the ipmi-modules are automatically loaded when /etc/init.d/hp-health start is called. rmmod for the IPMI modules beforehand does not work.

even the most recent CentOS 6.3 (from Mon, 23th July 2012) does not work with either hp-health 8.7 or 9.1.

As a concluding remark:

1) I will give a plain new installation of 6.2 a try - somewhen in 2 weeks

2) HP support is really movitated in trying to solve problems even for officially unsupported Operating Systems. Thank you! (And please try to continue the software support for at least one open source // freely available Linux distribution, as it had been the case since CentOS 6.1)

Best,

 /b

[xonxoff]

Logged a call with HP on this.

They had me run cfg2html and send th output to them, which I did.

After reviewing the output they got back to me to say that CentOS is not a supported OS. And sent this helpful information :

For support on Centos and other  non supported Linux distribution.

Please check the below web link for more information:

http://h18004.www1.hp.com/products/servers/linux/linuxcommunity/linuxfaqs.html

Brilliant, thanks HP!

[Fabrice Bacchella]

I'm just another poor victim of this bug.

Brand new DL180 G6, full firmware patch.

On a Scientific Linux 6.2 (another redhat clone) and all HP tools up to date, hp-health-9.1.0.42

[Naresh_Kumar]

Work Around: "Start the hp-health service prior to mcelogd service"

Steps:
Example-

1. List the service priorities using command

# ll /etc/rc.d/rc3.d/ |grep "mce\|hp-health"

lrwxrwxrwx. 1 root root 17 Aug 20 10:50 S50mcelogd -> ../init.d/mcelogd

lrwxrwxrwx. 1 root root 19 Aug 20 10:50 S91hp-health -> ../init.d/hp-health

As can be seen hp-health has lower priority (higher number S91) compared to mcelogd (numbered S50).

If hp-health to load befor mcelogd then hp-health service numbering should be lesser than mcelogd numbering.

This can be achieved by editing hp-health file.

2. Edit hp-health file as follows:

# vi /etc/init.d/hp-health

# (c) Copyright 2002, 2008 Hewlett-Packard Development Company, L.P.
#
# See "man chkconfig" for information on next two lines (Red Hat only)
# chkconfig: 2345 91 2
# description: hp System Health Monitor and Command line Utility Package.

3. Find the service number in line chkconfig (in this case 91) and change it to a number lesser than that of mcelogd (50).

4. For example, lets change it to by overwriting the existin number. chkconfig: 2345 41 2

5. Save the file and exit the editor.

6. Run th following commands to newly made changes to take affect

# chkconfig hp-health --list
hp-health 0:off 1:off 2:on 3:on 4:on 5:on 6:off

# chkconfig --del hp-health

# chkconfig hp-health --list
service hp-health supports chkconfig, but is not referenced in any runlevel (run 'chkconfig --add hp-health')

# chkconfig --add hp-health

# chkconfig hp-health --list
hp-health 0:off 1:off 2:on 3:on 4:on 5:on 6:off

7. Check for the service priorities changed by using following command

# ll /etc/rc.d/rc3.d/ |grep "mce\|hp-health"
lrwxrwxrwx. 1 root root 19 Aug 21 05:37 S41hp-health -> ../init.d/hp-health
lrwxrwxrwx. 1 root root 17 Aug 21 05:37 S50mcelogd -> ../init.d/mcelogd

[Gagan_Brahmi]

The following link should help for temporary work around.

http://gaganonthenet.com/2012/08/22/fix-for-hp-health-on-dl100-series-running-centos6/

[xonxoff]

Many thanks NK.

All working again now.

Tim