The new hpasmcli with hp-health 9.40 is very slow on DL380 G7 and DL380p Gen8

[Mark Ostroski]

We monitor our Linux systems using hpasmcli.  On DL380 G7 and DL380p Gen8 servers with the new hp-health 9.40 installed, hpasmcli commands now take several seconds to execute, compared with a near instantaneous response in the prior release.  A full health check now takes 20-30 seconds to execute.  The "show temp" command alone takes 10-15 seconds.  OS is RHEL5 64-bit.

Is this expected behavior?  Anyone else having the same problem?  The same test on DL585 G2 and G5 servers as well as DL380 G6 shows no slowdowns there.

[Matt_H]

1. Can you reply with the md5sum of the RPM you downloaded?
2. Can you or did you run the pre-req command below?

Prerequisites:

To get the list of all dependency files for hp-health, type:

      rpm –qp –requires hp-health-< version >.rpm

---

To ensure the integrity of your download, HP recommends verifying your results with this MD5 Checksum value:

9a0e5817c485be2177313867dbd65e5f hp-health-9.40-1602.36.rhel5.i386.rpm

Reboot Requirement:
Reboot is not required after installation for updates to take effect and hardware stability to be maintained.

---

Installation:

Login as the system administrator (root), download the RPM to a directory on your hard drive and change to that directory.

To install the package type:

      rpm -ivh hp-health-<version>.rpm

To check whether the package is loaded properly type:       /etc/init.d/hp-health status

Can you paste the output of the above status command?

Thanks

[Mark Ostroski]

Maybe I should clarify: hp-health-9.40-1602.36.rhel5.x86_64.rpm installs and works just fine.  There are no error messages.  It is just much slower than prior releases (9.30, 9.25 etc) on G7/Gen8.

Perhaps an example will help to explain:

   Before (hp-health 9.30):
      # time hpasmcli -s "show temp" >/dev/null

      real 0m0.002s
      user 0m0.001s
      sys 0m0.001s

   After (hp-health 9.40):
      # time hpasmcli -s "show temp" >/dev/null

      real 0m7.959s
      user 0m0.037s
      sys 0m0.017s

I obtained the RPMs from the new Service Pack for ProLiant ISO download (2013.09.0B).  The md5 sum matches the expected:

64b0543dfe19a9f78dfb3810489be560         hp-health-9.40-1602.36.rhel5.x86_64.rpm

The "rpm --requires" check isn't necessary because the RPM does in fact install successfully.

[Matt_H]

I haven't forgotten about your question, pardon for the annoyance, and I fully understood you and just wanted to see if there were and additional dependencies and the time delay you were seeing was due to a timeout. I will pass this along to the dev team.

Does the temperature provided by the "show temp" seem reasonable and accurate?

[Mark Ostroski]

Matt, thank you for reviewing this.

Yes, all data provided by hpasmcli seems to be correct, including the temperatures.  I have two identical test servers to use for comparison, one with hp-health 9.30 and one with hp-health 9.40.  The only significant difference that I see is the amount of time it takes to receive the response when running the command.  

The long delays are consistent on every G7 and Gen8 that I've tested so far... so I think it should be easy for HP to test.

[johsod]

Hi,

I just wanted to let you know that I'm experiencing the same problem. I have hp-health-9.40-1602.44 installed on a bunch of hp G7 and G8 servers. On my labservers, which are G5s, I do not see the slowdown.

System        : ProLiant DL360 G5
ROM version   : P58 08/03/2008
[root@kvm1 etc]# time hpasmcli -s "show temp" > /dev/null

real    0m0.023s
user    0m0.003s
sys    0m0.002s
[root@kvm1 etc]# rpm -qa |grep hp-health
hp-health-9.40-1602.44.rhel6.x86_64

System        : ProLiant DL360 G7     
ROM version   : P68 05/05/2011
[root@kvm01 ~]# time hpasmcli -s "show temp" > /dev/null

real    0m11.900s
user    0m0.003s
sys    0m0.003s
[root@kvm01 ~]# rpm -qa |grep hp-health
hp-health-9.40-1602.44.rhel6.x86_64

System        : ProLiant BL460c Gen8
ROM version   : I31 03/01/2013
[root@test01 ~]# time hpasmcli -s "show temp" > /dev/null

real    0m7.470s
user    0m0.000s
sys    0m0.003s
[root@test01 ~]# rpm -qa |grep hp-health
hp-health-9.40-1602.44.rhel6.x86_64

System        : ProLiant BL460c G7   
ROM version   : I27 12/03/2012
[root@stirling etc]# time hpasmcli -s "show temp" > /dev/null

real    0m3.550s
user    0m0.000s
sys    0m0.006s
[root@stirling etc]# rpm -qa |grep hp-health
hp-health-9.40-1602.44.rhel6.x86_64

Regards,

/Johan

[johsod]

I downgraded one of the servers to hp-health 9.31 and do not see the slowdown:

System        : ProLiant BL460c Gen8    
ROM version   : I31 03/01/2013
[root@test01 ~]# time hpasmcli -s "show temp" > /dev/null

real    0m0.002s
user    0m0.002s
sys    0m0.000s
[root@test01 ~]# rpm -qa |grep hp-health
hp-health-9.31-1572.3.rhel6.x86_64

[BlaBlaCar]

Hi,

same problem here:

# dpkg -l hp-health
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=======================-================-================-====================================================
ii hp-health 9.4.0.1.7-5. amd64 hp System Health Application and Command line Utilit

If I trace the program:

strace -r -ff -s 512 hpasmcli -s show\ iml

[...]

0.000135 socket(PF_FILE, SOCK_DGRAM, 0) = 3
0.000180 bind(3, {sa_family=AF_FILE, path=@"hphealthc-000c191b-cf89-00b2-0000-4754d46c300d"}, 110) = 0
0.000296 sched_get_priority_min(SCHED_OTHER) = 0
0.000098 sched_get_priority_max(SCHED_OTHER) = 0
0.000103 sendto(3, "\2\0\0\0\262\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 29, 0, {sa_family=AF_FILE, path=@"hphealths178"}, 110) = 29
0.000319 recvfrom(3, "\7\0\0\0\262\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 4125, 0, NULL, NULL) = 29
0.000134 sendto(3, "\1\0\0\0\262\0\0\0E\0\0\0\0\30\333K\377\177\0\0\0\0\0\0\26\0\0\0\17\0\0\0\212\23\0\0\2\\\360\27\333K\377\177\0\0\0\0\0\0\0", 51, 0,{sa_family=AF_FILE, path=@"hphealths178"}, 110) = 51
0.000327 recvfrom(3, "\7\0\0\0\262\0\0\0\0\0\0\0\360\27\333K\377\177\0\0\0\0\0\0\17\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 4125, 0, NULL, NULL) = 44
8.176230 stat("/dev/cpqhealth/cevt", {st_mode=S_IFCHR|0644, st_rdev=makedev(255, 176), ...}) = 0

[...]

Note the very long amount of time (>8s) needed for a single syscall like recvfrom().

However, I'm facing this behaviour only at startup. After a few minutes, the normal behaviour is back (execution <1s).

If I restart hp-health service, execution time gets very long time anew. And after a few minutes more, it's OK.

Hope this helps,

[Matti_Kurkela]

recvfrom() is a system call that receives information from a socket. The first argument to recvfrom() indicates the file descriptor number for the socket is 3. The earlier socket() and bind() system calls referring to the same descriptor reveal that it is an AF_FILE socket (AF_FILE is the same as AF_UNIX).

Since the recvfrom() does not have the MSG_DONTWAIT flag, the system call will wait until the process at the other end of the socket sends some data.

I would guess that the new hp-health service will delay some parts of its initialization until the first time it is actually used. That would explain the long time the first time hpasmcli starts communicating with it.

You might want to trace both the hpasmcli command and the hp-health service at the same time, and see what the service does when the hpasmcli starts communicating with it: find the corresponding socket-related system calls, and see what happens in the hp-health service after hpasmcli runs the sendto() calls.