topic SUM 8.2.0 Connection refused in ProLiant Servers (ML,DL,SL)

SUM 8.2.0 Connection refused

mschlossarek — Wed, 07 Mar 2018 12:06:34 GMT

Hello,

after upgrading to SUM 8.2.0 we get the following error when calling "smartupdate upgrade":

# smartupdate upgrade
sum : Smart Update Manager Version 8.2.0
Model: ProLiant DL380 Gen9, ROM: P89, ROM Version: 04/25/2017
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Searching for firmware....................................................................................................................................................................................
Analyzing results....................................................................................................................................................................................

Found multiple versions of 'Smart Array H240ar, H240nr, H240, H241, H244br, P240nr, P244br, P246br, P440ar, P440, P441, P542D, P741m, P840, P840ar, and  P841'
    [1] Smart Array H240ar, H240nr, H240, H241, H244br, P240nr, P244br, P246br, P440ar, P440, P441, P542D, P741m, P840, P840ar, and  P841 5.04
    [2] Smart Array H240ar, H240nr, H240, H241, H244br, P240nr, P244br, P246br, P440ar, P440, P441, P542D, P741m, P840, P840ar, and  P841 6.06
Which one do you want to apply? [1] : 2
Found multiple versions of 'HPE ProLiant DL380 Gen9/DL360 Gen9 (P89) Servers'
    [1] HPE ProLiant DL380 Gen9/DL360 Gen9 (P89) Servers 2.40_02-17-2017
    [2] HPE ProLiant DL380 Gen9/DL360 Gen9 (P89) Servers 2.52_10-25-2017
Which one do you want to apply? [1] : 2
Found multiple versions of 'HPE Broadcom NX1 Online Firmware Upgrade Utility for Linux x86_64'
    [1] HPE Broadcom NX1 Online Firmware Upgrade Utility for Linux x86_64 2.17.6
    [2] HPE Broadcom NX1 Online Firmware Upgrade Utility for Linux x86_64 2.19.6
Which one do you want to apply? [1] : 2
ERROR: [Errno 111] Connection refused
Traceback (most recent call last):
  File "sum_le.py", line 2518, in run
  File "sum_le.py", line 2501, in _run_command
  File "sum_le.py", line 876, in run
  File "/sum_le-8.2.0/hp/hpsum/hpsum6.py", line 588, in login
  File "/sum_le-8.2.0/hp/hpsum/hpsum6.py", line 540, in _rest_request
  File "/opt/python2.7/lib/python2.7/httplib.py", line 955, in request
  File "/opt/python2.7/lib/python2.7/httplib.py", line 989, in _send_request
  File "/opt/python2.7/lib/python2.7/httplib.py", line 951, in endheaders
  File "/opt/python2.7/lib/python2.7/httplib.py", line 811, in _send_output
  File "/opt/python2.7/lib/python2.7/httplib.py", line 773, in send
  File "/opt/python2.7/lib/python2.7/httplib.py", line 1154, in connect
  File "/opt/python2.7/lib/python2.7/socket.py", line 571, in create_connection
error: [Errno 111] Connection refused

This happens on DL380 Gen8 and Gen9 servers.

Re: SUM 8.2.0 Connection refused

troels_arvin — Sat, 24 Feb 2018 09:19:06 GMT

Same thing here :-(

Re: SUM 8.2.0 Connection refused

docana — Thu, 01 Mar 2018 15:47:46 GMT

In my case, I have two identical hosts but one has access to Internet and the other does not. The one with access to the Internet can run 'smartupdate upgrade' (8.2.0) perfectly fine , and the other shows the same error as yours, but in an earlier stage:

# smartupdate upgrade
sum : Smart Update Manager Version 8.2.0
Model: ProLiant BL460c Gen9, ROM: I36, ROM Version: 12/07/2017
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Searching for firmware..
Analyzing results..

ERROR: ''
Traceback (most recent call last):
File "sum_le.py", line 2518, in run
File "sum_le.py", line 2501, in _run_command
File "sum_le.py", line 876, in run
File "/sum_le-8.2.0/hp/hpsum/hpsum6.py", line 588, in login
File "/sum_le-8.2.0/hp/hpsum/hpsum6.py", line 542, in _rest_request
File "/opt/python2.7/lib/python2.7/httplib.py", line 1027, in getresponse
File "/opt/python2.7/lib/python2.7/httplib.py", line 407, in begin
File "/opt/python2.7/lib/python2.7/httplib.py", line 371, in _read_status
BadStatusLine: ''

Re: SUM 8.2.0 Connection refused

mschlossarek — Mon, 05 Mar 2018 10:59:58 GMT

After some debugging I found out that smartupgrade upgrade wants to start /opt/sum/bin/x64/sum_service_x64.

But this fails with the following error message (and weird debug output?):

# rpm -q sum
sum-8.2.0-53.rhel7.x84_64
# /opt/sum/bin/x64/sum_service_x64
/bin/sed
/bin/rpm
/bin/bash
/sbin/shutdown
/bin/gawk
/bin/grep
/bin/cat
/bin/nohup
/bin/dirname
/bin/cut
/bin/ls
/bin/uname
/bin/df
/bin/kill
/sbin/lsmod
/sbin/lspci
/sbin/dmidecode
/bin/awk
/bin/ps
/bin/unzip
QString::arg: Argument missing: Failed to create listener: %s
, failed to open listener

# netstat -anp | grep sum_service
#

With SUM 8.1.0 installed I'm able to start /opt/sum/bin/x64/sum_service_x64 (which opens two ports: 63001 and 63002):

# rpm -q sum
sum-8.1.0-14.rhel7.x86_64
# /opt/sum/bin/x64/sum_service_x64 80
/bin/sed
/bin/rpm
/bin/bash
/sbin/shutdown
/bin/gawk
/bin/grep
/bin/cat
/bin/nohup
/bin/dirname
/bin/cut
/bin/ls
/bin/uname
/bin/df
/bin/kill
/sbin/lsmod
/sbin/lspci
/sbin/dmidecode
/bin/awk
/bin/ps
/bin/unzip

# netstat -anp | grep sum_service
tcp6       0      0 :::63001                :::*                    LISTEN      28189/sum_service_x
tcp6       0      0 :::63002                :::*                    LISTEN      28189/sum_service_x
unix  2      [ ]         DGRAM                    21824581 28189/sum_service_x

Re: SUM 8.2.0 Connection refused

docana — Mon, 05 Mar 2018 11:24:36 GMT

Is it working for you then after downgrading to 8.1.0? If so, does you host have direct access to the Internet?

Re: SUM 8.2.0 Connection refused

mschlossarek — Mon, 05 Mar 2018 11:38:59 GMT

Yes, it works with SUM 8.1.0 and none of our servers have direct access to the internet. tcpdump does not show any attempt to connect to a external IP/URL.

Re: SUM 8.2.0 Connection refused

mschlossarek — Wed, 07 Mar 2018 12:08:32 GMT

Just tried it on our new Gen10 server with RHEL 6 and RHEl 7 - same error.

Re: SUM 8.2.0 Connection refused

docana — Wed, 07 Mar 2018 16:29:23 GMT

I installed sum-8.1.0-14.rhel7.x86_64 and still getting the exact same error. I can't spot any other difference than internet access between the hosts which can and the ones that can't run 'smartupdate upgrade'. But I did run tcpdump too and did not see any outbound connections.

Unlike you, I was always able to run sum_service_x64 without any problem, regardless the sum version. So my issue might be slightly different. I have tried running sum_service_x64 and can connect fine to 63001/2, so it's not firewall

I'll keep looking.

Re: SUM 8.2.0 Connection refused

rf2055 — Thu, 15 Mar 2018 18:58:22 GMT

on CentOS 6.9 installing sum-8.1.0-14.rhel6.x86_64, got me around the "ERROR: [Errno 111] Connection refused"

I also had to manually remove what appears to be duplicate or miss identifice firmware-smartarray , firmware-nic-intel rpms

Re: SUM 8.2.0 Connection refused

JHofmeister — Fri, 16 Mar 2018 19:42:30 GMT

This might be a hint:

# netstat -anp | grep sum_service
tcp6       0      0 :::63001                :::*                    LISTEN      28189/sum_service_x
tcp6       0      0 :::63002                :::*                    LISTEN      28189/sum_service_x
unix  2      [ ]         DGRAM                    21824581 28189/sum_service_x

tcp6 = ipv6 ; maybe test with ipv6 enabled on an intgerface with sum 8.2.0 installed?

Investigating this issue further at HPE...

Regards, jhofmeister

Re: SUM 8.2.0 Connection refused

JHofmeister — Fri, 16 Mar 2018 19:57:32 GMT

# sysctl net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1

# smartupdate
Error: Cannot launch sum_service_x64 locally. Reason: General failure. (failed to open listener)

Re: SUM 8.2.0 Connection refused

mschlossarek — Mon, 26 Mar 2018 07:26:30 GMT

You are right: After activating ipv6, "smartupdate upgrade" works again with SUM 8.2.0. But this is a problem for us because we have disabled ipv6 on every system. Enabling it only for SUM is not really an option.

Re: SUM 8.2.0 Connection refused

JHofmeister — Thu, 29 Mar 2018 16:17:22 GMT

- Implement workaround "enable IPv6" for SUM 8.2
~or~ install SUM 8.1 until fix/resolution.
- Monitor for availability of fix in Sum 8.3.0.

Re: SUM 8.2.0 Connection refused

lnux — Fri, 30 Mar 2018 12:46:47 GMT

This is what I got from HPE as well... you can enable ipv6 or downgrade to hpsum 8.1

Be careful if doing this for GEN10 systems as there are a few firmware udpates that require hpsum8.2

iegen10 and spsgen10 firmware for GEN10 requires hpsum8.2 for correct order of installation...

Re: SUM 8.2.0 Connection refused

gladiac — Tue, 12 Jun 2018 09:49:46 GMT

sum-8.3.0 does appear to fix this issue.

root@ # smartupdate upgrade
sum : Smart Update Manager Version 8.2.0
Model: ProLiant DL360 Gen9, ROM: P89, ROM Version: 01/22/2018
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Searching for firmware..............................................................................................................
Analyzing results..............................................................................................................

ERROR: [Errno 111] Connection refused
Traceback (most recent call last):
File "sum_le.py", line 2518, in run
File "sum_le.py", line 2501, in _run_command
File "sum_le.py", line 876, in run
File "/sum_le-8.2.0/hp/hpsum/hpsum6.py", line 588, in login
File "/sum_le-8.2.0/hp/hpsum/hpsum6.py", line 540, in _rest_request
File "/opt/python2.7/lib/python2.7/httplib.py", line 955, in request
File "/opt/python2.7/lib/python2.7/httplib.py", line 989, in _send_request
File "/opt/python2.7/lib/python2.7/httplib.py", line 951, in endheaders
File "/opt/python2.7/lib/python2.7/httplib.py", line 811, in _send_output
File "/opt/python2.7/lib/python2.7/httplib.py", line 773, in send
File "/opt/python2.7/lib/python2.7/httplib.py", line 1154, in connect
File "/opt/python2.7/lib/python2.7/socket.py", line 571, in create_connection
error: [Errno 111] Connection refused

root@ # yum update --nogpgcheck sum-8.3.0-44.rhel7.x86_64.rpm
Running transaction
Updating   : sum-8.3.0-44.rhel7.x86_64                                                                                                                                                                     1/2
Please read the License of the Agreement for this software at
         /opt/sum/sum.license

By not removing this package, you are accepting the terms
of the "HPE End User License Agreement - Enterprise Edition".

The SUM RPM has installed successfully.
Cleanup    : sum-8.2.0-53.rhel7.x86_64                                                                                                                                                                     2/2
Uploading Package Profile
Verifying : sum-8.3.0-44.rhel7.x86_64                                                                                                                                                                     1/2
Verifying : sum-8.2.0-53.rhel7.x86_64                                                                                                                                                                     2/2

Updated:
sum.x86_64 0:8.3.0-44.rhel7

Complete!

root@ # smartupdate upgrade
sum : Smart Update Manager Version 8.3.0
Model: ProLiant DL360 Gen9, ROM: P89, ROM Version: 01/22/2018
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Searching for firmware..............................................................................................................
Analyzing results..............................................................................................................

The following firmware will be applied:
N/A

The following firmware will be skipped (Use --downgrade or use --rewrite to apply.)
    Online ROM Flash Component for Linux - HPE Integrated Lights-Out 4   [2.55]
    Online ROM Flash Component for Linux - HPE ProLiant DL380 Gen9/DL360 Gen9 (P89) Servers   [P89 2.56 - (01/22/2018)]
    Online ROM Flash for Linux - Advanced Power Capping Microcontroller Firmware for HPE Gen9 Servers   [1.0.9]
    Supplemental Update / Online ROM Flash Component for Linux (x64) - Smart Array H240ar, H240nr, H240, H241, H244br, P240nr, P244br, P246br, P440ar, P440, P441, P542D, P741m, P840, P840ar, and P841   [6.30]
    HPE Broadcom NX1 Online Firmware Upgrade Utility for Linux x86_64   [2.20.7]