SUM 8.2.0 Connection refused

[mschlossarek]

Hello,

after upgrading to SUM 8.2.0 we get the following error when calling "smartupdate upgrade":

# smartupdate upgrade
sum : Smart Update Manager Version 8.2.0
Model: ProLiant DL380 Gen9, ROM: P89, ROM Version: 04/25/2017
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Searching for firmware....................................................................................................................................................................................
Analyzing results....................................................................................................................................................................................

Found multiple versions of 'Smart Array H240ar, H240nr, H240, H241, H244br, P240nr, P244br, P246br, P440ar, P440, P441, P542D, P741m, P840, P840ar, and  P841'
    [1] Smart Array H240ar, H240nr, H240, H241, H244br, P240nr, P244br, P246br, P440ar, P440, P441, P542D, P741m, P840, P840ar, and  P841 5.04
    [2] Smart Array H240ar, H240nr, H240, H241, H244br, P240nr, P244br, P246br, P440ar, P440, P441, P542D, P741m, P840, P840ar, and  P841 6.06
Which one do you want to apply? [1] : 2
Found multiple versions of 'HPE ProLiant DL380 Gen9/DL360 Gen9 (P89) Servers'
    [1] HPE ProLiant DL380 Gen9/DL360 Gen9 (P89) Servers 2.40_02-17-2017
    [2] HPE ProLiant DL380 Gen9/DL360 Gen9 (P89) Servers 2.52_10-25-2017
Which one do you want to apply? [1] : 2
Found multiple versions of 'HPE Broadcom NX1 Online Firmware Upgrade Utility for Linux x86_64'
    [1] HPE Broadcom NX1 Online Firmware Upgrade Utility for Linux x86_64 2.17.6
    [2] HPE Broadcom NX1 Online Firmware Upgrade Utility for Linux x86_64 2.19.6
Which one do you want to apply? [1] : 2
ERROR: [Errno 111] Connection refused
Traceback (most recent call last):
  File "sum_le.py", line 2518, in run
  File "sum_le.py", line 2501, in _run_command
  File "sum_le.py", line 876, in run
  File "/sum_le-8.2.0/hp/hpsum/hpsum6.py", line 588, in login
  File "/sum_le-8.2.0/hp/hpsum/hpsum6.py", line 540, in _rest_request
  File "/opt/python2.7/lib/python2.7/httplib.py", line 955, in request
  File "/opt/python2.7/lib/python2.7/httplib.py", line 989, in _send_request
  File "/opt/python2.7/lib/python2.7/httplib.py", line 951, in endheaders
  File "/opt/python2.7/lib/python2.7/httplib.py", line 811, in _send_output
  File "/opt/python2.7/lib/python2.7/httplib.py", line 773, in send
  File "/opt/python2.7/lib/python2.7/httplib.py", line 1154, in connect
  File "/opt/python2.7/lib/python2.7/socket.py", line 571, in create_connection
error: [Errno 111] Connection refused

This happens on DL380 Gen8 and Gen9 servers. 

[troels_arvin]

Same thing here :-(

[docana]

In my case, I have two identical hosts but one has access to Internet and the other does not. The one with access to the Internet can run 'smartupdate upgrade' (8.2.0) perfectly fine , and the other shows the same error as yours, but in an earlier stage:

# smartupdate list
sum : Smart Update Manager Version 8.2.0
Model: ProLiant BL460c Gen9, ROM: I36, ROM Version: 12/07/2017
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Searching for firmware..
Analyzing results..

 Info  | NAME                                | VERSION      | INSTALLED       | TYPE           |
------------------------------------------------------------------------------------------------
ifr-1  | smartarray-ea3138d8e8               | 6.06         | (details below) | smartarray     |
       | - AC_857342008                      |              |                 |                |
       | --> firmware                        | 6.06         | 6.06            |                |
icrd1  | system-i36                          | 2.56_01-22-2018 | 2017.12.07      | system         |

# smartupdate upgrade
sum : Smart Update Manager Version 8.2.0
Model: ProLiant BL460c Gen9, ROM: I36, ROM Version: 12/07/2017
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Searching for firmware..
Analyzing results..

ERROR: ''
Traceback (most recent call last):
  File "sum_le.py", line 2518, in run
  File "sum_le.py", line 2501, in _run_command
  File "sum_le.py", line 876, in run
  File "/sum_le-8.2.0/hp/hpsum/hpsum6.py", line 588, in login
  File "/sum_le-8.2.0/hp/hpsum/hpsum6.py", line 542, in _rest_request
  File "/opt/python2.7/lib/python2.7/httplib.py", line 1027, in getresponse
  File "/opt/python2.7/lib/python2.7/httplib.py", line 407, in begin
  File "/opt/python2.7/lib/python2.7/httplib.py", line 371, in _read_status
BadStatusLine: ''

[mschlossarek]

After some debugging I found out that smartupgrade upgrade wants to start /opt/sum/bin/x64/sum_service_x64.

But this fails with the following error message (and weird debug output?):

# rpm -q sum
sum-8.2.0-53.rhel7.x84_64
# /opt/sum/bin/x64/sum_service_x64
/bin/sed
/bin/rpm
/bin/bash
/sbin/shutdown
/bin/gawk
/bin/grep
/bin/cat
/bin/nohup
/bin/dirname
/bin/cut
/bin/ls
/bin/uname
/bin/df
/bin/kill
/sbin/lsmod
/sbin/lspci
/sbin/dmidecode
/bin/awk
/bin/ps
/bin/unzip
QString::arg: Argument missing: Failed to create listener: %s
, failed to open listener

# netstat -anp | grep sum_service
# 

With SUM 8.1.0 installed I'm able to start /opt/sum/bin/x64/sum_service_x64 (which opens two ports: 63001 and 63002):

# rpm -q sum
sum-8.1.0-14.rhel7.x86_64
# /opt/sum/bin/x64/sum_service_x64 80
/bin/sed
/bin/rpm
/bin/bash
/sbin/shutdown
/bin/gawk
/bin/grep
/bin/cat
/bin/nohup
/bin/dirname
/bin/cut
/bin/ls
/bin/uname
/bin/df
/bin/kill
/sbin/lsmod
/sbin/lspci
/sbin/dmidecode
/bin/awk
/bin/ps
/bin/unzip

# netstat -anp | grep sum_service
tcp6       0      0 :::63001                :::*                    LISTEN      28189/sum_service_x
tcp6       0      0 :::63002                :::*                    LISTEN      28189/sum_service_x
unix  2      [ ]         DGRAM                    21824581 28189/sum_service_x

[docana]

Is it working for you then after downgrading to 8.1.0? If so, does you host have direct access to the Internet?

[mschlossarek]

Yes, it works with SUM 8.1.0 and none of our servers have direct access to the internet. tcpdump does not show any attempt to connect to a external IP/URL.

[mschlossarek]

Just tried it on our new Gen10 server with RHEL 6 and RHEl 7 - same error.

[docana]

I installed sum-8.1.0-14.rhel7.x86_64 and still getting the exact same error. I can't spot any other difference than internet access between the hosts which can and the ones that can't run 'smartupdate upgrade'. But I did run tcpdump too and did not see any outbound connections.

Unlike you, I was always able to run sum_service_x64 without any problem, regardless the sum version. So my issue might be slightly different. I have tried running sum_service_x64 and can connect fine to 63001/2, so it's not firewall

I'll keep looking.

[rf2055]

on CentOS 6.9  installing  sum-8.1.0-14.rhel6.x86_64, got me around the  "ERROR: [Errno 111] Connection refused"

I also had to manually remove what appears to be duplicate or miss identifice firmware-smartarray , firmware-nic-intel rpms

[JHofmeister]

This might be a hint:

# netstat -anp | grep sum_service
tcp6       0      0 :::63001                :::*                    LISTEN      28189/sum_service_x
tcp6       0      0 :::63002                :::*                    LISTEN      28189/sum_service_x
unix  2      [ ]         DGRAM                    21824581 28189/sum_service_x

tcp6 = ipv6 ;    maybe test with ipv6 enabled on an intgerface with sum 8.2.0 installed?

Investigating this issue further at HPE... 

Regards, jhofmeister

[JHofmeister]

# sysctl net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1

# smartupdate
Error: Cannot launch sum_service_x64 locally. Reason: General failure. (failed to open listener)

[mschlossarek]

You are right: After activating ipv6, "smartupdate upgrade" works again with SUM 8.2.0. But this is a problem for us because we have disabled ipv6 on every system. Enabling it only for SUM is not really an option.

[JHofmeister]

- Implement workaround "enable IPv6" for SUM 8.2
~or~ install SUM 8.1 until fix/resolution.
- Monitor for availability of fix in Sum 8.3.0.

[lnux]

This is what I got from HPE as well... you can enable ipv6 or downgrade to hpsum 8.1

Be careful if doing this for GEN10 systems as there are a few firmware udpates that require hpsum8.2

iegen10 and spsgen10 firmware for GEN10 requires hpsum8.2 for correct order of installation...

[gladiac]

sum-8.3.0 does appear to fix this issue.

root@ # smartupdate upgrade
sum : Smart Update Manager Version 8.2.0
Model: ProLiant DL360 Gen9, ROM: P89, ROM Version: 01/22/2018
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Searching for firmware..............................................................................................................
Analyzing results..............................................................................................................

ERROR: [Errno 111] Connection refused
Traceback (most recent call last):
  File "sum_le.py", line 2518, in run
  File "sum_le.py", line 2501, in _run_command
  File "sum_le.py", line 876, in run
  File "/sum_le-8.2.0/hp/hpsum/hpsum6.py", line 588, in login
  File "/sum_le-8.2.0/hp/hpsum/hpsum6.py", line 540, in _rest_request
  File "/opt/python2.7/lib/python2.7/httplib.py", line 955, in request
  File "/opt/python2.7/lib/python2.7/httplib.py", line 989, in _send_request
  File "/opt/python2.7/lib/python2.7/httplib.py", line 951, in endheaders
  File "/opt/python2.7/lib/python2.7/httplib.py", line 811, in _send_output
  File "/opt/python2.7/lib/python2.7/httplib.py", line 773, in send
  File "/opt/python2.7/lib/python2.7/httplib.py", line 1154, in connect
  File "/opt/python2.7/lib/python2.7/socket.py", line 571, in create_connection
error: [Errno 111] Connection refused


root@ # yum update --nogpgcheck sum-8.3.0-44.rhel7.x86_64.rpm
Running transaction
  Updating   : sum-8.3.0-44.rhel7.x86_64                                                                                                                                                                     1/2
Please read the License of the Agreement for this software at
         /opt/sum/sum.license

By not removing this package, you are accepting the terms
of the "HPE End User License Agreement - Enterprise Edition".

The SUM RPM has installed successfully.
  Cleanup    : sum-8.2.0-53.rhel7.x86_64                                                                                                                                                                     2/2
Uploading Package Profile
  Verifying  : sum-8.3.0-44.rhel7.x86_64                                                                                                                                                                     1/2
  Verifying  : sum-8.2.0-53.rhel7.x86_64                                                                                                                                                                     2/2

Updated:
  sum.x86_64 0:8.3.0-44.rhel7

Complete!


root@ # smartupdate upgrade
sum : Smart Update Manager Version 8.3.0
Model: ProLiant DL360 Gen9, ROM: P89, ROM Version: 01/22/2018
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Searching for firmware..............................................................................................................
Analyzing results..............................................................................................................


The following firmware will be applied:
N/A


The following firmware will be skipped (Use --downgrade or use --rewrite to apply.)
    Online ROM Flash Component for Linux - HPE Integrated Lights-Out 4   [2.55]
    Online ROM Flash Component for Linux - HPE ProLiant DL380 Gen9/DL360 Gen9 (P89) Servers   [P89 2.56 - (01/22/2018)]
    Online ROM Flash for Linux - Advanced Power Capping Microcontroller Firmware for HPE Gen9 Servers   [1.0.9]
    Supplemental Update / Online ROM Flash Component for Linux (x64) - Smart Array H240ar, H240nr, H240, H241, H244br, P240nr, P244br, P246br, P440ar, P440, P441, P542D, P741m, P840, P840ar, and  P841   [6.30]
    HPE Broadcom NX1 Online Firmware Upgrade Utility for Linux x86_64   [2.20.7]