topic Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes) in StoreVirtual Storage

HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

jtslade — Wed, 01 Nov 2017 17:56:11 GMT

HP SAN 4730 Storevirtual trying to get it upgraded to 11.5 (later 12.6)

all nodes were on 10.5. Used an HPE provided ISO to get all the patches for each node updated to meet all of the 10.5. One node at a time.

My 4th node will not upgrade to HP SAN/iQ OS 11.5 no matter what.

The CMC will give out this error at the 25% mark:

Installation failed. An error was encountered with the configuration, RAID, or parts. (Exit Status 151).

===========================================

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

jtslade — Wed, 01 Nov 2017 18:55:22 GMT

Tail of the upgrade log, only place it notes any errors:

etc/lefthand/hwid.conf
etc/configs/.serialnumber
Update /mnt/upgraded/etc/passwd with entries in /etc/passwd
Update /mnt/upgraded/etc/shadow with entries in /etc/shadow
Update /mnt/upgraded/etc/group with entries in /etc/group
INFO: hostname SAN02 is NOT associated with 127.0.0.1 in /mnt/upgraded/etc/hosts, adding it now
Traceback (most recent call last):
File "/etc/lefthand/brand/common/bin/create-ldap-cfg-tar.py", line 20, in ?
from lhn import lockfile
EOFError: EOF read where object expected
INFO: created Active Directory config in /root/ldap-cfg.tar.bz2, SHA1 = []
ERROR: Failed to preserve Active Directory configuration.
/root/ldap-cfg.tar.bz2 is missing from current SAN/iQ version
umount: /mnt/upgraded/proc: not mounted
umount: /mnt/upgraded/dev: not mounted
umount: /mnt/upgraded: device is busy
umount: /mnt/upgraded: device is busy
#------------------------------------------------------------------------------#
MESG: Aborting due to an error encountered during installation of the software upgrade.
MESG: Contact Customer Support to resolve the issue.
MESG: Installation aborted.
INFO: Date: Wed Nov 1 16:54:29 GMT 2017
Reset filedescriptors
INFO: Copying log file to permanent location (/mnt/lhnupgrade/upgrade_from_10.5.00.0149_to_11.5.00.0673_20171101.log -> /var/log/upgrade_from_10.5.00.0149_to_11.5.00.0673_20171101.log) ...

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

jtslade — Wed, 01 Nov 2017 18:57:23 GMT

Top of upgrade log:

INFO: ServerEngines Corp. Emulex OneConnect 10Gb NIC installed
     10Gb NIC firmware version = [5719-v1.46]
WARNING: Cannot determine Emulex 10Gb firmware fields:
major=[5719-v1] minor=[46] patch=[] build=[]
/usr/sbin/hpacucli
response:ok:
response:ok:
INFO: f_checkMinimumFirmwareSmartArray: SmartArray P410 controller not detected in this platform, nothing to do.
response:ok:started
response:ok:started
INFO: Starting software upgrade (version 11.5.00.0673) ...
INFO: System: SAN02

INFO: Date: Wed Nov 1 16:53:07 GMT 2017
INFO: Action: installing software upgrade, version 11.5.00.0673
INFO: ##### Upgrade Mode: [reboot_required] #####
INFO: New Version Information:
INFO:     New SANiQ Version: 11.5.00.0673
INFO:     root OS directory tree will be extracted directly from gzipped tarball
INFO:     New Data Directory=/mnt/lhnupgrade/data/default
INFO: Current Version Information:
INFO:     Current SANiQ Version: 10.5.00.0149
INFO:     Current Kernel: Linux 2.6.18-274.el5 #1 SMP Fri Jul 8 17:36:59 EDT 2011
INFO:     Current OS: GNU/Linux
INFO:     Current Machine: x86_64
INFO:     Current Processor: x86_64
INFO:     Current Data Directory=/mnt/lhnupgrade/data/default
INFO: Supported Releases:
INFO:     Oldest Supported SANiQ Version: 9.5.00
INFO: Build Details:
INFO:     Target: 64
INFO:     Build Timestamp: Sun, Jul 27 2014 19:01:25 MDT
INFO: Platform Details:
INFO:     Platform: HPDLG8B
INFO:     Current Brand Directory: /etc/lefthand/brand/HPDLG8B/
INFO:     New Brand Directory:     /etc/lefthand/brand/HPDLG8B/
INFO:     Current HWID_CONFIG:     HPDLG8B
INFO:     New HWID_CONFIG:         HPDLG8B
INFO:     Model Number:            36
INFO: The current root OS, /, is mounted on /dev/cciss/c0d0p2
INFO:     Partition that / is currently mounted to: root_os_device = [/dev/cciss/c0d0p2]
INFO:     Size of root/OS partition (in bytes):                 [5121245184]
INFO:     Size of disk containing root/OS partition (in bytes): [34355036160]
#---------------------------------------------------------------#
INFO: Currently installed patches:
total 30
-rw-r--r-- 1 root root 594 Sep 12 2013 10054-07.xml
-rw-r--r-- 1 root root 677 Oct 25 17:58 10139-00.xml
-rw-r--r-- 1 root root 1059 Sep 12 2013 10142-00.xml
-rw-r--r-- 1 root root 705 Oct 31 18:37 10147-00.xml
-rw-r--r-- 1 root root 689 Oct 24 18:28 10152-00.xml
-rw-r--r-- 1 root root 781 Aug 10 05:03 10161-00.xml
-rw-r--r-- 1 root root 1100 Oct 25 15:44 10172-00.xml
-rw-r--r-- 1 root root 1048 Oct 28 00:25 10186-00.xml
-rw-r--r-- 1 root root 1376 Oct 25 19:39 35010-00.xml
-rw-r--r-- 1 root root 1178 Oct 25 19:54 35011-00.xml
-rw-r--r-- 1 root root 850 Oct 25 19:59 35012-00.xml
-rw-r--r-- 1 root root 1057 Oct 25 20:04 35014-00.xml
-rw-r--r-- 1 root root 961 Oct 25 20:10 35019-00.xml
-rw-r--r-- 1 root root 1181 Oct 25 20:17 35020-00.xml
-rw-r--r-- 1 root root 905 Oct 25 20:27 35022-00.xml
-rw-r--r-- 1 root root 915 Oct 25 20:36 35024-00.xml
-rw-r--r-- 1 root root 1154 Aug 10 06:06 35025-00.xml
-rw-r--r-- 1 root root 1461 Oct 25 20:46 35027-00.xml
-rw-r--r-- 1 root root 1023 Aug 10 06:29 35028-00.xml
-rw-r--r-- 1 root root 1373 Oct 25 21:19 35030-00.xml
#------------------------------------------------------------------------------#
INFO: Filesystems mounted right now:
rootfs / rootfs rw 0 0
/dev/root / ext3 rw,data=ordered 0 0
/dev /dev tmpfs rw 0 0
/proc /proc proc rw 0 0
/sys /sys sysfs rw 0 0
none /selinux selinuxfs rw 0 0
/proc/bus/usb /proc/bus/usb usbfs rw 0 0
devpts /dev/pts devpts rw 0 0
tmpfs /dev/shm tmpfs rw 0 0
tmpfs /tmp tmpfs rw 0 0
/dev/cciss/c0d0p5 /var/log ext3 rw,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
/dev/cciss/c0d0p7 /etc/lefthand/data ext3 rw,noatime,nodiratime,data=ordered 0 0
/dev/cciss/c0d0p6 /mnt/lhnupgrade ext3 rw,data=ordered 0 0
#------------------------------------------------------------------------------#
MESG: Starting software upgrade ...
MESG: Current Software Version: 10.5.00.0149
MESG: New Software Version: 11.5.00.0673
#------------------------------------------------------------------------------#
MESG: Checking storage node ...
INFO: The current root OS, /, is mounted on /dev/cciss/c0d0p2
Checking package contents ...
INFO: The flash image was not checked, using root/OS directory extraction.
#------------------------------------------------------------------------------#
Checking storage node configuration ...
INFO: Checking version information ...
INFO: Checking current version is a supported release ...
#------------------------------------------------------------------------------#
INFO: Memory check required for current platform and SAN/iQ version
INFO: Memory(in KB) on current NSM: 65837508.
#------------------------------------------------------------------------------#
INFO: Checking to see if the OS archiver needs to be halted ...
INFO: OS archiver successfully halted.
#------------------------------------------------------------------------------#
INFO: Looking for hardware reports ...
#---------------------------------------------------------------#
INFO: hydra get:/lhn/public/system/info/report/raidSetup/

HAL Diagnostics (version 4.0)

Platform: 4730 Build: 10.5.00.0149.0 ID: D8:9D:67:29:46:94 F/W: 3.30

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

jtslade — Wed, 01 Nov 2017 19:45:39 GMT

Installation process started on: 11/1/17 1:35 PM

Performance Instrumentation -- Upgrade: --MG Configuration, ID: GETSAN --

Performance Instrumentation -- Upgrade: --NSM Configuration--

Performance Instrumentation -- Upgrade: Number of NSMs: 1

Performance Instrumentation -- Upgrade: NSM Model Name: 4730

Performance Instrumentation -- Upgrade: NSM Software Version: 10.5.00.0149.0

Performance Instrumentation -- Upgrade: NSM Host Name: SAN02

Performance Instrumentation -- Upgrade: NSM IP Address:

Performance Instrumentation -- Upgrade: NSM Serial Number: D8:9D:67:29:46:94

Performance Instrumentation -- Upgrade: --End Of NSM Configuration--

Performance Instrumentation -- Upgrade: Regular Manager Count: 3

Performance Instrumentation -- Upgrade: Manager's Status: 4 managers configured: 3 regular managers, 1 Failover Manager

Performance Instrumentation -- Upgrade: systemIdDataModel Version: 8

Performance Instrumentation -- Upgrade: Has Failover Manager: true

Performance Instrumentation -- Upgrade: Has Virtual Manager: false

Performance Instrumentation -- Upgrade: Global Database Version: 168099840

Performance Instrumentation -- Upgrade: Number Admin Groups: 2

Performance Instrumentation -- Upgrade: Number Admin Users: 7

Performance Instrumentation -- Upgrade: Number Admin Users: 7

Performance Instrumentation -- Upgrade: Global Internal Version: 20121120

Performance Instrumentation -- Upgrade: Number of Clusters: 1

Performance Instrumentation -- Upgrade: Number of Managers: 4

Performance Instrumentation -- Upgrade: Number of Schedules: 0

Performance Instrumentation -- Upgrade: Number of Scratch Volumes: 0

Performance Instrumentation -- Upgrade: Number of Sites: 2

Performance Instrumentation -- Upgrade: Number of Stores: 4

Performance Instrumentation -- Upgrade: --End Of MG Configuration--

/192.168.57.202:11120: connectToServer received a Status message from the install server running on the node It was: 'OK', and the error flag was: false

/192.168.57.202:11120: sendPackageToServer received a Status message from the install server running on the node It was: 'OK', and the error flag was: false

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : Extracting the install file ...

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : Extracting package contents ...

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : MESG: Configuring platform specific settings ...

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : Configuring platform specific settings ...

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : HWID: 36

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : HWID Model: 4730

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : HWID Config: HPDLG8B

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : Brand Directory: /etc/lefthand/brand/HPDLG8B/

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : platform configuration: '/mnt/lhnupgrade/platforms/HPDLG8B.sh' ...

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : Starting software upgrade ...

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : Current Software Version: 10.5.00.0149

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : New Software Version: 11.5.00.0673

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : Checking storage node ...

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : begin MAIN install_saniq.sh: Preparing for upgrade ...

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : Installing new software ...

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : Aborting due to an error encountered during installation of the software upgrade.

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : Contact Customer Support to resolve the issue.

/192.168.57.202:11120: Received an UpgradeScriptStatus message from the install server running on the node It was : Installation aborted.

/192.168.57.202:11120: Received an UpgradeServerStatus message from the install server running on the node It was : INSTALL_STATUS_MESSAGE: done [151]

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

Assen — Thu, 02 Nov 2017 07:54:43 GMT

Hi jtslade,

could you please share the hardware info (CMC>Diagnostic>Hardware Information) from the 4th node, that does not want to update?

What is your localization setting of the CMC? (CMC>Preferences>Locale)

Assen

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

jtslade — Thu, 02 Nov 2017 12:49:25 GMT

Assen Thank you for checking this out.

localization setting of the CMC? (CMC>Preferences>Locale):

Current System Locale: English (United States)
Current CMC Locale: [Use System Setting]

HARDWARE INFORMATION

*Hardware Information*



*Hostname*

   SAN02

*Model*

   4730

*Storage System Software*

   Version 10.5.00.0149.0
    Software Patches    10054-07, 10139-00, 10142-00, 10147-00, 10152-00,
10161-00, 10172-00, 10186-00, 35010-00, 35011-00, 35012-00, 35014-00,
35019-00, 35020-00, 35022-00, 35024-00, 35025-00, 35027-00, 35028-00,
35030-00

*IP Address*



*Support Key*



**



*Warranty Information*


Product Part Number    B7E28A
Product Serial Number
Entitlement Order Number

**



*NIC Data*


Card    1
    Description    Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
PCIe
    Mac Address
    Address    0.0.0.0
    Mask    0.0.0.0
    Gateway    0.0.0.0
    Mode    disabled
    Firmware Version    5719-v1.46 NCSI v1.3.16.0
    Driver Name    tg3
    Driver Version    3.122g
Card    2
    Description    Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
PCIe
    Mac Address
    Address    0.0.0.0
    Mask    0.0.0.0
    Gateway    0.0.0.0
    Mode    disabled
    Firmware Version    5719-v1.46 NCSI v1.3.16.0
    Driver Name    tg3
    Driver Version    3.122g
Card    3
    Description    Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
PCIe
    Mac Address
    Address    0.0.0.0
    Mask    0.0.0.0
    Gateway    0.0.0.0
    Mode    disabled
    Firmware Version    5719-v1.46 NCSI v1.3.16.0
    Driver Name    tg3
    Driver Version    3.122g
Card    4
    Description    Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
PCIe
    Mac Address
    Address    0.0.0.0
    Mask    0.0.0.0
    Gateway    0.0.0.0
    Mode    disabled
    Firmware Version    5719-v1.46 NCSI v1.3.16.0
    Driver Name    tg3
    Driver Version    3.122g
Card    5
    Description    Logical Failover Device ( balance-alb )
    Mac Address
    Address
    Mask    255.255.255.0
    Gateway    0.0.0.0
    Mode    static
    Firmware Version    2
    Driver Name    bonding
    Driver Version    3.4.0-1
Card    6
    Description    Emulex Corporation OneConnect 10Gb NIC (be3)
    Mac Address
    Address
    Mask    255.255.255.0
    Gateway    0.0.0.0
    Mode    slave
    Firmware Version    10.2.477.10
    Driver Name    be2net
    Driver Version    10.2.477.6
Card    7
    Description    Emulex Corporation OneConnect 10Gb NIC (be3)
    Mac Address
    Address
    Mask    255.255.255.0
    Gateway    0.0.0.0
    Mode    slave
    Firmware Version    10.2.477.10
    Driver Name    be2net
    Driver Version    10.2.477.6

**



*Memory*


Total    62.79 GB
Free    56.05 GB

**



*CPU*


Number 1
Model Name    Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
Speed    1995.258
Cache Size    15360 KB

**



*Stat*


CPU (seconds)    user (871) sys (468) idle (34158) I/O (1383) IRQ (209)
Machine Uptime (hours)    up-time (0.86) idle (0.84)

**



*Backplane Information*


Backplane Status    Normal
Firmware Version    3.30
Serial Number
LED Support
    idLed    off

**



*Motherboard Information*


Chassis Part Number    B7E28A
Chassis Serial Number
BIOS Version    P70, 07/01/2015
IPMI Information
    Firmware Revision    2.50
    IPMI Version    2.0

**



*Drive Info*



*Model*



*Serial#*



*Capacity*



*F/W Version*

Drive 1
HP EG0900FBVFQ   KPJZDBAF   838.28 GB   HPDE

Drive 2
HP EG0900FBVFQ   KPJZZNDF   838.28 GB   HPDE

Drive 3
HP EG0900FCSPN   83H0A0AMFTM11333   838.28 GB   HPD0

Drive 4
HP EG0900FBVFQ   KPJZYDKF   838.28 GB   HPDE

Drive 5
HP EG0900FBVFQ   W8G8JS5X   838.28 GB   HPDE

Drive 6
HP EG0900FBVFQ   KPJZZUJF   838.28 GB   HPDE

Drive 7
HP EG0900FBVFQ   KPJY6AHF   838.28 GB   HPDE

Drive 8
HP EG0900FBVFQ   KPJZY9XF   838.28 GB   HPDE

Drive 9
HP EG0900FBVFQ   KPJXHSYF   838.28 GB   HPDE

Drive 10
HP EG0900FBVFQ   KPK0604F   838.28 GB   HPDE

Drive 11
HP EG0900FBVFQ   KPJZZRDF   838.28 GB   HPDE

Drive 12
HP EG0900FBVFQ   KPK06H6F   838.28 GB   HPDE

Drive 13
HP EG0900FBVFQ   KPK065JF   838.28 GB   HPDE

Drive 14
HP EG0900FBVFQ   KPJZZNZF   838.28 GB   HPDE

Drive 15
HP EG0900FBVFQ   KPJZZHSF   838.28 GB   HPDE

Drive 16
HP EG0900FBVFQ   KPJZYG3F   838.28 GB   HPDE

Drive 17
HP EG0900FBVFQ   KPK066NF   838.28 GB   HPDE

Drive 18
HP EG0900FBVFQ   KPJZYMYF   838.28 GB   HPDE

Drive 19
HP EG0900FBVFQ   KPJZYE6F   838.28 GB   HPDE

Drive 20
HP EG0900FBVFQ   KPJZYEMF   838.28 GB   HPDE

Drive 21
HP EG0900FBVFQ   KPJZY7XF   838.28 GB   HPDE

Drive 22
HP EG0900FBVFQ   KPJZYE9F   838.28 GB   HPDE

Drive 23
HP EG0900FBVFQ   KPJZYJ3F   838.28 GB   HPDE

Drive 24
HP EG0900FBVFQ   KPJZYADF   838.28 GB   HPDE

Drive 25
HP EG0900FBVFQ   W8H3LZLG   838.28 GB   HPDE

**



*Drive Status*



*Status*



*Health*



*Temperature*

Drive 1
Active   Normal   44

Drive 2
Active   Normal   45

Drive 3
Active   Normal   47

Drive 4
Active   Normal   44

Drive 5
Active   Normal   44

Drive 6
Active   Normal   43

Drive 7
Active   Normal   42

Drive 8
Active   Normal   40

Drive 9
Active   Normal   40

Drive 10
Active   Normal   40

Drive 11
Active   Normal   39

Drive 12
Active   Normal   39

Drive 13
Active   Normal   38

Drive 14
Active   Normal   38

Drive 15
Active   Normal   39

Drive 16
Active   Normal   38

Drive 17
Active   Normal   38

Drive 18
Active   Normal   37

Drive 19
Active   Normal   38

Drive 20
Active   Normal   39

Drive 21
Active   Normal   37

Drive 22
Active   Normal   37

Drive 23
Active   Normal   37

Drive 24
Active   Normal   37

Drive 25
Hot spare   Normal   37

**



*RAID*

   Normal
Rebuild Rate    Medium
Unused Devices    Disk Drive 25 Hot spare
Statistics    2 Arrays
Array 1    /dev/cciss/c0d1p2 : DATA Partition Raid 6 8351.32 GB Normal
Array 2    /dev/cciss/c0d2p2 : DATA Partition Raid 6 8383.31 GB Normal

**



*RAID OS Partitions*

   Normal
Statistics    2 Arrays
Array 1    /dev/cciss/c0d0p5 : LOG Partition Raid 6 977.00 MB Normal
Array 2    /dev/cciss/c0d0p7 : SANIQ Partition Raid 6 2930.00 MB Normal

**



*Controller/Cache Items*


Card 1
    Model    Smart Array P420i
    Serial Number
    Hardware Version    B
    Firmware Version    8.00
    Driver Version    3.6.28-RH2
    Cache Status    Normal
    Backup Power Status    Normal
    Backup Power Type    Capacitors
    Cache Enabled Status    Normal
    Cache Ratio    50% Read / 50% Write
    Cache Size    2048MB
    Backup Power Voltage    Info not available for this hardware platform

**



*Power Supply*

   Dual

*Power Supplies*


Number 1    Normal
Number 2    Normal

**



*Fans*



*Status*



*RPM*



*Min*



**

Fan3
Normal   ---   ---

Fan4
Normal   ---   ---

Fan5
Normal   ---   ---

Fan6
Normal   ---   ---

**



*Temperatures*



*Status*



*Celsius*



*Warning*



*Critical*

CPU (1)
Normal   0   66   68

Ambient
Normal   29   46   48

System Board
Normal   47   56   58

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

Assen — Thu, 02 Nov 2017 14:45:33 GMT

yaw Jtslade

All looks good from firmware versions, as it should be. Localization also ok.

Do you have some kind of virus scanner running on the machine where the CMC is? I know it's strange that all other nodes

updated without issues, but could you disable virus scan and try again.

As of description and log it behaves like the image is corrupt or got corrupted during transfer to node. Have seen virus scan corrupt install packages quite often.

Assen

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

jtslade — Thu, 02 Nov 2017 15:21:29 GMT

Yes the 3 other nodes upgraded via support mode one at a time.

I did however make a new Win2k12 VM with no AV (or anything installed) other than the 12.6 CMC

This was installed on a completely seperate esxi server on local storage not on the SAN cluster that was being upgraded.

and I get the exact same error message:

in upgrade logs: MESG: Aborting due to an error encountered during installation of the software upgrade.
MESG: Contact Customer Support to resolve the issue.
MESG: Installation aborted.

From the CMC: Installation failed. An error was encountered with the configuration, RAID, or parts. (Exit Status 151)

Where can we look up the actual error in the logs instead of this generic one.

What is exit status 151?

Thank you,

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

peyrache — Thu, 02 Nov 2017 15:49:14 GMT

Extract support bundle from CMC and look on update logs
(patchXXX.logs)
JY

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

jtslade — Thu, 02 Nov 2017 17:01:59 GMT

Ok so when comparing this log from the fail 2 upgrade to 11.5 node with one of the others that did upgrade. This is the only difference.

INFO: hostname SAN02 is NOT associated with 127.0.0.1 in /mnt/upgraded/etc/hosts, adding it now
Traceback (most recent call last):
File "/etc/lefthand/brand/common/bin/create-ldap-cfg-tar.py", line 20, in ?
from lhn import lockfile
EOFError: EOF read where object expected
INFO: created Active Directory config in /root/ldap-cfg.tar.bz2, SHA1 = []
ERROR: Failed to preserve Active Directory configuration.
/root/ldap-cfg.tar.bz2 is missing from current SAN/iQ version
umount: /mnt/upgraded/proc: not mounted
umount: /mnt/upgraded/dev: not mounted
umount: /mnt/upgraded: device is busy
umount: /mnt/upgraded: device is busy
#------------------------------------------------------------------------------#
MESG: Aborting due to an error encountered during installation of the software upgrade.
MESG: Contact Customer Support to resolve the issue.
MESG: Installation aborted.
INFO: Date: Wed Nov 1 16:54:29 GMT 2017
Reset filedescriptors
INFO: Copying log file to permanent location (/mnt/lhnupgrade/upgrade_from_10.5.00.0149_to_11.5.00.0673_20171101.log -> /var/log/upgrade_from_10.5.00.0149_to_11.5.00.0673_20171101.log) ..

Compare that to a successful SAN node upgade to 11.5 log file:

INFO: hostname SANF02 is NOT associated with 127.0.0.1 in /mnt/upgraded/etc/hosts, adding it now
INFO: created Active Directory config in /root/ldap-cfg.tar.bz2, SHA1 = [4ddf0cbcf382815b241b07290a885d3db78f2e2c]
etc/ldap.conf
etc/openldap/ldap.conf
root/.ldaprc
root/.ldaprc-pw
etc/lefthand/bin/.ssh/authorized_keys
etc/lefthand/nsm.cert
etc/lefthand/nsm.key
INFO: Active Directory config has been installed into new SAN/iQ version, files preserved are:
[etc/ldap.conf
etc/openldap/ldap.conf
root/.ldaprc
root/.ldaprc-pw
etc/lefthand/bin/.ssh/authorized_keys
etc/lefthand/nsm.cert
etc/lefthand/nsm.key]
#------------------------------------------------------------------------------#
#------------------------------------------------------------------------------#
INFO: Verifying restored configuration ...
INFO:     skipping directory/link "/etc/configs/monitor/"
INFO:     verifying "/etc/configs/monitor/Monitor.cfg"
INFO:     verifying "/etc/configs/monitor/PlatformMonVars.cfg"
INFO:     verifying "/etc/configs/monitor/PresetTriggers.cfg"
INFO:     verifying "/etc/configs/monitor/SensorProfile.cfg"
INFO:     verifying "/etc/configs/monitor/VariableDefinitions.cfg"
INFO:     verifying "/etc/group"
1c1,2
< root:x:0:root

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

peyrache — Thu, 02 Nov 2017 17:18:14 GMT

Shall try manual update ? may be
JY

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

jtslade — Thu, 02 Nov 2017 17:20:03 GMT

as in using Support Mode in the CMC (how these were performed) or is there another way like from the command line interface. Can I skip active directory LDAP creation and verificaiton maybe in manual mode?

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

peyrache — Thu, 02 Nov 2017 17:28:14 GMT

Manual mode just let you arrange to select package to be installed

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

peyrache — Thu, 02 Nov 2017 17:33:14 GMT

May be related ?
Active Directory external authentication
All P4000 and StoreVirtual systems supported by LeftHand OS 10.0 and later which use Active Directory external
authentication
Issue
The CLIQ setadinfo command accepts several keyword-value pairs which are passed through LeftHand OS to the Microsoft Active
Directory server.
The baseDN and bindusername keywords accept text string values which can contain shell meta characters.
These values must be quoted correctly if they are to be successfully processed.
Cause
Active Directory wants to receive quoted strings for the baseDN and bindusername keywords. LeftHand OS will remove the quotes if
they are not double quoted.
For example,
baseDN="DC=john,DC=contoso,DC=com" bindusername="admin@john.contoso.com"
will fail when used with the CLIQ setadinfo command. This will work if additional single quote characters enclose the value.
baseDN='"DC=john,DC=contoso,DC=com"' bindusername='"admin@john.contoso.com"'
Resolution
Use double-quoted string values for the baseDN and bindusername keywords. Here is an example of the entire setadinfo command.
cliq.exe setadinfo login=10.10.10.100 username=admin password=pArx33g
baseDN='"DC=john,DC=contoso,DC=com"' adserver=ldap://10.10.10.137
bindusername='"admin@john.contoso.com"' bindpassword=eocws92uPW testonly=false
Another variation which can be used is the triple quotation, for example,
bindusername="""admin@john.contoso.com"""

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

jtslade — Thu, 02 Nov 2017 18:08:29 GMT

Yes Support mode (the Yellowist CMC mode ever)

is what has been used to upgrade 3 of the 4 nodes.

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

Assen — Fri, 03 Nov 2017 07:58:44 GMT

Hi jtslade,

could it be that the upgrade partition is still mounted?

you could check it from the cliq. Putty in to node (port 16022), log in with CMC credentials.

You can find more info in the HP StoreVirtual LeftHand OS Command Line
Interface User Guide

run the command

utility run="ps -ax"

look for process containing lefthand/upgrade, upgrade or similar

if you find something like the above, reboot the node and start the upgrade again. After reboot the upgrade partition should be unmounted.

Assen

Re: HP SAN 4730 Storevirtual Node error message when trying to update (last of 4 nodes)

jtslade — Fri, 03 Nov 2017 14:34:26 GMT

This is the error here:

File "/etc/lefthand/brand/common/bin/create-ldap-cfg-tar.py", line 20, in ?
from lhn import lockfile
EOFError: EOF read where object expected
INFO: created Active Directory config in /root/ldap-cfg.tar.bz2, SHA1 = []
ERROR: Failed to preserve Active Directory configuration.
/root/ldap-cfg.tar.bz2 is missing from current SAN/iQ version

Either the Python script is corrupt (hence no hash on the tar AD).

Its fixable, but only by those that can access the HP SANiQ OS files directly.

My last thought on this would be to downgrade the node to 10.0 which as I understand is what had the original AD/LDAP intergration added. then upgrade back.

Its a lost cause. Thank you for everyone's help! Have a great weekend.