dHCI DL325 and Nimble HF60 upgrade from 67 to 7 major issues

[BobSCH]

Hi,

We have recently become the owners of a shiny new dHCI setup that came out of the box with ESXi 6.7 installed.  We've managed to get everything up and working in 67 fine.  The problems has started since upgrading (trying to upgrade) to v7u2 esxi.  We have some sync rep volumes and as I said all the 67 hosts are happy and booting nicely.  When we first attempted to upgrade one of the host to V7 we hit issues.  We followed the guidance and removed the SSACLI and NCM bits before starting the LifeCycle update to 7u2.  The host would post and get to vmw_satp_alua loaded successfully and then hang for ever.  Lifecycle manager then gave up and marked the upgrade as a timeout, we had to hard reset the host which then reverted to 6.7.  I had seen this behaviour before with some incorrect pathing to storage and also FW/Driver issues so I removed the initiator for the host from the nimble and tried the upgrade again.  This time all went smoothly and the host upgraded and patched A1.  I then added the HPE Storage manager 7.0.2 went through a painful process of updating the FW on the Broadcom 25gb NIC's.  When I connected the storage back to the host and tried a reboot the host then took 30+ to join back into the VC and even after that is still not responding very well.  If you Alt+F12 on the console there is lots of messages around failed to reserve volume and Reclaiming HB.  To me its as if the host is trying to connect to the replicated volumes and getting stuck in a loop.  

The supplier was going round in circles with HPE so I took over and now I'm going round in circles with them.  Has anybody seen this behaviour before?  The Nimbles are on 5.2.1, host is 7u2.  Thanks a lot.

[TGraves · Accepted solution]

Hello BobSCH

It sounds like you're not using the dHCI single-click update feature? Is there a reason you've opted to not use that functionality? 

I would suggest engaging support if you are having issues with single-click update. This feature will update all your components, and would have helped identify the issues introduced by VMware and the upgrade path restrictions they have imposed. For more info check out the upgrade matrix: 

https://infosight.hpe.com/org/83c24ce6-1a16-4ef0-90ec-42d798e4857c/resources/nimble/validated-configuration-matrix

Are all the hosts running official HPE ESXi images? This is quite important. With the 6.7U3 May 2021 HPE image, you cannot directly upgrade to 7U2 as there is a bootbank space issue. You must upgrade to 7U1 first, then you can upgrade to 7U2. 

Also contact support about SPP update integration which can ease the pain of manually upgrading your host firmware. 

Regards,

Tyler

[BobSCH]

Hi, I would love to use this feature but its not working for me at the moment.  It gives me an option to upgrade to V1.34 6.7U3, it then asks for the .ZIP which I have downloaded from link and had correct name.  It then states the is a file size problem.  Does the single-click update option allow upgrade from ESXi 6.7 to 7.0?  

[TGraves]

For reference, the HPE Images available are here: https://www.hpe.com/us/en/servers/hpe-esxi.html

6.7 U3 May 2021 -> 7u1 May 2021 is unfortunatly considered a back-in-time upgrade from VMware's perspective. If you are running 6.7U3 May 2021 or earlier you can engage support as we want support to be engaged due to the back-in-time upgrade path. 

What version are you running exactly on the 6.7 hosts now? 

[BobSCH]

Hi,  support has been engaged but so far we've not made much progress.  The out of the box hosts came with VMware ESXi, 6.7.0, 15160138.  We then tried an upgrade to VMware-ESXi-7.0.2-17867351-HPE-702.0.0.10.7.0.52-May2021 using the lifecycle manager.  This fails as the host reboots and then sticks at vmw_satp_alua indefinately until we reset the host.  VMware stated it could be the FW on the broadcom cards that were at FW 214 so we upgraded to v218.  I've even tried a clean install of 7.0.2 but as soon as I give it shared storage it hangs.  Cheers

[Vinodhdamodar]

is the issue resolved now, we are planning to upgrade the dHCI box which came with 6.7 version with factory image.

is the below process is recommended to upgrade.

1. Upgrade vCenter to 7 U3 (latest)
2. Upgrade the ESXi nodes using the HPE custom image for ESXi 7 U1
3. Upgrade the HPE storage connection manager (SCM) to 7.0.2-7.0.2-700014.

[TGraves]

There are Upgrade catalogs that include the 6.7 factory image. There are 7u1, 7u3 and 8.0 target ESXi versions. I suggest you confirm if the upgrade is visible in the vCenter plugin and use the provided Single-Click upgrade function to handle this for you and bring the system to a known recent catalog.