Problem installing HPE StoreVirtual Multipathing Extension Module for Vmware vSphere

[erwinbre]

Hi all,

we have a VMWare ESXi-cluster of 6 hosts (all 6 hosts are identical Proliant DL 385 G7 servers and version of ESXi is 5.5.0, 2718055)

Our HP-Lefthand/StoreVirtual SAN is running version 12.6 of the StoreVirtual OS.

I downloaded the latest driver (.vib file) from HP: StoreVirtual Multipathing Extension Module for Vmware vSphere 5.5 (AT004-10536.vib)

Installation went fine on 5 ESXi-hosts. After reboot of the hosts I can see that these 5 hosts are using the HPE_SATP_LH storage array type and the HPE_PSP_LH path selection, which indicates a succesfull installation of the multipathing driver.

The problem is one 1 host. Installation of the driver goes fine, the CLI-window confirms installation is succesfull and that the host requires a reboot.

During this first reboot after installation the host goes into a purple screen (see attachement)

After this we have to do a hard reset of the host after which the host boots successfully, but with the default VMWare driver active (VMW_SATP_DEFAULT_AA / Round Robin) and not the Multipathing driver.

Can anybody help me please?

Thank you very much,

Erwin B.

[Stor_Mort]

It does sound like an installation issue on that host, as the others seem to be working. Try un-installing and re-installing. Check if there are any differences in the versions of the NIC drivers between this host and the others.

[Kuiper]

Hi,

We have the same issue on all of our hosts (DL360G7 and DL360G5), I've tried first two G7's in different clusters and after that a G5, but all the the systems gave an PSOD. After a cold restart the HPE-LH-MEM driver seems to be deinstalled. TIll this moment we didn't have an issue and all the ESX servers has the latest VMware 5.5 updates and VUM is updated with the HPE VIBS, but the driver should give us less latency and more IOPS we need. We have two P4500G2 nodes (cluster 1) and four P4530 (cluster 2) nodes.

What can cause this issues?

Best regards,

Jan

[erwinbre]

Update: After trying to individually upgrade every possible driver on the ESXi-host, I still couldn't get this to work. So I decided to upgrade the host to ESXi 6.0 (after first upgrading the vcenter server to 6.0). Then I downloaded and installed the HPE StoreVirtual Multipathing Extension Module for Vmware vSphere 6.0 and it finally worked. I then went on and upgraded the 5 remaining hosts to ESXi 6.0 + Multipathing extension without a problem.

To upgrade the DL385G7 servers I used the custom HP ESXi 6.0 ISO as it contains more specific HP-drivers.

Kind regards,

erwinbre

[Kuiper]

Hi,

Update: I've tried also to install the MEM extension at one of our customers using four 4530 with three DL360Gen8's and Esxi5.5 , but they also got a Purple Screen of Deadth. So I think the 5.5 version is buggy and must not be installed. If someone has a better solution than upgrade to ESXi 6, please let me know.

Best regards,

Jan

[Kuiper]

Hi everyone,

I think I found the issue and reason why a purple screen is displayed after the installation reboot of the extension. Last night I've tried several times to install the module but every time it failed. After I changed first the Path Selection on all the datastores to Fixed or MRU the server was rebooted successfully. Then I could change the path selection to the HP extension. After checking everything seems to be working (a few warnings are displayed in the vmkernel.log complaining on a path change). After checking all paths the datastores I accidentally changed one datastore to Round Robin again and instantly the server crashed with the PSOD described earlier. After a reboot the servers crashed again because of the wrong path selection, the only solution I did find was to block the nics on the switch (disable port), after waiting very long the server was started up again and I was able to remove the extension. After enabling the ports on the switch and rebooted the server the ESXi environment was up and running again.

Lessons learned:
•    Don’t use or deactivate Round Robin if you are planning to use the HPE-LH-MEM
•    Inform all technicians to keep their hands of the path policy selection if the HPE-LH-MEM is installed

I hope that the issues described above are fixed in the extension for vSphere 6.

Best regards,
Jan Kuiper

[Stor_Mort]

Use VMware 5.5 update 3 with the StoreVirtual MEM driver for 5.5 for best results. VMware fixed something that was causing issues. VMware 6.0 and MEM for 6.0 seem to work well.

[Ste7N]

Hi All,

Same issue here with :

VMware-ESXi-6.0.0-Update2-3620759-HPE-600.9.5.0.48-Apr2016 (patched last week)

and VIB

Hewlett-Packard-Enterprise_bootbank_hpe-lh-mem_6.0.0-12.6.0.14

As soon I started modifying the path selection method (mixing HP MEM and previously set Roind-Robin) the system became unresponsive, and I had to reboot.

Then I started to get PSOD at every boot, and there was no way to get out of the loop until I decided to disable the read/write access to the volumes using the CMC.

After the timeout, the esx completed the load process and then I removed the driver via SSH:

esxcli software vib remove -n hpe-lh-mem

Hope to save you some time. I spent three hours trying to get out of this.

Best regards.

Stefano.

[Convergo]

Hi,

we had similar problems with the MEM.

We have multiple Storevirtual VSA clusters hosted on the same ESXi hosts. Running vSphere 6 and some external Windows 2008R2 servers the latest version we can use of LHOS is 12.5. We upgraded the first storevirtual cluster to 12.5. On the physical windows servers the DSM also upgraded succesfully (remind install this before upgrading). When we tried installing the MEM on one ESXi host we configured the acceptance level however we receive the message:

Unable to set acceptance level of accepted due to installed VIBs Hewlett-Packard (followed by a list of HPE vibs). Unable to solve this issue we decided to install the MEM vib for ESXi 6 : HP_StoreVirtual_Multipathing_Extension_Module_for_Vmware_vSphere_6.0_AT004-10525.vib

Hitting alt+F11 on the ESXi host a redline appeared :

Jumpstart dependency error : Initialization barier "vmkeventd" is not provided by any plugin but the following plugins require it : "nmp-hp-satp-lh", "nmp-hp-psp-lh", "hp_vaaip2000". During the boot some other messages appear like cpu0: HP_PSP DEBUG VERSION and char_driver : ALL Done. Being written all in red we suspect this is not good, but decided to go further while we could not solve these messages, or interpret them as being "valid".

It appeared that the LHOS 11.5 volumes of the second cluster were still configured with RR. Due to a six node cluster we had about 30 volumes and thus a lot of paths on the ESXi host. When we powered off one VSA in the first cluster, it seemed that the ESXi host entered a "stress mode" not being able to manage this host anymore. After reseting the host all volumes of the first VSA cluster disappeared.

For now we reverted to the RR pathpolicy by uninstalling the driver.

Questions that arise:

- is it supported to use the MEM vib on an ESXi host even when this host also connects to a 11.5 LHOS cluster ?

- when using 12.5 as LHOS in a cluster, is it supported to use the 12.6 MEM (while looking to the vib for vSphere 6 it is still version 12.0)

- Can the jumstart dependency error and the char_driver messages be ignored or do they indicate a fault ?

Anyone else experienced this behaviour on a ESXi 6 site with 12.5 clusters in combination with MEM ? The performance gain of 150% - 500% mentioned in the forum is definately worth testing and creating a stable environment.

thanks !

[danevald]

Hi VISABATI,

Quick fix, please correct the guide to uninstall, the command for uninstalling now contains HPE and not only HP :-)

~ # esxcli software vib remove -n hp-lh-mem <----

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=c04586354

[danevald]

After having a PSOD following the install of the MEM module I found this post. Here's my setup and workaround/fix to "quickly" move on:

HP DL380G9 running ESXi 6.0u2 fully patched feb2016, running Storevirtual VSA 12.5

If you get the PSOD, just disable the switchports where iSCSI is connected and boot the server, then it boots up, uninstall the module and reconfigure the Path policy on each iSCSI path to use either MRU or Fixed, then boot the server, enable ports again while booting, install module again, boot server, change path policy to HP LH policy, boot again.

This seems to work for me - using the 12.6 MEM module, but seeing quite a lot of "path errors" on the Alt+F12 log, and seeing CHAR_Driver ALL DONE in Alt+F11, I hope this will stop eventually...maybe after upgrading VSA to 12.6.

[Stor_Mort]

There will be a new version 12.6.0.20 of the MEM driver available this month (Feb 2017) which fixes all known bugs. It will be compatible with LHOS 12.5, 12.6 and 12.7.

If the path errors don't go away after updating, check this VMware article. ATS heartbeat failure sometimes produces path failure messages. 

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2113956

[Johannes_we]

So I´m running 12.6.0.20 on ESXI 6.0 U2 Build 4600944 but still see lot´s of char_driver: ALL Done.

Shouldn´t this have been fixed in 12.6.0.20?

[Johannes_we]

right in that second i got a PSOD so i think i´ll go back to RoundRobin native VMware.

[BenLoveday]

Anyone else seeing issues with the new MEM driver?

We are running the older 12.6.0.14 with vSphere 6.0U2 and are seeing a lot of random performance issues. Some clusters are rock solid, others have latency going through the roof.

If we switch to the RR PSP we sometimes hit a PSOD which is not great.

Does anyone have any release notes specific to the MEM driver only? The 12.7 LHOS release notes are very generic and don't mention anything in particular to the MEM which is frustrating.

Cheers,

Ben

[SVprodmgr]

Hi Ben, there is a customer advisory on the PSOD: http://h20566.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=4118659&docLocale=en_US&docId=emr_na-c05393399 on that version of 12.6.

Also, we have a separate multipathing user guide that might be helpful with MEM questions: http://h20566.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=4118659&docLocale=en_US&docId=emr_na-c05049830

Amy Mitchell

StoreVirtual Product Manager

[BenLoveday]

Thanks Amy!

I actually missed the multipathing guide recommendation on the delayed ACK. We worked with HPE support a few years back and at that time they didn't have a firm answer on disabling this so good to know it's the recommendation.

I'd really like to see a separate set of release notes for the MEM as I've seen several versions of the MEM released completely separate to the LHOS with no detail on what the updates have fixed.

Appreciate your help :-)

Cheers,

Ben

[ComboIT]

Spoiler

We're on MEM 12.6.0.20 and we're seeing repeated 'WARNINGS' like this hit the vmkernel log every five minutes:

2017-07-27T03:13:31.248Z cpu13:962933)WARNING: HPE_SATP_LH: satp_lhn_updatePath:430: In satp_lhn_updatePath setting path state to OK. vmhba32:C2:T1:L0
2017-07-27T03:13:31.248Z cpu6:964156)WARNING: HPE_SATP_LH: satp_lhn_updatePath:430: In satp_lhn_updatePath setting path state to OK. vmhba32:C2:T3:L0
2017-07-27T03:13:31.248Z cpu0:964778)WARNING: HPE_SATP_LH: satp_lhn_updatePath:430: In satp_lhn_updatePath setting path state to OK. vmhba32:C2:T0:L0
2017-07-27T03:13:31.249Z cpu8:960178)WARNING: HPE_SATP_LH: satp_lhn_updatePath:430: In satp_lhn_updatePath setting path state to OK. vmhba32:C2:T2:L0
2017-07-27T03:13:31.249Z cpu13:962933)WARNING: HPE_SATP_LH: satp_lhn_updatePath:430: In satp_lhn_updatePath setting path state to OK. vmhba32:C3:T1:L0

(this is a small sample, it's actually 32 events logged every 5 minutes).

I've tried disabling ATS heartbeat but this has made no difference. Does anyone have any ideas?

Thanks

David

[vdepagter]

I had a host yesterday that was unresponsive for management. Unable to connect with vSphere Client or through the Web Client. This started happening after attempting to spin up a Veeam SureBackup/Virtual Lab. A process in which (I guess) it connects different storage. The host showed disconnected in vCenter Server, the console was responding very delayed (by minutes). Alt+F12 showed floods of HPE_SATP_LH messages (see below). VM's were running fine. It eventually recovered. I rebooted it and rediscovered the iSCSI Static Paths as a precaution afterwards.

ESXi 5.5.0 build 4345813 (5.5U3+), hpe-lh-mem 5.5.0-12.6.0.20 connecting to 8x StoreVirtual 4530 12.6.0.155.

/var/log # tail vmkernel.log | grep -i "satp_lhn_pathfailure" | more
2017-07-30T17:34:24.396Z cpu10:61485)WARNING: HPE_SATP_LH: satp_lhn_pathFailure:1194: In satp_lhn_pathFailure unknown failure.
2017-07-30T17:34:24.449Z cpu46:65747)WARNING: HPE_SATP_LH: satp_lhn_pathFailure:1047: In satp_lhn_pathFailure status = 9 sense key = 4 and sense code = 2. path vmhba33:C0:T2:L0
2017-07-30T17:34:24.449Z cpu46:65747)WARNING: HPE_SATP_LH: satp_lhn_pathFailure:1048: path=vmhba33:C0:T2:L0 cmd[0]=c1 cmdid=8527
2017-07-30T17:34:24.449Z cpu46:65747)WARNING: HPE_SATP_LH: satp_lhn_pathFailure:1194: In satp_lhn_pathFailure unknown failure.
2017-07-30T17:34:24.502Z cpu16:66083)WARNING: HPE_SATP_LH: satp_lhn_pathFailure:1047: In satp_lhn_pathFailure status = 9 sense key = 4 and sense code = 2. path vmhba33:C1:T2:L0
2017-07-30T17:34:24.502Z cpu16:66083)WARNING: HPE_SATP_LH: satp_lhn_pathFailure:1048: path=vmhba33:C1:T2:L0 cmd[0]=c1 cmdid=8529
2017-07-30T17:34:24.502Z cpu16:66083)WARNING: HPE_SATP_LH: satp_lhn_pathFailure:1194: In satp_lhn_pathFailure unknown failure.

Additionally, I'm seeing the same WARNINGS David is having displayed on all hosts using the below combinations of software. Also seeing the "char_driver:  ALL DONE" message on the console Alt+F11 screen. These messages may have occurred during the time of the system boot.

VMware 5.5.0 build 4345813, hpe-lh-mem 5.5.0-12.6.0.20 connecting to 8x StoreVirtual 4530 12.6.0.155.

VMware 5.5.0 build 4722766, hpe-lh-mem 5.5.0-12.6.0.14 connecting to 8x StoreVirtual 4530 12.6.0.155.

I have not disabled VTS heartbeats. I have also not disabled Delayed ACK, as I found we are supposed to according to the MEM installation guide. http://h20566.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=4118659&docLocale=en_US&docId=emr_na-c05049830