StoreVirtual Storage
1748176 Members
4032 Online
108758 Solutions
New Discussion

StoreVirtual Multipathing Extension Module for vSphere 5.5 missing VMFS datastores

 
rossiauj
Occasional Advisor

StoreVirtual Multipathing Extension Module for vSphere 5.5 missing VMFS datastores

Hi,

 

We have a 16-node storevirtual 4530 multis-site cluster which we use to publish VMFS datastores to our ESXi 5.5 hosts across two sites. Each site contains 8 storage nodes and 5 ESXi 5.5 hosts. Both nodes and ESXi hosts are connected through 2 x 10 Gb connections (iSCSI configured on ESXi according to HP best practices and ALB on storage nodes, FlowControl enabled on switchports, hosts and storage nodes).

 

When we use the default Storage Array Pluggin VMW_SATP_DEFAULT_AA with Round-Robin (VMW_PSP_RR) we have no problems. However when we install the recently (re)published HP StoreVirtual Multipathing Extension Module for VMware vSphere 5.5 (AT004-10518) we experience problems when the ESXi hosts are not cleanly shutdown (iLO poweroff or reset).

After the ESXi hosts are started again, we can see only a small number of the recently available VMFS datastores. I can still see the devices listed (so the luns are available to the ESXi host), only the VMFS datastores are not mounted.

 

When I run the command 'esxcli storage filesystem list' I get these errors on the ESXi host:

 

 

Error getting data for filesystem on '/vmfs/volumes/5551c88a-f3041c44-4ab8-8cdcd4afa378': Cannot open volume: /vmfs/volumes/5551c88a-f3041c44-4ab8-8cdcd4afa378, skipping.

According to some VMware KB article this could be attributed to ATS locking behaviour of the VMFS datastores after a unclean shutdown. I disabled ATS on all ESXi hosts and then rebooted them but the reboot got stuck after the message "vfc loaded successfully". It probably took half an hour or even more to boot.

 

A normal reboot or shutdown (initiated from the vSphere client) does not cause problems, only a dirty/unclean/unexpected poweroff or reset.

 

I removed the LH MEM  from the ESXi hosts and the problem has gone away. However, the LH MEM gave a very big performance boost (site aware iSCSI connections, true multipathing to all nodes in the site instead of only one gateway connection). I noticed performance and throughput boosts of up to 150 to 500 percent. It would be really bad to miss out on all that performance goodies but I just don't feel very safe with the LH MEM installed.

 

Anyone else seen this problem or know how to fix it.

 

The ESXi hosts and storage nodes are not yet running production but will be the next couple of days when we migrate our old environment to this new one. I hope this gets sorted out soon.

 

Thanks in advance.

 

Jos Rossiau

43 REPLIES 43
fqu
Frequent Visitor

Re: StoreVirtual Multipathing Extension Module for vSphere 5.5 missing VMFS datastores

I've had a big problem with the LH MEM too.

We are using 8 nodes P4500G2 upgraded to StoreVirtual 12 and we are on Vsphere 5.0.

I've tried the new LH MEM module instead of using the default RoundRobin (which works perfectly), and i've started to loose connection with LUNs (we have 15 LUNs in RAID 10 and 1 LUN in RAID 10+2 for essential VMs).

I was loosing the connection with the LUN in RAID 10+2 :the Virtual Center and the CMC were on that LUN !

I've had to connect directly to each ESXi to re-configure them to use the good old RoundRobin.

I think that the new LH MEM module is not ready for production now...

 

balblas
Advisor

Re: StoreVirtual Multipathing Extension Module for vSphere 5.5 missing VMFS datastores

We have exactly the same problem. It looks like if the connection to the iSCSI device is interrupted the LH PSP does not always recover correctly, even though all paths are fully functional again. Logs show issues with ATS locking, but I believe the root cause has nothing to do with ATS but with paths to the volume not being recovered by the LH PSP.

 

Looking at the iSCSI connections to the volume in the CMC I noticed only HP MPIO Control connections existed for the failing host and no HP MPIO Data connections. I suspect without data connections the host can't read or write to the LUN.

 

Rebooting the host is the only way to recover from this. Sometimes the host seems to be stuck and a hard (ungracefull) reset is needed to recover the host.

GarethRUK
Occasional Contributor

Re: StoreVirtual Multipathing Extension Module for vSphere 5.5 missing VMFS datastores

Hi,

 

Does anyone have an update on this issue. Has anyone spoken to HP support regarding this.

 

I've seen some very strange behaviour and failures, but would like to understand the problem.

 

Regards,

 

 

Gareth

JazzyB
Occasional Advisor

Re: StoreVirtual Multipathing Extension Module for vSphere 5.5 missing VMFS datastores

I experience a similar problem when i enabled the driver on 1 ESXi 5.5 host and changed 1 datastore to use the HP_MEM driver.

 

The datastore immediately disappeared from that host and could not be rescanned or refreshed. I could see the iSCSI paths but the partition was "unknown"

 

Changing the paths back to VMWare round robin did not reset the SCSI paths back to MPIO Data + Control. They were stuck on the HP ones.

 

I finally gave up - came here to see if there were issues and removed the driver and rebooted the host. Now everything is back to normal.

 

I'm guessing this is not ready for non lab use right now. Also are there plans to support ESXi 6?

 

 

HPstorageTom
HPE Pro

Re: StoreVirtual Multipathing Extension Module for vSphere 5.5 missing VMFS datastores

It seems that the issue is not always there after a unclean shutdown or power outage. My systems came at least up without any problems after a hard power outage due to thunderstorms.

I've asked some engineering folks whether they could reproduce the issue, buth they didn't have any luck either. What they asked me now is whether somebody of you would be able to provide a vm-support log pulled after experiencing the failure?

Generally, I can only recommend to open a support call if you experience such an issue (yes I am assuming here that your systems are still under an HP support contract ...).

rossiauj
Occasional Advisor

Re: StoreVirtual Multipathing Extension Module for vSphere 5.5 missing VMFS datastores

After the initial tests with the multipathing module for vSphere 5.5 we decided to go to production without it.

We used the VMware standard plugin instead.

 

However, I've found some time to fiddle with it and I have installed it on one of the ESXi 5.5 hosts in our farm.

 

I did a normal reboot (from the console) and I did a Reset (via iLO) and both times the host rebooted and all the LUNs where available.

 

Although I did download the VIB again it seems to be the same version as before (<name>hp-lh-mem</name><version>5.5.0-12.0.0.55</version>).

 

We did however split our 16 node storage cluster into two clusters (1 x 10 node and 1 x 6 node), but that should not be the cause of the problem, now does it? Unless it was a max paths problem.

 

So unfortunately I will not be able to provide HP information on the problem.

 

I will keep running some more tests on the host and see if it is stable.

 

Greetings,

 

Jos Rossiau

oikjn
Honored Contributor

Re: StoreVirtual Multipathing Extension Module for vSphere 5.5 missing VMFS datastores

depending on the number of servers and nics, a 16 node cluster can cerfainly become a problem for # connections with MPIO.  I forget the max numbers offhand, but I do think it is something around that 10 node cluster that you have to start paying attention to that.

rossiauj
Occasional Advisor

Re: StoreVirtual Multipathing Extension Module for vSphere 5.5 missing VMFS datastores

From the deployment guide a 16-node cluster connected to ESXi hosts with 2 x 10Gbit NICs (32 paths per device) would only just be supported. However, a 16-node cluster would not be best practice, so we decided to split it.

 

Besides, I do not believe the other people having the same problem as we did were using 16-node clusters, not sure how many NICs though.

 

I installed the module on another of our ESXi 5.5 hosts and will keep it for a while, during which I will do some tests with them.

rossiauj
Occasional Advisor

Re: StoreVirtual Multipathing Extension Module for vSphere 5.5 missing VMFS datastores

Today installed version 61 of the HP LH MEM driver and after reboot was missing 8 of the 26 volumes on the hosts that I updated.

 

All paths were up and used. CMC showed 2 control and 10 data connections, as normal. ESXi just did not mount the volumes.

 

I can leave one of the hosts as is, if HP support needs me to do some log searching/troubleshooting, but not for too long as it is not usuable to us now and I need to revert back to the default VMware multipathing driver as soon as possible.