Storevirtual 3200 Latency Issue

[mtroper]

Yes, I ran both updates at one time. (should be possible in my opinion)

My setup is one SV3200 10GBE iSCSI with 3 ESXi 5.5 U3 hosts.

[Nor3344]

The patching of 136-16-00 just finished here and seems everything went ok here except the patch seems to have removed all the custom routing we had added on the MGMT interface... Crossing my fingers that it keeps running stable and I will rerun some of the benchmarks to see if the performance have improved. 

> Yes, I ran both updates at one time. (should be possible in my opinion)
Definitely! But after talks with L2 regarding some of our issues HPE does not seem to publish all the best practice advises

Rich: I saw the same temprature errors during my update. All cleared after a short while.

[referencepoint]

@Nor3344how is your performance on the SV3200? Most people in this thread seem to have major performance issues when using it with VMware hosts, and you said you were using ESXi 6.0u3.

[Nor3344]

@referencepoint I am not to impressed. I have only performed tests over a short periode of time so I will need to rerun some tests after the pathcing. The benchmarking I have done is available on github. I will rerun the same tests soon and extend the test time to see if I see any improvements. If you look under the log folder and the lat files you will find that we experienced instability in the latency during the testing.

[referencepoint]

Just having a look at your outputs, thanks for these. Is the vertical axis in all of your graphs IOPS x 10¹?

[Nor3344]

@referencepoint I only have SAS drives so I would really like to multiply the numbers with 10, but I am affraid that is not the case.

Site A:
2xRaid5 with 6 disks each. 1xSpare (13 disks in total)

Site B:
2xRaid5 with 6 disks each 1xspare (13 disks in total)

All volumes are Network RAID-10 (2-way mirror)

[mtroper]

Ok, HP support is officially driving me mad....

After I solved the messed up Storage Controllers by myself (which were caused by changing the bond traffic routing type regarding http://h20565.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-a00019441en_us . Which was adviced from L2 support prior to the patches) i made some performance tests again.

Performance has not increased after installing the latest patches (135-015-02  & 136-16-00).

After i played a bit with failing over the storage controllers it i saw that the performance increases (almost dubbled) when all disks were failed over to only one controller.

The answer from HP L2 support on this:

"Lets do some testing again.... (i declined and asked for perfomance values from HP test environment first)"

HP answer:

Ok you are doing the test on  the formatted drive and not on the RDM.  We are aware of the performance issue on the SV3200 and there is a fix which is coming soon which will fix the performance issue with ESXI.  The fix will be released end of October.

[GuillaumeRainer]

Octobre ? Man, that's tough. That's another three months to go...

[SVprodmgr]

Thank you for posting on this forum about your findings on SV3200 with the two patch sets.  I talked to our engineering team about the findings and they suggested that you should elevate this case to L3.  You might have a unique case that needs the subject matter experts to investigate.

Amy Mitchell

StoreVirtual Product Manager

[GuillaumeRainer]

Well, I had pretty much the same unique case of running esxi-iscsi against sv3200; finally had to give up since L3 found the same problems as I did. I feel the number of esxi Customers with high spec' ed arrays (read: ssd or a good amount of disks) AND 10g is pretty low and thus patching and testing have not been done so far. 

Thanks for keeping us posted, through, I would love to use those arrays in our smaller projects...

[SVprodmgr]

We recently updated the SV3200 quickspecs to show some of our test results here: https://www.hpe.com/h20195/v2/GetDocument.aspx?docname=c05158565

The benchmarks are at the end of this document.

Amy Mitchell

StoreVirtual Product Manager

[shapirm]

Hi , 

I also have the same 0x28 errors in hp gen 9 ESXi 6.5  on  10Gb copper with x540-at2 cards in servers.

The speed that I can get in windows server  with crystal mark benchmark and atto benchmark is about 100MB/s on 10Gb links while storevirtual 3200 is connected to nexus 3000 and scaled out to 2 nodes and total of 4 controllers.

Iscsi nexus ports is breaout cable of 4 x 10Gb ports.

What can be the reason for such poor performance ?

Any news about the october patch ?

It will solve the 0x28 buffer errors?

Thanks

Michael

[sebastianmair]

hello, any news on updates to sv3200, we have a customer that will burn down this holy crap of *****. i cant understand why hpe is not helping us, just reinstalled all 4 controllers, we have 2 sv3200 with 10gbe iscsi. same problem again. on some time, the datastores are so slow, that the virtual machines on vmware just stops and crashes..

hpe, what the hell are you doing?

any news?????

[mtroper]

Dear HP

October is almost over and still no patch or problem fix. Could someone from HP give us a specific date for the update to be released?

Warranty of our productive MSA storage is running out in two weeks and we have still no solution until now.....

regards

Marc

[richa3312]

Hi Marc,

Not an official update but we had this back from a ticket we had open.

Problem Resolution:

• No reason observed from the received log for the Downtime, Advised  customer to upgrade to  LHOS 13.6 which will be released by End of October 2017 for improved performance of the System

We got this on the 21^st of September.

Rich

[mtroper]

Hi Rich

Thanks for your information. (HP Support told me the same in August)

But where is the update? Announced but not delivered until now.

Also I do not see any official HP announcement for LeftHand OS 13.6 (no official infos found on this release)

In my opionion HP is only buying time... (and i do not have time anymore because the warranty of our productive SAN is running out)

I bought the SV3200 one year ago and cannot use it because of the worst performance ive seen on a ISCSI SAN since years. (even a cheap QNAP NAS brings better performance on ESXi)

My patience is definitely over at the end of this month...

regards

Marc

[LeeTse]

The fix will be released end of October? 

Now is mid November, but no news from HP.

[Bart_Heungens]

Same question here, no update available... ???

[richa3312]

Still no word on the update but I noticed this on Critical Alerts for your Servers and Software bulletin.

Not sure if this refers specifically to the 3200 DSM or the Storevirtual O/S DSM. Its a bit scant on information.

I've been running the Storevirtual 3200 DSM with no issue for some time and it provided a performance boost when installed.

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: a00035478en_us

Version: 1

Advisory: HPE StoreVirtual 3200 Storage - Multipath Configurations Require The Operating System-Provided Multipathing Driver

NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.

Release Date: 2017-11-08

Last Updated: 2017-11-08

---

DESCRIPTION

In HPE StoreVirtual 3200 Storage multipath configurations, use the OS-provided multipathing driver, such as the Windows multipath driver. The HPE StoreVirtual DSM driver is not compatible with HPE StoreVirtual 3000 storage products and may cause performance issues or system instability.

SCOPE

This advisory applies to HPE StoreVirtual 3000 Storage systems.

RESOLUTION

Use the OS-provided multipathing driver in HPE StoreVirtual 3200 multipath configurations.

If the HPE StoreVirtual DSM driver has been installed, uninstall it and revert to the OS-provided driver to avoid future issues.

[Romain BRICAUD]

The HPE StoreVirtual OS v13.6 Release Notes release note says :

The StoreVirtual DSM is no longer supported in the StoreVirtual OS software. For information about using

the Microsoft Windows OS-provided multipathing driver, see the HPE StoreVirtual 3200 Storage User Guide

The StoreVirtual Migration Manager is not supported in StoreVirtual OS version 13.6 or 13.5.

I think no more DSM is supported !

The 13.6 upate is visible in the SVMC.

[Kipp_Glover]

StoreVirtual OS 13.6 released Tuesday (14 November).  Systems can be upgraded through the StoreVirtual Management Console (SVMC).

There are several performance related improvements in this release.  For details, see the HPE StoreVirtual 13.6 Release Notes on the HPESC:

https://support.hpe.com/hpesc/public/home/documentHome?docId=emr_na-a00035652en_us&document_type=134%2C138%2C140&language_code=1871440&sp4ts.oid=1009115875

[mvbk]

Good morning,

i there anyone else who upgraded their systems and have some performance results? Last week I upgraded a two-node setup with a FOM from 13.5 to 13.6 but I still get read and write latencies.

The upgrade itself didn't go very smooth, but at least I was able to upgrade the four controllers (two nodes). The FOM is still having issues and I can't upgrade the FOM. The web interfaces of the cluster and all controllers are just not workable. I'm able to log in, but most of the time I only see some 'loading' symbols. If I'm lucky I get some feedback after half an hour.

Still no feedback after two days from HPE after logging a case. I assume I won't be the only one still having the latencies?

[ThomasU2]

Hi mvbk

Have you had any issues when upgrading the 4 controllers? I am stuck at the first one, saying "rebooting" at 90%, 3hours and ongoing.

I bought 2x sv3200 encolsures 2 months ago, had unbeliveable huge problems to get those 2 up and running, also 1 controller was DOA. Luckily my sv3200 cluster is still not live, but I am very, very worried if I really should use it in the live enviroment from what I read here. I have an esxi 5.5 U3 cluster, and the sv3200 will be connected with iscsi. It should replace a P2000 G2.

As soon as I get those **bleep**ty 4 controllers up and running, I will make performance tests as well.

Just to be with all of you. I had to re-image all 4 controllers multiple times until I managed to get it up and running. had an HP technician on the phone for at least 5 hours (2hours straight one time).

regards
Thomas

[mvbk]

ThomasU2,

I'm "glad" to hear I'm not the only one.

We installed the two nodes in June but it was a pain in the ass. One node was DOA as well, but only after 4 days they started to believe it. I was able to configure the other node, but after an IP-change I couldn't reach anything anymore, so a reimage was the only way out. I also had to wait for 3 days for someone to give me a guide how to perform a reimage. At that time it wasn't documented yet.

During my upgrade last friday, the process crashed multiple times. After each controller upgrade, it failed, but the OS version was the new 13.6 though. However I can't be sure the upgrade was completely succesfull... It was also around the 90% when I was having some issues, but the process failed quite rapidly after that. So I didn't have to wait for multiple hours. The last item was the FOM, which was corrupt during the upgrade process and up till now, I still haven't been able to add a new 13.6 FOM to the management group.

[ThomasU2]

Hi mvbk

Ok, After I removed the one controller (reboot manually ;-) the update to 13.6 was successfull.

It seems everything is fine now.

One question, why do you have a FOM when you have 2x sv3200 enclosures? You dont need that, there are already 4 controllers. I didn't even find a FOM to download for my sv3200's

The performance could be better and it is in fact slower when using an ESXi storage virtual HD THAN attach it to a Windows VM with iscsi directly...but not that much.

I think about adding SSD's as cache, dont know if that would make a huge difference. I have 2 nodes with 2x5 disks each.