What are options for upgrading a 2-node StoreVirtual VSA cluster to improve its disk storage??

[Tom Lyczko]

We have a 2-node StoreVirtual VSA cluster running on StoreVirtual VSA 12.6 with an NFS quorum witness, two HP DL380s with 10 x 15k 600GB SAS drives in RAID 5 per server, then HPVSA makes the two servers into network RAID 10.

We're presently collecting data to determine if the two servers' arrays do or do not supply enough IOPs for the VMs involved, all running on vSphere 5.5 Update 2, hopefully to upgrade to 6.0 this spring. Would prefer vSphere 6.5 but HPE has not caught up to 6.5, wanting everyone to wait for 12.7.

The problem we're having is that the two servers each have 10 x 15k 600GB SAS drives in RAID 5 per server, then HPVSA makes this into network RAID 10, meaning a lot more writes and IOPs and disk latency than originally anticipated.

Anyway, If we get a 3rd server and add it to the cluster, can this 3rd server be all or mostly flash/SSD drives and the other two servers remain all 15k 600GB SAS drives?? and juse adaptive optimization to properly situate data being used by SQL and Exchange servers??

Or must the over-priced low-capacity HPE Gen8 flash/SSD disks be added into the two existing servers, replacing the SAS drives?? That is, for the adaptive optimization to work, must I have flash/SSD drives in all the servers or can I just have the flash/SSD drives in one new additional Gen9 server and leave the other two servers alone??

I'm asking b/c it would really be easier to just add the 3rd server than buy over-priced low-capacity Gen8 flash/SSD drives...

What other options do I have to increase IOPs and reduce disk latency with this 2-node cluster?? besides adding a 3rd server (and all the storage reconfiguration that will ensue no matter what)??

Also, when using adaptive optimization how should the disks/LUNs be configured wrt the physical disks, must each server contain some flash/SSD disks or can one (1) server contain all the flash/SSD drives in it??

Thank you, Tom

[Stor_Mort]

Hi Tom,

Sorry, the rule for StoreVirtual clusters is that all nodes in a cluster must be as identical as possible. Same disk configuration, raid controller, memory, CPU resources. It's impossible for them to balance the load correctly if they have different characteristics.

AO requires at least two mirrored SSD drives in each host to provide tier 0 storage for each VSA. This solution is well proven and easily provides a performance boost for frequently-used blocks.

But be sure your existing SAS cluster is performing as well as possible before resorting to an expansion project. VMware does not come optimally configured for SAN operation out of the box. There have been a number of VMware KB articles about improving storage performance.

ESX/ESXi hosts might experience read/write performance issues with certain storage arrays (1002598)

Controlling LUN queue depth throttling in VMware ESX/ESXi (1008113)

Poor TCP performance may occur in Linux virtual machines with LRO enabled (1027511)

Adjusting Round Robin IOPS limit from default 1000 to 1 (2069356)

Issuing a 0x85 SCSI command from a VMware ESXi 6.0 host with the EMC XtremIO storage array may give PDL error (2133286)

ESXi host loses connectivity to a VMFS3 and VMFS5 datastore (2113956)

Also be sure to follow recommendations in the VSA Installation and Configuration Guide,  https://www.hpe.com/info/storevirtualdownloads

Network Design Considerations and Best Practices,  https://www.hpe.com/h20195/v2/getdocument.aspx?docname=4AA2-5615ENW

VSA Design and Configuration Guide, http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA4-8440ENW.pdf

[Tom Lyczko]

Wow. This is really extreme. And excessively difficult to justify to other people, sigh. "Oh, we need a third server."

How does one calculate what quantity of storage per server should be flash/ssd drives for AO to work well??

And how should the server be set up for this to work??

Say we have 12 LFF drive bays available per server and we have 2 or 4 flash/ssd drives per server how are things set up wrt the physical arrays and in the VSA configuration??

I'll look at the articles and see if any of them make any sense.

We probably have mostly defaults in the configuration, I'm not sure we could improve anything there but I'll look.

Thank you, Tom

[Stor_Mort]

The ratio of tier 0 SSD storage to tier 1 SAS depends on your application environment. There's an 80/20 rule of thumb that estimates only 20% of your storage is accessed "frequently". That's probably as good a starting point as any. We usually recommend 10% to 20% of your storage should be in tier 0. If, for example, your primary application is a database that constantly hits only 1% of your total storage, you might be able to get by with less in tier 0.

You would need to rebuild your VSAs to remove some of the SAS disks and replace them with SSD. The contents of the store will be destroyed when you remove disks. Set up two raid sets, one for SSD and one for SAS. Assign them to the VSA as hard drive (1,0) and hard drive (2,0). In the CMC, select Storage under the VSA storage system and select the RAID setup tab. Click the RAID Setup Tasks button. Reconfigure RAID for both devices. Reconfigure Tiers to assign the SSD raid set to tier 0 and SAS to tier 1. Note that a 10TB or 50TB VSA license is required to enable Adaptive Optimization.

[Tom Lyczko]

---

@Stor_Mort wrote:

The ratio of tier 0 SSD storage to tier 1 SAS depends on your application environment. There's an 80/20 rule of thumb that estimates only 20% of your storage is accessed "frequently". That's probably as good a starting point as any. We usually recommend 10% to 20% of your storage should be in tier 0. If, for example, your primary application is a database that constantly hits only 1% of your total storage, you might be able to get by with less in tier 0.

I'm not sure I would know what percentages of total storage our databases use, we have two db servers, one of which is several hundred GB, almost 1 TB, the really busier one of the two is presently only a couple hundred GB I think. Plus the mail server is around 300-some GB. These are all approximate total storage for each of the three servers, how do I know what data about these servers to use for deciding how much flash/ssd to install?? To the best of my knowledge, flash/ssd disk options for HP DL380 Gen8 are reallly minuscule compared to what should be available, given the current ubiquity of flash/ssd disks.

You would need to rebuild your VSAs to remove some of the SAS disks and replace them with SSD. The contents of the store will be destroyed when you remove disks. Set up two raid sets, one for SSD and one for SAS. Assign them to the VSA as hard drive (1,0) and hard drive (2,0). In the CMC, select Storage under the VSA storage system and select the RAID setup tab. Click the RAID Setup Tasks button. Reconfigure RAID for both devices. Reconfigure Tiers to assign the SSD raid set to tier 0 and SAS to tier 1. Note that a 10TB or 50TB VSA license is required to enable Adaptive Optimization.

I know the whole storage would have to be redone. I imagine the flash/ssd and SAS are configured as separate arrays that logically become the above-mentioned "hard drives" and RAID configurations. Is the VSA config something that HPE support can help out with??

Right now I'm dealing with the data collection aspect to be able to explain to non-technical people why our apps etc. are slow.

Thank you, Tom

---

[Mark_A]

You don't say what models of DL380 you have.  A couple of simple things to optimise I/O.  Firstly make sure you've got the biggest FBWC caches in your RAID controllers and then set them to 90% write cache.  This will solve many write wait problems.  Frankly setting high read caches on a SmartArray is a waste of money, it's taken HP years to realise this and only with Gen9 servers are they now setting 90% write cache as default!

Secondly make sure each of your VMs have enough memory to minimise the amount of read activity.   Giving SQL Server and Windows Server 2012 a bit of extra main memory can work wonders in reducing I/O.

[Tom Lyczko]

---

@Mark_A wrote:

You don't say what models of DL380 you have. 

DL380 Gen8 with embedded Smart Array P420i Controller

A couple of simple things to optimise I/O.  Firstly make sure you've got the biggest FBWC caches in your RAID controllers and then set them to 90% write cache.  This will solve many write wait problems.  Frankly setting high read caches on a SmartArray is a waste of money, it's taken HP years to realise this and only with Gen9 servers are they now setting 90% write cache as default!

How/where does one set write cache?? I don't know the existing settings.

Secondly make sure each of your VMs have enough memory to minimise the amount of read activity.   Giving SQL Server and Windows Server 2012 a bit of extra main memory can work wonders in reducing I/O.

---

The servers have enough memory and CPU, that I can see for sure, these are not the issue...

Thank you, Tom