Single pool with MSA-DP+ or two pools with RAID5 or 6?

[JDDellGuy]

Hello,

I'm new to SAN planning, so I'm discovering a best practice question after already ordering and receiving equipment.

We have a dual controller HP MSA 2062 SAN with 2x 1.9 TB SSD and 12x 1.8TB 10K HDD.

I see that MSA-DP+ is a best practice and 12x HDD is the minimum in order to use that.  However, I see that it's also a best practice to have two storage pools so that the controllers can balance their load effectively.

So I have three options.

1. I can configure a single storage pool with 12x 1.8 TB HDD in MSA-DP+ and the two SSDs as a read cache.
2. I can configure two storage pools, each consisting of 5x 1.8 TB HDD in RAID 5, 1x SSD for RAID cache, and 1x spare HDD.
3. I can configure two storage pools, each with 6x 1.8 TB HDD in RAID 6, 1x SSD for RAID cache, and no spare HDDs.

Net storage capacity is sufficient in any of the three scenarios.

I don't like option 2 because the drives are a bit large in capacity for me to be comfortable with the potential rebuild times, even with a hot spare.

I don't like option 3 because it leaves me without any hot spares.

The only downside I see to option 1 is that it leaves me without good load balancing between the two controllers.

What would you do?

[JonPaul]

Great question.

Short Answer:  Option 1 should be used
The MSA systems are capable of impressive performance on a single controller. So much so that now unless you have all SSDs or more than a 24 drives you will not see an improvement in performance using 2 Pools.  Each Pool is controlled by a controller almost independently, therefore if you have 2 pools you are using both controllers and maximizing the system performance. But if the media on the backend, in your case 12x 1.8TB 10k HDDs and 2x SSDs, is not capable of exceeding the performance of a single controller then you are better off with a single Pool to limit overhead (Parity/Sparing) and as a side benefit when a controller is shutdown for a firmware update or fault the performance of the system will remain consistent, running 1 Pool on the other controller during the shutdown.
RAID 5 is not recommended for spinning media.  MSA-DP+ would be recommended as it has additional features which improve the experience:  Included spare capacity - this is spread throughout the disk-group so that if a drive failed a rebuild starts without a new drive introduced, it also will spread the reconstruct load out to all the remaining disks instead of just one drive getting all the WRITEs. This improves recovery time, with a lot of drives dramatically.  
Expansion is possible - MSA-DP+ lets you incrementally expand your system, as you are starting out with 10 open slots this may be important in the future.
As you purchase an MSA 2062, you also have the included license to allow a performance Capacity Tier (RAID 1 on SSDs).  This would also improve performance over the Read Cache option you describe but I would recommend to start with the Read Cache and see if your performance is where you need. It's an easy task to remove the Read-Cache and re-deploy as Capacity tier, both initially and later.

[Hamed_Masoud]

Thanks HPE Team, I also had another question in this regard which was mostly in the availability and redundancy side rather than performance. In our company we usually configure MSA storages by splitting available phisycal disks into two disk-groups and create two Volumes in each disk-group resultign in 4 volumes. then one volume from each pool will be mapped to one phisical host (usually two hosts) so each host has two volumes mapped to it one from each disk-group.

The application that is installed on physical machines and using the storage can use any of the available volumes mapped to the servers to write its data on and has no preference or limitation on the number of available volumes, the only consideration is the size and availability.

I was wondering how moving away from our current two volume (RAID6) configuration towards one volume DP+ can affect the availability of the volume in case of possible hardware failures. is there any scenario that can render the whole pool being unavailable (Except for power and both controllers failure scenario)? how many individual disk failures does it take for the whole volume to be unavailable in DP+?

Thanks

[JonPaul]

@Hamed_Masoud 
The first part of your question: Is there a scenario that could render the pool offline?
You point out the obvious of power outage and dual controller fault.  There are other infrastructure issues which could cause a data unavailability case, ex switches go offline, all host cables removed.  Also when a disk-group loses too many disks the disk-group will go into a quarantined state and the pool will go offline.
For MSA-DP+, the disk-group is closely related to a RAID 6 and from full Fault Tolerant state can lose 2 drives at one time.  Depending on how many disks in the MSA-DP+ disk group, this will effect different stripe zones differently.  Some stripe zones will be missing 2 chunks of data or parity, some 1 and some none.  On of the additional features of an MSA-DP+ disk-group is the distributed spare capacity.  Each drive in the disk-group will have some spare space so the stripes with missing chunks of data will be rebuilt onto available spare capacity.  By default the disk-group has 2 drives of available disk-capacity so these first 2 drives loses will be completely rebuilt even with a completely full disk-group.  The rebuild is fast as the data is not rebuilt onto 1 or 2 disks but the entirety of the remaining disks.  After the rebuild, the system can lose 2 more disks as it will again be in RAID 6 fully fault tolerant.  That is the minimum number of drive loses the MSA-DP+ disk group can sustain.  Typically a disk-group is not fully used and in that case un-used stripes need not be rebuilt, resulting in more spare capacity and additional drive loss ability.  Then reaching beyond that the MSA-DP+ disk-group has the Rebalance Fault Tolerant Stipes (REFT) where stripes which are fully fault tolerant can donate capacity, making them degraded but still able to sustain 1 drive loss, to improve Critical stripes (no fault tolerance left) to degraded.  So in answer to the second part of your question, it's hard to tell how many drive losses you can sustain in an MSA-DP+ disk-group.  It depends on timing of drive losses,  amount of data residing in the disk-group and the number of drives in the MSA-DP+ disk-group.   As a best practice it is recommended to increase the spare capacity as you increase the number of drives, there is a table in the MSA 2060 Best practices guide.  The above is also explained in the Virtual Storage Reference Guide.
https://www.hpe.com/psnow/doc/a00103247enw
https://www.hpe.com/psnow/doc/a00105260enw

[ZVE-Minh]

Dear  JonPaul  - HPE PRO

We are using Fibre HPE MSA 2060 SFF with 12 HDD SAS * 2.4TB (No SSD for Read-cache), So is it Ok if we intend to use 10HDD for RAID6 (rather MSA-DP+) in a single POOL and 02 HDD for spare.

thank you for your advice

[JonPaul]

@ZVE-Minh 
10 Drive RAID 6 w/ 2 spares is certainly ok and fully supported but you give up the benefits of MSA-DP+.
MSA-DP+ will utilize all 12 drives in I/O, this translates into 2 more spindles for performance.  As all the drives are in use you don't have to worry about a spare drive being faulty when you need it (note: the MSA Gen6 by default scans AVAIL and SPARE drives periodically anyway).  But the biggest 2 benefits are faster rebuilds and incremental expansion.
When a drive faults and a rebuild needs to happen on RAID 6, all the data that was on the faulted drive is re-created on a new drive.  The rebuild performance is limited by a single drive performance.  With MSA-DP+ there is spare capacity on every drive in the disk-group so when the rebuild happens the data is rebuilt across segments of all the drives and rebuil can happen much more quickly. For example in your scenario there is ~2TB of data on every 2.4TB drive and a drive faults.  With RAID 6 all that data needs to be recreated on a single drive. If rebuild can run @ 100MB/sec per drive:  2,000,000MB / 100MB/sec = 20,000 sec > 5hrs  but if you had 11 drives to rebuild to this could be less than an hour to rebuild the 2TB the rebuild time will likely be more than this as you are reading from the same drives you are writing to but the actual will be less than a RAID 6 rebuild.  As the number of drives increases the rebuild times drop. 
The other big benefit is incremental expansion with RAID 6 if you needed to expand the POOL to support another project you would need to add another complete disk-group and best practice would be to add 10 more 2.4TB drives.  Another option would be to create a second POOL but the system only allows 2 pools.  With MSA-DP+ the system will allow the addition of a single drive or multiple drives and will rebalance the existing data on the new drives so if you only needed another TB, just add 1 drive. Or if you needed to double the capacity add 10 drives, up to 128 total.

[ZVE-Minh]

Dear JonPaul  HPE PRO

Thanks for your excellent explanation and advice, i will use all disks (12 * 2.4TB SAS SFF) for a DiskGroup running MSA-DP+ in a single Pool

Your sincerely and have a nice weekend

[darrenwardle50]

@JonPaulIf I may, we've just bought a 2062 to replace a couple of 2040's. We've got 8x 3.8TB SSD and 46x 2.4TB SAS 10K discs, which we're intending to performance tier with the spinning discs using MSA-DP+.

I've read the Best Practices guide, and there's a standing recommendation to use single-pool in most circumstances due to the capabilities of the controllers, however I've not found if there's a practical limit. With this sort of quantity of disks would it be best to use a single-pool, or split evenly into dual-pool?

Thank you in advance.

[JonPaul]

@darrenwardle50 
With the number of SSDs you have, you can easily saturate a single controller and come pretty close to saturating both controllers. 300k+ IOPS of READ with both controllers (See quickspecs all the way at the bottom) 
The question then becomes what performance do you need and how you are going to use the storage.  If you believe that the performance of a single controller  (~175k Random READ IOPs, dividing the quickspec number by 2) then for simplicity in configuration a single pool would be preferred.  If you have multiple different workloads or uses for the storage and you want to divide that workload separately on different storage then 2 Pools would be better,
Some reasons for a single Pool:
  - Simplicity of configuration
  - Lower overhead - RAID parity
  - better performance for small configs (24 or less drives)
  - Active/Passive configuration
Some reasons for dual Pool:
  - Differing workloads/uses (eg. Database that runs your business vs Backup file share)
      - Maybe non-equal Pools
  - Maximum performance, equal Pools
      - Complexity on the host side of concatenating the multiple LUNs

If you go with a Single Pool make sure that you are adjusting the MSA-DP+ 'Spare Capacity' to stay inline with Best Practices, this must be done in the CLI.

[darrenwardle50]

@JonPaulThank you very much for the response.

From what you've said, it sounds like creating two pools of 4x SSD and 23 discs in a MSA-DP+ group seems appropriate.

I am a little confused with the comment about adjusting the MSA-DP+ Spare Capacity - I thought it assigned this at the time of creation to be 2x the largest drives? The Best Practices document only mentions about manual intervention if you are expanding with larger discs later - and I'm not sure why it would be different between a Single or Dual pool?

One last question if I may - reading the manuals, it appears that if there's an SSD fault then it will drain all the remaining SSD's onto the Standard Tier. In this context, can I assume that it won't attempt to use any excess Spare Capacity to do this and you just have to ensure that the Standard Tier has enough unused space to do this? I don't believe it otherwise reserves the space for this operation?

Again, many thanks.

[JonPaul]

@darrenwardle50 
Sounds good.
The comment about additional 'Spare Capacity' was in reference to the Best Practices Table 4.  With 46 drives you will have 2 enclosures. The recommendation is that UP to the maximum drives of 1 enclosure the default of 2x the largest drive is correct for spare capacity.  If you have an additional enclosure (for SFF drives 25-48) you should add an additional drive to the spare capacity,  with 46 drives spare capacity should be 3x the largest drive.
The 3 scenarios above Table 4 include:  During the initial creation of the disk group ... 'if the quantity of disks is greater than one enclosure.... Table 4 for guidance on target spare space capacity'.

For SSD automatic DRAIN, you are correct.
Due to the high cost of SSD drives, and this recommendation started with MSA 2040 ~2015 where SSDs were even more expensive, it is not common to purchase a spare SSD drive for rebuild when a drive faults.  This is typically ok as the fault rate of SSDs is much less than spinning drives.  But as a further safeguard if there is capacity available on other disk-groups the degraded SSD disk group will be DRAINed of all data.  Available capacity does not include 'spare capacity' on other disk-groups.

[JonPaul]

@ZVE-Minh 
Saw your post in email but appears to be gone.  Please post new topic if you need suggestions for your issue.

[ZVE-Minh]

Dear JonPaul  HPE PRO

- Our new MSA-2060 and Servers just bought, So out total System are having followings:
+ 01 MSA-2060 FC
+ 03 Gen10 HPE Server, which installed ESXi 7.03 on Each and using vCenter for center management.
- With 12 * HDD SAS SFF 2.4TB, which already created 01 Diskgroup, MSA-DP+ Level in POOL-A.

- Please help to classify following question:
+ With 12 HDD 2.4TB, should i Create:
* 01 MSA-DP+ Diskgroup in POOL-A using all Disks? or
* 02 DiskGroup in both Pool-A and POOL-B (01 DiskGroup with 06 HDDs) on Each Pool for better performance or FailOver function?
+ With one POOL-A with total 12 HDDs, should we create 01 large volumes or some small volumes for hosts access? which method have better performance?
+ Does MPIO (MultiPath Input Output) is a must and should I have to do on every ESXi host, which connect to same volumes?

Thank you for your excellent help
Trịnh Văn Minh

[JonPaul]

@ZVE-Minh 
Maximum performance is about finding the current bottleneck and then working to increase the throughput at that point.  The bottleneck could be at the host (Memory, Processor, HBA), in the fabric (A 'pinch' in the path such as going through a single FC interswitch link), in the array controller (exceeding the maximum throughput of a controller) or at the backend of the array, the disks (too few spindles/SSDs).
In your case the 12 spinning drives will be your bottleneck.  That is ok as long as the performance meets or exceeds your needs.  With only 12 drives the best bet for performance is to use them all together.  This is where MSA-DP+ disk-group will help.  All 12 drives are being used for data and you get the benefit of built-in spare capacity.  With a single MSA-DP+ disk-group you also set yourself up for adjustments in the future with disk-group expansion with 1 or more disks.
As a 'rule of thumb' unless you have a full enclosure (24 drives) or many SSDs, the single Pool option will be best for performance.  
Your other questions about the MPIO on ESXi will be found by default. It's a good idea to check the settings and adjust the entries if needed using the esxcli commands.  When sharing LUNs between hosts you must have the clustering software enabled on ESXi.

[ZVE-Minh]

Dear JonPaul (HPE Pro) 

Thanks for your feedback, so from your recommendation i already have a correct choice of POOL creation.

Trinh Van Minh

[ZVE-Minh]

@JonPaul 
When we access to MSA-2060 through SMU, we got a problem awarning "The system is currently unavailable." (login to both controllers through SMU).
- After trying to access to MSA-2060 via Console
+ We restart the Array by using command "restart mc both", After confirming restart both controllers
We got error message "The command was not recognized" even this is the correct systax and the system already inform to confirm if restart or not!

+ We then try to issue shutdown command "shutdown both" and got other error again "The MC is not ready, Wait a few seconds then retry the request"

- Could you please help to clasify the error and let me know method to fix this

thank you for you excellent support!

[JonPaul]

@ZVE-Minh 
Please create a new thread - this is not inline with the initial title

[ZVE-Minh]

Dear JonPaul  - HPE PRO

thanks for your support, we already post the other case on the system.

thank you

[TBC2]

Hello @ALL,

We are planning the following configuration:
3x ESXi HPE 385 G10 incl. FC card
1x MSA 2060
2x SAN Switch
4x HPE MSA 23TB SAS 12G Read Intensive SFF (2.5in) 6-pack SSD Bundle S2E45A = 24xSSD
Splunk Indexer data of the 3 servers should be stored on the MSA.
My question is: How much pool and volumes are best in terms of performance and maximum storage size.
I have already read numerous articles and also the best practice documentation, but I have not yet fully understood this.

Many thanks in advance

TBC2

[JonPaul]

@TBC2 
How about a referral to one more resource? 
https://ninjaonline.ext.hpe.com/
Going into the HPE MSA Sizer,   use the manual mode and plug in your hardware config for the 1 or 2 pools,  select an application (workload type)
The results should show you the expected performance for the pool.  You can even reset the workload parameters and re-plot the graphs.

As for volumes,  the pool is a holds all data for all volumes in the pool.  If you have a heavy sequetial workload you will likely want 1 or very few volumes.  Once you have multiple volumes the data on physical disk will start to look random no matter the workload.

But....  if you are using all SSDs and you have a lot of them then the total performance will be limited by the controller and not the backend storage.  ex. take your 24 SSDs and move that up to 36, no change in total throughput in a single pool.

Once you get to your needed/expected performance then it's a matter of determining what is more convenient for you to administer both in the MSA and in the application/host.  One pool, one volume or 2 Pools (max performance if you have the disks/ssds to support it) and tons of volumes to simplify the deployment.

And if you haven't found it yet, there is an MSA Gen6 Virtual Storage technical guide:  https://www.hpe.com/psnow/doc/a00103247enw
Which goes into the specifics of how the tiering, RAID (MSA-DP+) and other features of the MSA work.

[TBC2]

Thank you very much! That helps me a lot!

greetings from Germany

[AllincoSD]

Hello,

We are getting a MSA2060 Dual controller FC with 12 3.8TB SSD , and 3 DL380 with FC adapters.

I see some advice to use DP+ in one diskgroup , but would we bennefit if we dont'configure DP+ but two diskpools 6xRAID5 so we are using both controllers ?

Kind regards

jacco Dominicus

[JonPaul]

@AllincoSD 
For absolute best performance 2 Pools would be best.  But that is often not the only target.
Optimal performance -> 2 Pools  (utilize both controllers for I/O operations)
Simplicity of configuration -> 1 Pool 
Limit of capacity overhead -> 1 Pool (assuming MSA-DP+ or traditional RAID on both pools)
Ability to expand -> 1 Pool (assuming no additional drives purchased to do MSA-DP+ on both pools)
And those are just the points on the MSA side.  What would your application prefer?
One big Volume/LUN -> 1 Pool
Lots of little volumes -> ~2 Pools
Required division between application data/volumes -> 2 Pools (data will be interspersed through a Pool from multiple volumes)
The MSA performance is impressive for an entry storage array.  You can see performance expectations in  the Ninja Online tool:  https://ninjaonline.ext.hpe.com/

[AllincoSD]

@JonPaul.

Thanks for your advice.
Our dealer told me the best woud be 1 volume 12 ssd dp+ instead of 2 Pools rad5 because dp+ requiem minimum of 12 disks and that is the Total off ssds in the msa excluding 1 spare. 13 Total.

Previews msa's we installed at customer sites ( dual sas or iscsi) we always configured 2 Pools to utilize both Controllers.

That is why i woud like to know what to do, also used the ninja tool what gave double performance using 2 Pools instead of 1.
Only thing woud be that we loose capacity of 1 disk per volume if i use 2 Pools of 6 ssds

Load type will be hyperv csv 3 servers physical.
Vms will be db failovercluster and fileserver failovercluster
Rds farm with fslogix 8 vms
Senegal otter lessen demanding vms

Regards
Jacco dominicus