MSA 1050 SAN Storage Issue (Degraded performance in Virtual Pool A, backups failing)

[Chris-M]

I currently have a problem that I desperately need help with. I am a little new to managing SAN's (though I've dabbled in it before, I've never encountered something like this which is a problem inherited due to a lack of following best practices from the former IT). Here's what's going on:

I have two storage pools each set at 2395GB. Both consist of 6, 600GB 10k SAS drives.

Storage A has 2 volumes (volumes 1 and 3 sized 1999GB and 2047GB respectively), with a total allocation of 2346 GB. The unallocated space in this pool is 1701GB. (This is the Storage Pool that is having problems, note the pool size vs allocated size with only ~49GB of space left in the pool, yet much more unallocated space that extends beyond the virtual pool size.)

Storage B has 1 volume (volume 2 sized 1999GB), with a total allocation of 958GB. The unallocated space in this pool is 1041GB.

Both of these storage pools are connected to an ESXi cluster of 2 servers. both with read/write to the 3 LUN's 0, 1, and 2 for volumes 1, 2, and 3 respectively. Volumes 1 and 2 are set at Performance Tier Affinity, and 3 is set at No-Affinity.

In VMWare, there's a pretty large discrepancy between the size of available space in the respective data stores vs. the volumes that they are on (in the SAN). I don't know if that has anything to do with this at all, but it's worth mentioning.

We noticed problems starting to occur on Friday when backups started failing for our VM's. We checked on the SAN after contacting support through the backup vendor and they indicated there may be some storage issues. We found that Storage Pool A is very overcommitted (I believe that's the term for this), as you can see above we're incredibly close (within 49GB of completely full). This was at 171 GB free yesterday, and even after deleting a VM that was about 175GB the allocated space on the SAN has increased!

I'm not too sure of the best way to go about fixing this. I gather that the only way to increase the pool size is to add a disk group to it, and given the two disk groups available are already assigned to the two virtual pools A and B I have to add more logical storage (to make another disk group out of). Am I on the right track? Is there any other way to go about this given my situation? I suppose what I'm trying to do is increase the pool size of Pool A and need to know what my options are (or if there's some other way to fix this)?

Thanks in advance for your help, sorry for the long post, I wasn't sure how to explain the situation any better with more brevity. Please let me know if I can provide any more information to help you help me!

[support_s]

System recommended content:

1. HPE MSA 1050 Storage - Troubleshooting

2. HPE MSA 1050 SAN Storage - Overview

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[ArunKKR · Accepted solution]

Hi,

In over commit enabled Pools (which is the default setting) once the space usage exceeds high threshold the controller cache Write Back Status moves to disabled state resulting in performance degradation.

From the issue description shared so far this seems to be a case of pool exceeding high threshold.

You would need to create more free pace in Pool by adding a new disk group under Pool A. This would need the addition of new hard disks to the array.

You could also try moving data out of Pool A volumes to a different storage and then performing space reclamation using the unmap command listed in below advisory:

https://kb.vmware.com/s/article/2057513

Please share the output of below commands:

show pools

show disk-groups

show volumes

show cache-parameters

[Chris-M]

Thanks for the info. The issue was indeed the degradation due to being above the high threshold. I contacted VMWare support who was able to assist me (much like you mentioned) with the unmap command and then rescan the disks. This cleared a decent amount of space and allowed backups to resume normally.

Thank you for reinforcing that this was the right way to go - I was unfamiliar with what unmap does, and if it was safe to do in a live environment, but now I know that it's safe and a good thing for us to do. I was under the impression that the data is set to automatically reclaim zeroed-out space, but I guess that wasn't happening.

We will additionally be looking at adding another disk group (but that is very expensive) and using VMotion to move some VMs from one pool to another while changing from thick to thin-provisioned (which should ultimately help us fix this in a more long-term way). 

Thank you for your response!