Serious Wrong "Overprovisioned" Error on VSA 2014

[banaie]

Dear Friends,

I have faced a serious problem with VSA 2014. It's provisioned on two BL460 servers that are attached to D2220 DAS.

Each storage has about 7TB of storage & I'm using them in a cluster RAID 0 (14TB of real storage). I have created a thin provisioned and presented to vmware as 50TB.

The problem is I have been using this solution for three months & I never go above 4TB on the storage.

But, as vmware shows I haven't used a lot & now even after removing a lot of files vmware shows that it's almost empty:

But, CMC shows it's full and it's using 14TB!!

Now, the storage has stopped working & I can't copy any file or use it in any way. I really need an urgent help.

I have migrated all the important virtual machines but, the overprovisioned error still persists. I even pachted the system to the latest updates.

Please, help me on this problem.

Thanks

[Mr_NKC]

 There could be a possibility that the changes made to the TP volume on VMware haven't got refreshed at the VSA software end. May be you could manually delete the volume from CMC. 

[banaie]

Thanks a lot for your reply. I can do that for this time. Because, I have saved all my important data. However, this solution is becomming more & more operational in our servers. Then, I'm worried that this problem rise again in a bad situation where I can't do anything.

Is there any solution for me to solve this problem without deleting the volume? Can't I force the storage to refresh its data or validate the data again to solve the problem?

Thanks a lot

[oikjn · Accepted solution]

you are going to have to delete.  THere is no SCSI UNMAP feature to the SAN so once thin space is claimed, it cannot be reclaimed other than by deleting the LUN.

If you have the data backed up and it really is mostly empty, maybe you could roll the dice and resize the LUN to shrink it down and hope it doens't take out part of your data.  Then you can create a new LUN and move your data to that one.

For the love of god, use NR10 for any production data.  Having production data on a Raid0 anything is just irresponsible.  It appears that you really don't need all the raw storage so I see no reason why you can't use NR10.

In the future, you should consider many smaller LUNs instead of a single large LUN as that limits the affect of a situation like this or one where the filesystem gets corrupted. 

I'm not an ESX guy so I can't say for sure, but I"m pretty sure there is a format style you can chose that forces writes to the disk and that will end up expanding thin provision LUNs...  don't pick that one ;)

When I overprovision my SAN, I generally create one full provisioned LUN called "emergency space" that I leave with enough space for me to be able to delete to regain my storage functionality in case I'm not paying attention and the SAN fills to 100% and then by deleting it I can get production going again and deal with the response (spool up a VSA or buy another node or trim data).

[banaie]

Thanks a lot for your enlightening reply. However, I thought it's more easy to present the storage as a single LUN & manage it in the vmware.

Then, you say I should try using different LUNs for different data. Is it the correct aproach.

About the RAID 0, as I have raided all the hard disk physically on the server and I presented it to the server as a single storge, I thought I don't need to re-RAID it in the VSA, too. Should I present hard disks (9*900GB & 3*250GB) to the server separately & try to raid them in the VSA?

Many thanks again

[oikjn]

the VSA software doesn't do raid and only does Raid0 for disks presented to it, so you should continue to provide hardware raid volumes to each VSA for it to use as storage.  This will provide you with disk level redundancy.  What you are missing is the node level redundancy which is the biggest selling point for the StoreVirtual SAN.  When you have a SAN LUN configured for NR0, the LUN requires ALL nodes to be online and functioning to provide service to that LUN, when you have a NR10 LUN, at least one (sometimes more depending on the situation) Node can go offline and the LUN will keep functioning.  This allows for online software updates to be non-disruptive.

CMC should be shouting at you giving you warnings that you are using NR0 and to use at your own risk.

your environment isn't THAT big...  I would suggest that you limit each LUN to 2TB in NR10, or something like 2.3TB or 1.7TB if you want to make all LUNs equal size and have some spare space.

Either you haven't moved all your data over, of you have some serious room to grow with your space.  If you have room to grow, I would simply create three 1TB thin NR10 LUNs and leave the rest of the space unprovisioned for future use (you might want to provide direct iSCSI access to a guest VM for something like SQL clustering or file server clustering in the future.

I don't know if its possible, but if you can afford the downtime on the SAN, your environment is small enought that I would suggest you delete it and set it up with a better naming convention.  VSA-1 for your managment group AND cluster is just confusing if you ever add a 2nd cluster.  name the management group something like VSA-SAN or VSA-MGT-GRP or COMPANY-SAN-1.  Unfortunately this can't be renamed so you either have to delete the group or leave it as is.  The cluster can be renamed and at the least it should.  Call it something like VSA-Cluster-1 or Cluster-1 or anything other than the same name as your management group.  At my company we used COMPANY-SAN -> COMPANY-Cluster-# -> COMPANY-VSA-#

As far as disk Raid and then network raid on top sounding really inefficient... it is.  If you use NR10 and R10, you end up with a 1:4 ratio of usable to Raw space which is just aweful, but what you get for that is something that is just 100% bulletproof, so your question becomes, what costs more... a couple disks and a little power or system downtime and for most places with <50TB usable space required, the answer is easy...  AVAILABILITY.  The answer is the same over 50TB, but then you can get that availability through other SAN solutions that might be more efficient than this one.

[banaie]

Thanks again for your precious points.

I was thinking that presenting more LUNs to vmware might create a performance issue. Then, I was trying to present it as a whole LUN & try to manage it on different vmware clusters that has access to the same storage. I thought it might help me avoid wasting unused storage that would be left on each LUN.

 Moreover, one of my friends told me that Thin Provisioning does not work correctly in the VSA (and it's the cause of my problem with the storage). Then, you don't supoort this argument & say that it's perfectly safe to use this feature?

[oikjn]

it works and it doesn't.  The problem is that the SAN does NOT support space reclaimation, so once a thin volume expands, its exapanded forever.  If you do a complete secure format to a thin LUN it will expand to 100% capacity and stay that way.  So if your definition of thin provisioning requires space reclaimation, then yes it is broken.  My biggest problem with this SAN is that is lacks this feature, but its something you can work around once you know its limitation and was definitely a contributing factor in your problem here.

That said, you SHOULD ALWAYS use thin provisioned volumes on this SAN because it is required for creating think provisioned snapshots.  What you should be careful of is your overprovisioning rate.  To present 100TB of space when you only have 10TB unless you know your usage pattern really welll and how it interacts with the SAN because as you saw, you can quickly "run out of space" even when you should have plenty of space available.  There is generally no good need to have anything so highly overprovisioned unless you are talking about a huge number of very small LUNs.

Your intuition about consolidating LUNs is actually backwards... the more LUNs you make, the more connections the nodes can make to the SAN so the more load balancing they can do...  of course this assumes you are within the limitations of your switch capacity and iSCSI connection limits.  The drawback you mention about wasted space for free space on each LUN is what you should be able to make up with thin provisioning.   If you do this I would suggest you also create a single thick LUN that is something like 50GB in size just as an emergency space that you can quickly delete if you do manage to fill your SAN again.  I would also suggest that you design your LUN sizes based on the assumption that you would size them as if you were using FULL provisioning and oversize them only as little as required for your application, then make sure that if you leave at least enough available space on the SAN to cover one LUN filling up its free space and a little extra.  For example, lets say you have 10 LUNs at 100GB in size that you want to keep with 30GB free space in them.  That means you have a total of 700GB of expected data and 300GB of expected free space.  Given what I said above, you should be able to get away with something like 750GB of usable storage which would be overprovisioned to present like 1000GB.  This assumes you get some experiance with what causes and doesn't cause LUNs to expand... like don't try a secure erase to a LUN as that will quickly fill it to max capacity on the SAN.

[Bart_Heungens]

Hi,

Know that HP announced at VMworld recently that with the next update on the Lefthand OS software space reclamation will be in there... Release date not specified, I assume somewhere beginning of next year? To be continued...

Kr,

Bart

[oikjn]

I'd love that if it were true bart.  Hopefully they are better at holding their schedule than they were for v10.5... or v11... or v11.5.  I think they average 9 months late on their projections :(

I'm at a coin flip on if I'm going to renew support for the upcomming year or try and migrate to something else as even the built in iscsi target in windows is getting close to feature pairity!