StoreVirtual Storage
cancel
Showing results for 
Search instead for 
Did you mean: 

upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

 
expert34
Occasional Visitor

upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

Hello,

the cluster has a total capacity of 40TB, provisioned 36TB (full), free 4TB

The last updates where always successful, but the latest to 12.7 fails immediately with the message
"Cannot perform the upgrade because cluster xxx is reaching its total capacity and has exceeded the lmit of 90% full"

Is there any way to perform the upgrade without reducing or cancelling volumes? Or do I have to change one volume to thin provisioning to free up space?

best regards,

 

18 REPLIES
testmado662
Advisor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

Hello,

I did solution you gave. I had few Cluster in Thick provisionning and I change them to thin.

First provisionning went over 90% and I wasn't able to upgade. If used size is under provisionning, you will have free space and be able to upgrade. I understand that Thin can be used safely if you set a max size under the 90% of global space. You couldn't add space over the max size you configure.

About this new rule, We can't use over 90% of StoreVirtual global size. Have you send a claim to HPE ?

I did it and I know that other clents did it. Current HPE position is to force end user to free space.

 

Regards

Johannes_we
Advisor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

It seems HPE is trying to make a very good product very poor.

SVprodmgr
Frequent Advisor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

We plan to release a new version of the CMC for 12.7 that will allow an upgrade from 12.6. to 12.7 in support mode.  In support mode, you can upgrade from 12.6 to 12.7 even if the free space threshold is exceeded.  You can also adjust the threshold levels on the cluster configuration in CMC.  We are planning to release the new version of the CMC by the end of August.

Thank you for the comments about the threshold changes.

Amy Mitchell

StoreVirtual Product Manager

I'm an HPE employee working in product management
Richard7
Advisor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

Hello Amy,

The version of CMC you refer to, has that been made available now? I ask since I believe that I am using the most up to date version, but I cannot seem to find a Support Mode or adjust the threshold levels on my cluster.

My 3-node VSA cluster is 98% full, so I am unable to upgrade from 12.5 to 12.7.

Would selecting thin-provisioning be a workaround for this? Will changing this setting on a production cluster have any adverse effects? If the volume were thin provisioned, I expect it would only be around 80% full.

Many thanks,
Richard

 

oikjn
Honored Contributor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

as long as you know how to manage your capacity, there is no performance benefit to running a LUN as fully provisioned instead of thin provisioned.  WIth thin provisioning you can use snapshots to assist with backup activities as well.

Richard7
Advisor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

Ok, thanks for that Oikjn.

So, I've switched the LUN to thin provisioning. This didn't free up as much space as I'd hoped, only a few percent so we're still over the 90% threshold and can't run the upgrade.
Is there much else I can do, there should only be enough data on there to account for 75 - 80% or so. Would the space reclamation feature help us in this case? It isn't currently enabled. 

Thanks,

oikjn
Honored Contributor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

space reclaimation will definitely help if your reported consumed space in the LUN is much higher than the used space reported on the host(s).

Keep in mind that in order for space reclaimation to work you need to enable to function on the cluster AND on the initiatiors.  Just enabling it on the SAN will not get it to function on exsiting LUNs until the initiators connecting to those LUNs are aware of the new support for the trim function and start sending trim commands to the SAN.  New LUNs will have the feature right away, but ones that are currently connected will need to "refreshed" to inform them of their new capabilities.

I don't remember what the proceedure is for esx since I"m a hyper-v shop, but for windows I found it was easier to be sure it was working by disconnecting the LUN and reconnecting it from each host.  I believe it was reported that a disk scan in disk manager SHOULD work too, but I couldn't find an obvious way to report quickly if trim/unmap was active on the luns and reconnecting the LUN wasn't a big deal for most of my LUNs at the time.

Once that is done, you may need to trigger a disk optimization on the initiator for it to go through and inform the LUN that the free space is available for reclaimation.  Once the initiators actually send those commands, the LUNs will report back on the newly available space virtually immediately.

Richard7
Advisor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

Hi Oikjn,

Thanks for posting this information, very interesting.

Unfortunately, when I try and enable space reclamiation on the SAN it tells me I cannot as it's over the 90% full limit. It tells me to delete necessary volumes and snapshots.

 

Richard

oikjn
Honored Contributor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

wow.  that is annoying.  and at 36 of 40TB, you are basically at 90% too.  I don't know why they have that 90% requirement for either action and not some minimum space requirement.  Anyway, short of opening a support case to have them address it, hopefully you can move off enough data to get under the 90% to allow space reclaimation to be enabled and then recover even more space before doing the update to 12.7.  

Christato71
Visitor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

Hello, 

Just checking to see if there are any updates on this. I am having this same exact issue. 

Richard7
Advisor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

Hi Oikjn, I'm going to try and open a case with HP. If they can't resolve, then I'll be migrating the data to another server for the duration of the upgrade. That will take a fair bit of time so I hope we can avoid it! 

Richard7
Advisor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

It was recommended that we run the CMC in support mode, which allow us to run the upgrades on each node individually.

We're trying this soon, so I'll report back when successful.

Richard7
Advisor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

Running the upgrades in support mode worked fine. It doesn't resolve the low space issue of course, so I expect when the next software update comes out we'll need to do the same thing.
Thanks.

Richard

oikjn
Honored Contributor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

glad to hear you updated ok.  are you giving up on enabling space reclaimation?  I don't know how many LUNs you have, but assuming there are a few, that should clear up a lot of potential free space so you fall well under that 90% limit.

Christato71
Visitor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

How do you run the update in support mode? Is it an option in the CMC?

Richard7
Advisor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

Oikjin, yeah I'm not going to worry about space reclaimation for now. We just have the one big LUN taking up all of the space, otherwise I'd probably have sone some rearranging of data

Christato71, You copy these 3 lines into the preferences.txt file found in C:\Users\UserName\.storage_system on the system that has CMC installed.

CmcSystemPreference.supportMode=True
CmcUpgradePreference.useOldUpgrades=true
CmcUpgradePreference.advancedUpgradeMode=true

Careful though as the CMC won't manage quorum for you during the upgrades, so you need to make sure each node is back up and running and the arrary is synched before moving to the next one.

 

Christato71
Visitor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

Richard7, Thanks for the information! Will I need to apply one patch at a time to each node?

Richard7
Advisor

Re: upgrade from 12.6 to 12.7 fails (cluster reaching its total capacity)

Yes, install them one at a time. It's fairly time consuming, but it's worth taking time to be careful as it won't check to make sure that quorum is maintained during the upgrades, you need to do that yourself.