The cluster 'xxxxxxx' is becoming full

[Mike Molyneux_3]

Hi 

I've inherit support of a HPE Storevirtual centralized mangement console, which I've no experience of and I've got an alert and really struggling to understand it!

There's one management group and 1 cluster. 2 storage systems and one volume. Volume shows Network RAID-10 and a reported size of 5.5TB and consumed space 10.99TB, thin provisioning.

I'm seeing this alert -

The cluster XXXXXXX is becoming full. The cluster 'XXXXXXXXX' space utilization is at 92.20% of capacity. This utilization value exceeds 80.00%. As the cluster approaches being full, scheduled snapshot or remote snapshot creation may fail due to a lack of space, application writes may fail on thinly provisioned volumes when the cluster is full, and other tasks that require cluster space may be impacted. Some actions to reduce capacity utilization on the SAN are to delete volumes and snapshots, reduce snapshot retention, add nodes to the storage cluster, use thin provisioning, move volumes to other clusters, etc.

If I click the volume is shows just 1 volume and zero snapshots and it's assigned to a Windows server.

If I open explorer on that server I see the iSCSI drive and it's a 5.5TB drive, with 2.25TB free.

So I'm confused what it means when it says 92% utilised, 952GB free. Probably some really obvious that I'm missing here, but as I say, never used these before!

Thanks in advance.

[Rachna-K]

@Mike Molyneux_3 

If I understand correctly there is only one Virtual Machine which is using the StoreVirtual Storage and that one has  enough space. If so may be the data on the Virtual Machine was deleted and space is not reclaimed.

You might have to use third party tools to reclaim the space. Since you said its Windows based VM, Microsoft has Sdelete Utility to free up space.

[Mike Molyneux_3]

HI Rachna, thanks for the reply.

Does that mean that the StoreVirtual CMC doesn't take into account the fact that data has been deleted from the Windows server? That doesn't sound likely to me, as the data stored on a volume goes up and down all the time.

Mike

[Rachna-K]

The issue is at the Virtual Machine level.

If the Space gets released from there, StoreVirtual Volume is able to reclaim it. 

[Mike Molyneux_3]

Thanks again for your reply.

If I click the management group it says space reclamation isn't enabled - is that the cause of this issue?

If I right click and choose to enable it says it can't as it's over 90%. There's no volumes or snapshots I can remove.

Thanks

[Rachna-K]

Thank you for bringing that up. Yes Space Reclamation should be enabled. There are some pre-requiste to enable it and also Space should be lesser that 90%.

You can refer the StoreVirtual User Guide to understand how Space Reclamation works and how it depends on the T10 unmap SCSI standard. Below is the link for the same.

https://support.hpe.com/hpesc/public/docDisplay?docId=c05049834&docLocale=en_US

If you need further assistance with Space Reclamation this might need a Support Ticket. 

[Mike Molyneux_3]

Thanks.

So with this being disabled is that the cause of the space issue?

I can't enable it as it's over 90%. If I need to log a support ticket to get help, do I use the serial number of one of the storage system 4530 SANs?

Thanks

[Rachna-K]

Yes. You can use the Serial Number to raise a ticket with the Support. 

[Mike Molyneux_3]

Thanks @Rachna-K .

I tried with the serial number but get a 'Support for this product has expired'. 

So now I'm a bit stuck!

So it's showing as having 952GB free on the SAN. Does that mean if the server that has this SAN as the iSCSI target tries to write more than 952GB of data to the volume then the SAN will be 100% full?

Thanks

[Rachna-K]

@Mike Molyneux_3 

StoreVirtual is End of Support Life since 2022 and any support is on best effort basis and would be chargeable.

VMs would usually stop responding once the Custer Space reaches 95%

[Assen]

Hi Mike,

I've been following the discussion and have a couple of ideas that might help you address the issue. Here's what you could try:

Step 1: Backup Your VM
First and foremost, make sure to back up your VM. Given the 2.5TB size you mentioned, it should be manageable. Once backed up, you can proceed to delete the current volume. Then, recreate it as a thin-provisioned NR10 volume with UNMAP enabled. Finally, restore the VM onto the new volume.

Step 2: Adjust Thresholds for Space Reclamation
Alternatively, you can attempt to enable space reclamation by adjusting the thresholds temporarily:

Log into StoreVirtual CLIQ:

Use SSH on port 16022 to access the node.
Log in with the administrator credentials you use for CMC.
Modify Thresholds for Space Reclamation:

For versions 12.0 and 12.5, use:
modifyCluster clustername=cluster1 alarmthreshold=98

For versions 12.6 and 12.7, use:
modifyCluster clustername=cluster1 SpaceReclaimCrit=98

Note: After reclaiming the space, revert the thresholds back to 95%:

For 12.0 and 12.5:
modifyCluster clustername=cluster1 warnthreshold=95

For 12.6 and 12.7:
modifyCluster clustername=cluster1 SpaceReclaimWarn=95

Replace cluster1 with the actual name of your cluster in the management group. Adjusting the thresholds may not be a guaranteed fix, but it’s definitely worth trying.

Hope this helps!

Cheers,
Assen

[Mike Molyneux_3]

Hi @Assen 

Thanks for the reply. The SAN volume is purely a data volume that's mapped as a drive on a physical server. 

I've just found the CLI guide and the parameters for modifycluster you mention suggest these are just for alarms and not the limit for being able to turn on space reclamation...?

Mike

[Mike Molyneux_3]

Bump. Anyone else got any thoughts on this?

[Assen]

I was wondering if you have tried my suggestio to increase the thresholds, enable unmap and free up space? Don't forget to set them back to default after you are done with freeing up space