SimpliVity reached 100% space.

[Cleber Zorzi]

Hello,
I have a HPE SimpliVity solution with 02 nodes.
Storage reached 100% capacity and all virtual machines shut down (including vCenter).
I was able to install another vCenter and establish the connection to OVC.
I've already looked for backups (manual/automatic/deleted) and haven't found anything.
Below is the output of the command "dsv-balance-show -showHiveName -consumption -showNodeIp"

OmniStack host | Node GUID | Space I/O (Read / Write) | Space I/O |
+----------------+-------------------------------- ------+------------------------------+--------------- ------------------+
| 10.255.255.14 | 420c6a24-a501-20b0-0135-8043a62897c7 | 100% 1% ( 592 / 87 ) | 0B ( 22124 / 15057 ) |
| 10.255.255.13 | 420c7ec0-890f-0e14-f585-2a8865ea0211 | 100% 1% ( 592 / 87 ) | 0B (22124/15057)

When I try to connect a virtual machine the message "Insufficient Resource" appears.
I cannot, through node, remove a file or directory from a virtual machine. When I try to remove the message "rm: can't remove : Connection timed out" appears.

It seems that the datastores are in read mode and I can't do anything.

Can someone help me?

Thanks.

[s-]

Hi @Cleber Zorzi 

Simplivity solution internally uses different type of spaces to manage the user data like Index space, HDD space and SDD space . If any of the space utilization goes above 97%, system will be in read only state. 
It is 2 node cluster . We don't have option to perform  manual balancing and you have already tried to delete backups(manual/automatic) but didn't make any difference. I would request you to engage support ASAP so they could look at logs and suggest next POA.

Hope it helps.

Regards,
Jaipal

[Cleber Zorzi]

Hi @Jaipal,

Thanks for the answer.

Do you have any procedure that can reverse this situation?

As the environment is old, I do not have a support contract with HPE.

Thanks.

Cleber Zorzi

[s-]

Hi @Cleber Zorzi 

I understand the situation. It is difficult to tell what caused the situation. without looking into the logs. 

You may contact HPE support and get one time support by paying some amount. However, if still want to try by yourself you may try below steps.

1.Login to OVC using vcenter username and password OR svtcli account. Run the below command.

#sudo su
#source /var/tmp/build/bin/appsetup

2.Suspend all the backup policy which should not trigger any more backup and keep in suspended state until node became healthy.

#svt-policy-suspend --all

Once everything comes online you can resume the policy using below command.

#svt-policy-resume --all

3. SSH to ESXI using root and find if any VM has got  VMware snapshot. If delta file is there you will have to delete it and let it consolidate.

#find ./ -iname *delta*

4. Verify if any VM has got any manual or policy-based backup. if it is there, please try to delete manual/policy-based backup .
#svt-backup-show

5. If above steps don’t help. You may consider moving VMs from Simplivity cluster to other storage solution.

Hope it helps.

Regards,

Jaipal

[gustenar]

Also look at the actions recommended here: 

https://support.hpe.com/hpesc/public/docDisplay?docId=sd00004279en_us&page=GUID-BAEDE61C-9245-4243-9798-03D9CADAE47D.html

[support_s]

Hello,

Let us know if you were able to resolve the issue.

If you have no further query, and you are satisfied with the answer then kindly mark the topic as Solved so that it is helpful for all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.