SimpliVity Virtual Controller using swap memory

[sergeisokolov]

Greetings!

After upgrading OmniStack to 3.7.10 U1, a couple of legacy hosts are reporting this error.

In the 3.7.10 U1 Event Reference PDF there is the following description:

SimpliVity Virtual Controller using swap memory, error.
Reports when the SimpliVity Virtual Controller is using over 2Mb of swap memory. This indicates that internal system memory is not adequate to handle current processing.

SimpliVity Virtual Controller using swap memory, warning.
Reports when the SimpliVity Virtual Controller is using at least 256Kb of swap space. This indicates that internal system memory is not adequate to handle current processing.

This kind of warning/error was not mentioned in the 3.7.10 Event Reference, so this seems to be a new one.

Does it mean that we should allocate more RAM to the OmniStack Virtual Controller? Is it generally a good idea? The default RAM size for CN-3000/3400 hosts was 100GB. But it's 114GB on HPE380, for example.

[AnkiN]

Hi @sergeisokolov ,

Do not change the memory allocation for the OVC, the best action would be trying to restart the svtfs service and checking if the alert comes back or not.

In case if it does not help then please log a support case providing the support capture bunddle for the analysis.

[Randolfini]

Hi Sergie,

I just upgrade from 3.7.10 to 3.7.10 U1 on my legacy hosts as well and immediately one of the nodes reported a SimpliVity Virtual Controller using swap memory, error.. I put the effected host in maintenance mode, safely shutdown the OVC, then powered it back on and am now monitoring.

If you open a ticket, and you don't mind sharing, please let us know what the resolution is.

Thanks!

[sergeisokolov]

Hi! 

I won't be able to create a support ticket as those legacy hosts are not under support contract anymore.

But what my observation was, is that it seems to be related to OVC load. In that cluster we have one VM that creates a lot of IOPS most of the time - like 2000...5000 (that's more than all the other VMs that reside on that host) and the "swap memory" error occurs only on the host were this particular VM is currently running. So when I move the VM to Host-B, then its OVC starts reporting that error and if I move it back to Host-A, then the error starts popping up there. 

If the error reappears to you, you can try checking cluster performance top chart to see which VMs are particulary IOPS-intensive and try relocating those, to see if the error message travels with those as well.

While the old 3.7.10 did not list this error message in the Event Reference PDF, probably this swapping issue was present before, but just wasn't reported by the OVC. 

[Leventilkbahar]

Hello , what is the result?

can you see error message again?

[sergeisokolov]

Hello!

Yes, those error messages are randomly appearing on CN-3000/3400 OmniCubes.

This never happened with OmniStack versions up to 3.7.10 and started occuring with the latest legacy update 3.7.10 U1.

If we trust the Events Reference Guide, then "com.simplivity.event.control.phys.capacity.swap.usage.error" indicates an over-burdened system. So, with OmniStack 3.7.10 the systems were not over-burdened, but with 3.7.10 U1 immediately became over-burdened. One of the clusters that is affected has ~40% utilization, has only daily backups and file server type workloads. So, nothing I/O intensive.

Suggested action is to stop and restart the svtfs service on the Virtual Controller. That supposedly brings the swap space for svtfs back to zero.

Well, I even restarted the hosts several times but this doesn't change anything. Or the effect does not last for long. I guess those legacy hosts have to keep living with those errors until they are completely replaced at some point in the future.