Simplvity OVC VM out of CPU resources

[steez]

Hello everyone,

I have noticed high OVC VM CPU usage in our cluster.

In moments where our DB VM start to actively process data OVC cpu usage gets very high, sometimes even staying at 100% for the time DB data process are happening.
Due to that VM performance is impacted as the OVC cannot handle the load. It takes longer for some processes to be completed than it was before on regular servers (same configuration).

At peak time we've noticed about 0.5-0.7GB/s of throughput (replication); ~8K of IOPS and about ~30-40ms of latency when OVC CPU gets fully used. Are those numbers okay? When in "normal" daily use performance is okay - 100-150MB/s Throughput, 2K IOPS at 15ms.

One cuplrit found was internal data copy for "backups" that was generating a lot of IO for no real use (as the "backup" still gets deduped and no real copy is made). However data copy between VMs may still occur. The question is - is this the most performance we can get? Is there a way how to assign more CPU resources for the OVC VMs (they already have 6CPUs).

We're on 4.0.1 1+1 cluster, DB VMs are 16K. We have applied SAP best practices - Paravirtual adapters, VMXnet3 network. Even CPU and RAM reservation, disk shares high.

Edit: Also, there are snapshots for each of OVC VM. We will remove them at earliest convenience. 

Any ideas? More information can be provided if necessary.

[Rajini_Saini]

HI Steez,

Thank you for choosing HPE.

The snapshot can be causing this issue. We suggest to immediately log a SimpliVity support case for the same and get them removed.

regards,
Rajini Saini

[steez]

Hello and thanks for the quick answer Rajini.

We will log the case with HPE. I will post whether the performance has improved.

[tonymcmillan]

Are you saying the snapshot on the OVC is causing a performance issue on this system?

[AnkiN]

The snapshot is created if we have retried failed deployment option with 4.0.1.
This snapshot size usually grow by the time if it's left out post successful deployment. It's advisable to delete the snapshot of OVC if present. Performance issues can occur due to increased size, However we need to check the logs for better understanding.

[steez]

Hello Tony,

We're not sure whether it is the snapshot that causes performance issues, we're still resolving it with the support. Once I get any news i'll post them here.

[steez]

Hello Ankin, thank you very much for your input.

For us it seems like the snapshot size has not increased, VMware states that the size is ~5GB (however could not be correct).

As for deployment, Im not sure if it failed at any point, it looks like it did. We noticed the presence of the snapshot but didn't think much of it because if it was automatically created it probably should be there.

As for removing it - The snapshot has not been removed yet, we wanted to do it with the help of HPE because the OVC VMs has to be turned off for this procedure. HPE Support ir still investigating both AHS and SVT Support capture, i'll keep you updated once any news come.

[tonymcmillan]

We have deployed numerous SimpliVity nodes and in every instance, a snapshot was left on the OVC. This occured with versions 4.01 and 4.01. U1.

We did not see this behavior with version 3.7x. Not sure about 4.0 as we haven't done that version.

We always remove the snapshots before putting the nodes into production. Hopefully, this issue will be addressed in future versions.