Slow KVM Native Backup and Failures.

[DavidJP1]

Hello,

I am running into a couple of issues when attempting to work with the KVM Native Backups within VME. The first and most glaring is near instantaneous failures when running a backup job. The 2nd is when they do sucessfully run they are only coming through with ~7-8MiB/s transfer rate. 

The lab environment is a 3 Node Synergy Solution connected to a Alletra MP B10000 via ISCSI with the plugin running HVM Cluster 1.2 on top of Ubuntu 24.04. I also have pulled in a G10 Alletra 6000 DHCI as a VMWare Cloud. I have 3 Backup Targets configured - StoreOnce VSA (running on the DHCI Cloud), Ubuntu NFSv3 (VM running in VME) and Ubuntu S3 /MinIO (VM running in VME). All connections are 10Gb/s. These failures and poor speeds happen on all backup targets. 

Windows VMs have never been sucessful, Ubuntu VMs sometimes are sucessful, Morpheus Community Edition running as a VM is usually sucessful but slow. VME Manager is usually sucessful but slow. I have done a write speed test to the VSA from the Ubuntu Server VME is running on and was able to get well over 100MiB/s writes to the StoreOnce VSA so the networking should be sound. 

When those jobs do fail they sometimes leave unremovable task countdown bar that is stuck at 0:01 remaining in the Instance. 

1. What can do to make these jobs run sucessfully?
2. Is there anything I can do to improve this backup performance?
3. Is there a way to remove those jobs stuck at :01?

[Moto_PK]

Trying to share the Possible Causes & Fixes:

1. Permissions & Libvirt Integration
Ensure libvirt and qemu-guest-agent are properly installed and running in all guest VMs.

For Windows VMs, check that:

Integration services (or qemu-guest-agent equivalent) are installed.

Volume Shadow Copy Service (VSS) is functioning correctly.

b. Snapshot Mechanism
KVM Native Backups often rely on libvirt internal snapshots.

Ensure the VMs are disk image-based (qcow2/raw) and not using passthrough LUNs or non-snapshotable storage.

For failures, examine the logs:
journalctl -u vme-backup.service or vme-backup-agent.service (depending on the implementation).

 c. LVM/Overlay Issues
If VMs use LVM or Docker-like overlays, snapshot creation might fail unless tuned to support these.

2. Performance Bottleneck (~7–8 MiB/s on a 10Gb/s link)
This is clearly suboptimal. Likely root causes:

a. iSCSI Multipathing & Queue Depth
Verify if iSCSI multipathing is configured correctly on Ubuntu (use multipath -ll).

Tune iSCSI queue depth and sysctl settings (e.g., net.core.rmem_max, wmem_max, etc.).

 b. Backup Transport Mode
If VME is reading via libvirt/qemu agent, it might use single-threaded block copy methods. Look into whether VME supports:

nbdkit, qemu-nbd, or other offload mechanisms.

Parallel stream tuning (e.g., multiple threads, compression settings).

 c. CPU Throttling / I/O Wait
Monitor top, htop, and iostat during backups. Are you seeing high I/O wait or CPU steal time?

d. Backup Target Bottlenecks
Even if StoreOnce VSA shows good raw write speeds, the data ingest path may be affected:

Dedupe/compression overhead.

StoreOnce throttle policy (verify in StoreOnce GUI).

MinIO (S3) — tune minio server options like --storage-class and ensure parallel PUTs are enabled.

NFSv3 — confirm it's using async exports and not being throttled by disk IOPS.

3. Stuck Backup Jobs at 0:01
This is a known problem in many backup systems when:

Cleanup scripts fail.

Snapshot remains locked.

Task is orphaned in the UI.

Fixes & Workarounds:
Restart the backup daemon:

sudo systemctl restart vme-manager
sudo systemctl restart vme-backup.service
If that doesn’t help, look for the PID of the stuck job (ps aux | grep backup) and manually kill it.

If stuck VM snapshot exists, manually delete with virsh snapshot-delete.

[support_s]

Hello,

Let us know if you were able to resolve the issue.

If you are satisfied with the answers then kindly click the "Accept As Solution" button for the most helpful response so that it is beneficial to all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".