Slow KVM Native Backup and Failures.

[DavidJP1]

Hello,

I am running into a couple of issues when attempting to work with the KVM Native Backups within VME. The first and most glaring is near instantaneous failures when running a backup job. The 2nd is when they do sucessfully run they are only coming through with ~7-8MiB/s transfer rate. 

The lab environment is a 3 Node Synergy Solution connected to a Alletra MP B10000 via ISCSI with the plugin running HVM Cluster 1.2 on top of Ubuntu 24.04. I also have pulled in a G10 Alletra 6000 DHCI as a VMWare Cloud. I have 3 Backup Targets configured - StoreOnce VSA (running on the DHCI Cloud), Ubuntu NFSv3 (VM running in VME) and Ubuntu S3 /MinIO (VM running in VME). All connections are 10Gb/s. These failures and poor speeds happen on all backup targets. 

Windows VMs have never been sucessful, Ubuntu VMs sometimes are sucessful, Morpheus Community Edition running as a VM is usually sucessful but slow. VME Manager is usually sucessful but slow. I have done a write speed test to the VSA from the Ubuntu Server VME is running on and was able to get well over 100MiB/s writes to the StoreOnce VSA so the networking should be sound. 

When those jobs do fail they sometimes leave unremovable task countdown bar that is stuck at 0:01 remaining in the Instance. 

1. What can do to make these jobs run sucessfully?
2. Is there anything I can do to improve this backup performance?
3. Is there a way to remove those jobs stuck at :01?

[Moto_PK]

Trying to share the Possible Causes & Fixes:

1. Permissions & Libvirt Integration
Ensure libvirt and qemu-guest-agent are properly installed and running in all guest VMs.

For Windows VMs, check that:

Integration services (or qemu-guest-agent equivalent) are installed.

Volume Shadow Copy Service (VSS) is functioning correctly.

b. Snapshot Mechanism
KVM Native Backups often rely on libvirt internal snapshots.

Ensure the VMs are disk image-based (qcow2/raw) and not using passthrough LUNs or non-snapshotable storage.

For failures, examine the logs:
journalctl -u vme-backup.service or vme-backup-agent.service (depending on the implementation).

 c. LVM/Overlay Issues
If VMs use LVM or Docker-like overlays, snapshot creation might fail unless tuned to support these.

2. Performance Bottleneck (~7–8 MiB/s on a 10Gb/s link)
This is clearly suboptimal. Likely root causes:

a. iSCSI Multipathing & Queue Depth
Verify if iSCSI multipathing is configured correctly on Ubuntu (use multipath -ll).

Tune iSCSI queue depth and sysctl settings (e.g., net.core.rmem_max, wmem_max, etc.).

 b. Backup Transport Mode
If VME is reading via libvirt/qemu agent, it might use single-threaded block copy methods. Look into whether VME supports:

nbdkit, qemu-nbd, or other offload mechanisms.

Parallel stream tuning (e.g., multiple threads, compression settings).

 c. CPU Throttling / I/O Wait
Monitor top, htop, and iostat during backups. Are you seeing high I/O wait or CPU steal time?

d. Backup Target Bottlenecks
Even if StoreOnce VSA shows good raw write speeds, the data ingest path may be affected:

Dedupe/compression overhead.

StoreOnce throttle policy (verify in StoreOnce GUI).

MinIO (S3) — tune minio server options like --storage-class and ensure parallel PUTs are enabled.

NFSv3 — confirm it's using async exports and not being throttled by disk IOPS.

3. Stuck Backup Jobs at 0:01
This is a known problem in many backup systems when:

Cleanup scripts fail.

Snapshot remains locked.

Task is orphaned in the UI.

Fixes & Workarounds:
Restart the backup daemon:

sudo systemctl restart vme-manager
sudo systemctl restart vme-backup.service
If that doesn’t help, look for the PID of the stuck job (ps aux | grep backup) and manually kill it.

If stuck VM snapshot exists, manually delete with virsh snapshot-delete.