Linux KVM and assigning devices to a VM, PCI cards or SR-IOV

[Ken Green]

I've been trying to exploring assign HW devices directly into KVM based virtual machines. I've tried both with assigning whole PCI cards and also using SR-IOV functions and I'm not getting anywhere.

I'm trying this on low end systems, an ML110 G7 and an ML310e Gen8.

I'm being told I problems may be firmware related, in fact the Linux kernel is complaining about FW bugs on the G7.

Has anyone managed either of these types of configuration on these boxes?

If so I'd love to know what configurations you have and what options you've needed to pass into the kernel to enbale things.

Kernel log message (CentOS6.4) while starting up the Intel IOMMU stuff (Intel VT-D is enabled from RBSU) to support assigning a PCI card into a VM.

------------[ cut here ]------------
WARNING: at drivers/pci/dmar.c:586 warn_invalid_dmar+0x7a/0x90() (Not tainted)
Hardware name: ProLiant ML110 G7
[Firmware Warn]: Your BIOS is broken; DMAR reported at address fed90000 returns all ones!
BIOS vendor: HP; Ver: J01; Product Version:
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.32-358.el6.x86_64 #1
Call Trace:
[<ffffffff8106e2e7>] ? warn_slowpath_common+0x87/0xc0
[<ffffffff8106e37f>] ? warn_slowpath_fmt_taint+0x3f/0x50
[<ffffffff8103c4bd>] ? native_set_pte_at+0xd/0x40
[<ffffffff8103be59>] ? native_flush_tlb_single+0x9/0x10
[<ffffffff812ae3ea>] ? warn_invalid_dmar+0x7a/0x90
[<ffffffff81c57444>] ? check_zero_address+0xd6/0x118
[<ffffffff812fe6db>] ? acpi_get_table_with_size+0x5a/0xb4
[<ffffffff81519195>] ? _etext+0x0/0x3
[<ffffffff81c57498>] ? detect_intel_iommu+0x12/0x91
[<ffffffff81c3001a>] ? pci_iommu_alloc+0x5e/0x6c
[<ffffffff81c43373>] ? mem_init+0x19/0xec
[<ffffffff81c27d78>] ? start_kernel+0x221/0x430
[<ffffffff81c2733a>] ? x86_64_start_reservations+0x125/0x129
[<ffffffff81c27438>] ? x86_64_start_kernel+0xfa/0x109
---[ end trace a7919e7f17c0a725 ]---

For SR-IOV I'm using an Intel quad port PCIe card based on the 82576 chip set, the card claims to have SR-IOV functionality, it shows up if you do lspci -vv, but when I try to reload the driver with

modprobe igb max_vfs=7  I get  each of the ports reporting

SR-IOV: bus number out of range

Thanks in advance for any suggestions, or simply anyone able to confirm whether they have managed to get these faclities to work on these boxes. 

[Johan Guldmyr]

Nothing conclusive, but perhaps it works better with newer kernel + libvirt than what comes with CentOS6 (perhaps even there's newer/better support in 6.5 compared to 6.4?).

Ubuntu / RHEL7 / SUSE

That FW bug message has been around for quite a while, seeing it on bl460c G1 we still use..

[Ken Green]

I've tried with Ubuntu Precise too, with the same sorts of problems. I'm installing RHEL7.0 onto the G7 box at the moment to try. It still gives the warning about buggy BIOS in dmesg, but I've not got any further yet. I'll keep you posted.

[Ken Green]

OK I've rebuilt the ML110 G7 using RHEL7.0 and tried to add a PCI card to a VM.

To keep things simple I used the Virtual Machine Manager this time. When I click the button asking to create the VM it gives the error

Unable to complete install: 'unsupported configuration: host doesn't support passthrough of host PCI devices'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 100, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 1925, in do_install
    guest.start_install(False, meter=meter)
  File "/usr/share/virt-manager/virtinst/Guest.py", line 1143, in start_install
    noboot)
  File "/usr/share/virt-manager/virtinst/Guest.py", line 1211, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2892, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices

Looking in the syslog file I get

Jun 30 13:45:22 ml110c journal: libvirt version: 1.1.1, package: 29.el7 (Red Hat
, Inc. <http://bugzilla.redhat.com/bugzilla>, 2014-03-24-06:10:00, x86-021.build.eng.bos.redhat.com)
Jun 30 13:45:22 ml110c journal: unsupported configuration: host doesn't support passthrough of host PCI devices

I've enabled the intel_iommu=on kernel option.

I've enabled Intel(R) VT-d in BIOS.

[ken@ml110c ~]$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-121.el7.x86_64 root=UUID=15d0146f-02fb-4110-8d56-520ed00ab1d1 ro console=ttyS1 rd.lvm.lv=rhel_ml110c/swap vconsole.font=latarcyrheb-sun16 crashkernel=auto rd.lvm.lv=rhel_ml110c/root vconsole.keymap=uk intel_iommu=on

I'd just love to know whether anyone has been able to do this on one of these systems.

Cheers

[Ken Green]

I've repeated the test using an DL380 Gen8 server, which is running RHEL6.4. This was quite happy to let me assign one of it's LAN ports directly into a KVM VM.

Checking the iommu behaviour on the DL380 I get 

Command line: ro root=/dev/mapper/vg_station15-lv_root rd_NO_LUKS LANG=en_US.UTF
-8 rd_LVM_LV=vg_station15/lv_root rd_NO_MD KEYTABLE=us SYSFONT=latarcyrheb-sun1
6 console=ttyS1 crashkernel=auto rd_LVM_LV=vg_station15/lv_swap rd_NO_DM rd_NO_P
LYMOUTH intel_iommu=on
Kernel command line: ro root=/dev/mapper/vg_station15-lv_root rd_NO_LUKS LANG=en
_US.UTF-8 rd_LVM_LV=vg_station15/lv_root rd_NO_MD KEYTABLE=us SYSFONT=latarcyrh
eb-sun16 console=ttyS1 crashkernel=130M@0M rd_LVM_LV=vg_station15/lv_swap rd_NO_
DM rd_NO_PLYMOUTH intel_iommu=on
dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping.
dmar: Host address width 46
dmar: DRHD base: 0x000000f8ffe000 flags: 0x1
dmar: IOMMU 0: reg_base_addr f8ffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
dmar: RMRR base: 0x000000bdffd000 end: 0x000000bdffffff
dmar: RMRR base: 0x000000bdff6000 end: 0x000000bdffcfff
dmar: RMRR base: 0x000000bdf83000 end: 0x000000bdf84fff
...

and on and on... listing lots of IOMMU stuff too.

On the ML110 G7 I get

[root@ml110c ~]# grep -e dmar -e iommu /var/log/dmesg
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-121.el7.x86_64 root=UUID=15d0146f-02fb-4110-8d56-520ed00ab1d1 ro console=ttyS1 rd.lvm.lv=rhel_ml110c/swap vconsole.font=latarcyrheb-sun16 crashkernel=auto rd.lvm.lv=rhel_ml110c/root vconsole.keymap=uk intel_iommu=on
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-121.el7.x86_64 root=UUID=15d0146f-02fb-4110-8d56-520ed00ab1d1 ro console=ttyS1 rd.lvm.lv=rhel_ml110c/swap vconsole.font=latarcyrheb-sun16 crashkernel=auto rd.lvm.lv=rhel_ml110c/root vconsole.keymap=uk intel_iommu=on
[ 0.000000] [Firmware Warn]: drivers/iommu/dmar.c at 484: Your BIOS is broken; DMAR reported at address fed90000 returns all ones!
[ 0.055464] dmar: Host address width 39
[ 0.056663] dmar: DRHD base: 0x000000fed90000 flags: 0x1
[ 0.058228] dmar: IOMMU: failed to map dmar0
[ 0.059517] dmar: parse DMAR table failure.
[root@ml110c ~]#

And trying to dettach a PCI function gives

[root@ml110c ~]# virsh nodedev-dettach pci_0000_10_00_1
error: Failed to detach device pci_0000_10_00_1
error: Operation not supported: neither VFIO nor KVM device assignment is currently supported on this system

Which is why I wondering wether this is a problem with the ML110 G7.
I'll rebuild my ML310e Gen8 box into RedHat and repeat the tests there, It running Ubuntu 12.04 at the moment. Then I can repeat the same test.

[fpinedo]

Ken

ML110 Gen7 and ML310e Gen8  are not SR-IOV Compliant and DL380 Gen8 is SR-IOV Compliant, look the paper in table 2

https://www.hpe.com/h20195/v2/Getdocument.aspx?docname=4aa4-7658enw
Although this documentation is for Microsoft Windows Server 2012 R2, must be the same for linux virtualization.

Also vt-d is prerequisite for "SR-IOV", but does not mean it is "SR-IOV Compliant". Until now Hp not has released a bios with "SR-IOV Compliant" in the Proliant Ml110 or Ml310 servers. Unlike proliant Dl380 that if is "SR-IOV Compliant".