Debian, KVM and PCI-E passthrough for Intel 82599 / Intel X-540 NIC's

First of all, please enable hardware virtualization and VT-d (I/O virtualization) in BIOS. It's mandatory!

Install KVM tools and distro install tools on the Debian:

apt-get install -y qemu-kvm  libvirt-daemon libvirt-clients libvirt-daemon-system virtinst

Enable libvirt and libvirt.guests services:

systemctl restart libvirtd.service
systemctl restart libvirt-guests.service 

Run default network:

virsh net-autostart default
virsh net-start default

Install Debian Jessie:

virt-install --location http://ftp.us.debian.org/debian/dists/stable/main/installer-amd64/ --ram 16000 --file  /var/lib/libvirt/images/fastnetmonvm.fastvps.ru.disk  --file-size 20 --name fastnetmonvm.fastvps.ru -x console=ttyS0 

Switch on autostart for VM:

virsh autostart fastnetmonvm.fastvps.ru 

Well, we should pass whole NIC to VM.

List all available NIC's on the server:

lspci |grep Ether
03:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

We will pass two NICs (it's two port NIC card):

03:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Get PCI device names in virsh format (03_00_0 created from 03:00.0 with changing ':' by '_'):

virsh nodedev-list | grep pci |grep 03_00_0
pci_0000_03_00_0
pci_0000_03_00_1

Get XML descriptions for both PCI devices:

virsh nodedev-dumpxml pci_0000_03_00_0
virsh nodedev-dumpxml pci_0000_03_00_1 

Switch off drivers for this NIC's on server:

ifconfig ethX down
rmmod ixgbe

Detach both NIC from the Linux:

virsh nodedev-dettach pci_0000_03_00_0
virsh nodedev-dettach pci_0000_03_00_1

If you got: "error: Operation not supported: neither VFIO nor KVM device assignment is currently supported on this system".

Well, we should enable IOMMU for fixing this issue:

vim /etc/default/grub

Change following line to:

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"

Apply grub settings:

update-grub

And reboot server:

shutdown -r now

Then repeat last steps.

List devices from virsh:

virsh nodedev-dumpxml pci_0000_03_00_0 |grep address
   <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
   <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> 

Then please build something like this using data above:

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
      <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
  </source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
      <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/>
  </source>
</hostdev>

Then open VM edit and insert this block to "devices" block:

virsh edit fastnetmonvm.fastvps.ru

Then you could try to run VM:

virsh start fastnetmonvm.fastvps.ru

You could get error similar to this:

error: Failed to start domain fastnetmonvm.fastvps.ru
error: internal error: early end of file from monitor: possible problem:
2015-05-13T13:12:39.353614Z qemu-system-x86_64: -device vfio-pci,host=0a:00.0,id=hostdev0,bus=pci.0,addr=0x6: vfio: failed to set iommu for container: Operation not permitted
2015-05-13T13:12:39.353761Z qemu-system-x86_64: -device vfio-pci,host=0a:00.0,id=hostdev0,bus=pci.0,addr=0x6: vfio: failed to setup container for group 23
2015-05-13T13:12:39.353780Z qemu-system-x86_64: -device vfio-pci,host=0a:00.0,id=hostdev0,bus=pci.0,addr=0x6: vfio: failed to get group 23
2015-05-13T13:12:39.353804Z qemu-system-x86_64: -device vfio-pci,host=0a:00.0,id=hostdev0,bus=pci.0,addr=0x6: Device initialization failed.
2015-05-13T13:12:39.353827Z qemu-system-x86_64: -device vfio-pci,host=0a:00.0,id=hostdev0,bus=pci.0,addr=0x6: Device 'vfio-pci' could not be initialized

If you have something like " vfio-pci 0000:0a:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor." in dmesg that's not a good news for you. You could not use IOMMU for your server and need to buy another.

For fix issue "vfio_iommu_type1_attach_group: No interrupt remapping support" please use:

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

And reboot again.

Sources:   https://www.paloaltonetworks.com/content/dam/paloaltonetworks-com/en_US/assets/pdf/framemaker/61/pan-os/NewFeaturesGuide/section_7.pdf and http://h20564.www2.hp.com/hpsc/doc/public/display?docId=emr_na-c03645796