How to passthrough a PCIe or On-Board device/controller in Proxmox VE v.8.2.4.
At times you might want to passthrough certain devices or controller to a VM Guest for full access and control. This can be a GPU, RAID Controller, NIC etc. This is very useful in scenario such as testing a NAS (TrueNAS).
Pre-requisites :-
1. System CPU must support IOMMU (I/O Memory Management Unit) interrupt remapping. Usually this is called Intel VT-d or AMD-Vi or similar names. Most newer Mainboard and CPU already have such features.
2. This passthrough methods means the entire device or controller is no longer in use by Proxmox VE itself and only manage by the VM Guest. This means you cannot passthrough a controller where the "Boot-Pool" is connected. It must be a separate controller or device.
Scenario :-
My test system or home lab system is a HP Z800 WorkStation series with the following hardware configurations, if your system is newer than mine then most probably it will works too :-
- Dual Intel Xeon X5660 2.80Ghz, 6-Cores, 12-Threads, 12MB-Cache.
- 96GB DDR3 1333Mhz ECC RAM.
- Intel 5520 Chipset.
- On-board Intel Matrix 6-ports SATA/RAID Controller.
- On-board LSI 1068E 8-ports SAS/SATA Controller.
- LSI 9212-4i SAS/SATA 4-ports PCIe Controller.
The "LSI 9212-4i SAS/SATA 4-ports PCIe Controller" is where Proxmox VE is installed, thus I cannot passthrough this controller.
Let's Get Started :-
1. Login to the Proxmox VE's WebUI.
2. Select the Node, in this case I have only a single node.
3. Open --> Shell
4. Verify that your system have IOMMU enabled.
|
|
The results should shows "DMAR: IOMMU enabled" OR "DMAR: Intel(R) Virtualization Technology for Directed I/O". If either of these is missing then you need to check your BIOS/UEFI whether those virtualization technology are enabled.
5. Edit the "Kernel" file.
|
nano /etc/kernel/cmdline
|
Add the following line to the file.
| root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on |
If your CPU is and AMD type, then add "amd_iommu=on" to it.
6. Next is to edit the "Modules" file.
| nano /etc/modules |
Add the following into the file.
| vfio vfio_iommu_type1 vfio_pci vfio_virqfd |
7. Now we also need to edit the "GRUB" file.
| nano /etc/default/grub |
Look for the following line and append the line as shown.
| GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on" |
If your CPU is and AMD type, then add "amd_iommu=on" to it.
8. Once all done, we need to refresh the "Proxmox" boot.
| proxmox-boot-tool refresh |
9. Update the "initfamfs" also.
| update-initramfs -u -k all |
10. Next is to update "GRUB" also.
| update-grub |
11. Finally, we need to REBOOT the Proxmox VE host server.
12. After reboot, verify the "IOMMU" configuration.
| dmesg | grep -e DMAR -e IOMMU -e AMD-Vi |
Results :-
DMAR: IOMMU enabled OR DMAR: Intel(R) Virtualization Technology for Directed I/O
13. Verify "vfio" configuration.
| lsmod | grep vfio |
Results :-
vfio_pci 49152 0
vfio_virqfd 16384 1 vfio_pci
irqbypass 16384 2 vfio_pci,kvm
vfio_iommu_type1 32768 0
vfio 32768 2 vfio_iommu_type1,vfio_pci
14. Verify "Interrupt Remapping" configuration.
| dmesg | grep 'remapping' |
Results :-
AMD-Vi: Interrupt remapping enabled OR DMAR-IR: Enabled IRQ remapping in 2xapic mode (only for older CPU model)
If you encounter an error message on the remapping interrupts, type the following command and reboot the Proxmox server again.
| echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf |
15. Now you can create a VM and passthrough a PCIe device to that VM. Take note you need to ensure the "Machine = q35" else it will not work.
Also note that the device must not be in the same IOMMU Group, if the device is in the same group of another device then high chances it will not work or the entire group will also be passthrough to the VM.
For more information :-
- https://pve.proxmox.com/wiki/PCI_Passthrough
- https://pve.proxmox.com/wiki/PCI(e)_Passthrough
- https://www.thomas-krenn.com/en/wiki/Enable_Proxmox_PCIe_Passthrough
!!! HAPPY COMPUTING !!!
