GPU passthrough on Proxmox for local model servers

Virtualizing AI infrastructure on Proxmox Virtual Environment (VE) now forms a stable foundation for anyone seeking flexibility without sacrificing raw compute. When we want to deploy inference servers such as vLLM, Ollama, Aphrodite, or TGI on consumer or enterprise hardware, we do not want to be locked into a single bare-metal installation. Through PCIe GPU passthrough via VFIO (Virtual Function I/O), we assign a physical graphics card directly and exclusively to a virtual machine (VM). This gives the guest VM direct access to the CUDA cores, Tensor cores, and the full video memory bandwidth (VRAM), with negligible virtualization overhead of less than one percent compared to bare-metal inference.

In this article we analyze the technical architecture behind GPU passthrough on Proxmox VE 8.x, walk through the configuration steps across IOMMU boundaries and kernel modules, and examine how to set up a resilient model server. In doing so we look critically at performance losses, hardware pitfalls, and architectural considerations around memory management and network throughput.

Architecture: bare metal vs. virtual machine with VFIO

When running large language models (LLMs), the memory bandwidth between the GPU chip and onboard VRAM is the primary limiting factor for generation speed (tokens per second). Hosting models on bare-metal Linux offers the simplest configuration, but introduces significant operational drawbacks: snapshots are hard to realize, hardware allocations are rigid, and isolating networks and storage simultaneously requires complex container configurations. We already saw a comparison with broader homelab infrastructures in the homelab and self-hosted AI signals, where the balance between isolation and compute density took center stage.

With Proxmox and VFIO passthrough, the hypervisor acts purely as an arbiter of physical PCIe registers and memory pages via the IOMMU (Input-Output Memory Management Unit). The host kernel loads the vfio-pci driver instead of the official nvidia or amdgpu drivers. As soon as the VM starts, the QEMU/KVM hypervisor claims the PCI memory space via Direct Memory Access (DMA), so the guest VM drives the GPU as if it were plugged directly into the virtual motherboard. As long as DMA remapping is correctly enabled, all PCIe transactions are handled in hardware without CPU context switches.

Hardware requirements and IOMMU grouping

A successful passthrough configuration stands or falls with the physical PCIe topology and the motherboard design. The CPU and motherboard must support Intel VT-d or AMD-Vi respectively. Beyond that, the quality of the IOMMU grouping is decisive: the IOMMU groups devices based on electrical circuits and PCIe switches. A GPU can only be passed through safely to a VM if it sits in an isolated IOMMU group, or if all other devices within that same group (such as the accompanying HDMI audio controller or USB-C controllers) are assigned to the same VM at the same time.

On consumer motherboards (such as B650 or Z790 chipsets), the secondary PCIe x16 slot (which often runs electrically at x4) frequently shares an IOMMU group with SATA controllers, network ports, or M.2 NVMe slots. Attempts to pass a GPU out of such a mixed group invariably result in kernel panics on the host or errors while booting the VM. Enterprise and HEDT platforms (such as AMD EPYC, Threadripper, or Intel Xeon) generally offer strictly separated IOMMU groups for each individual PCIe slot, thanks to dedicated PCIe lanes straight from the CPU package.

Host configuration: kernel parameters and VFIO modules

To prepare Proxmox VE for passthrough, we must pass specific boot flags to the Linux kernel and prevent the host drivers from claiming the video signal. Depending on the boot system (GRUB or systemd-boot), we adjust the kernel command line. For a detailed overview of the initial boot flags and configuration files, we refer to the guide on configuring GPU passthrough in Proxmox VE for LLMs, where the basic parameters are documented step by step.

In the configuration file /etc/default/grub or /etc/kernel/cmdline we add for AMD systems amd_iommu=on iommu=pt , or intel_iommu=on iommu=pt for Intel systems. The parameter iommu=pt (passthrough mode) ensures that the hypervisor performs IOMMU translations only for devices actually assigned to guests, which optimizes I/O performance for the remaining host devices.

Next we configure the required kernel modules in /etc/modules so that they are loaded during the early boot phase:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

To prevent the Proxmox host from initializing the GPU with the default open-source drivers, we blacklist those drivers in /etc/modprobe.d/pve-blacklist.conf:

blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist radeon
blacklist amdgpu

We then bind the specific PCI vendor and device IDs to vfio-pci. We obtain these IDs via the command lspci -nn | grep -E "VGA|Audio". We place the result in /etc/modprobe.d/vfio.conf:

options vfio-pci ids=10de:2684,10de:22ba disable_vga=1

After updating the initial RAM disk file via update-initramfs -u -k all and restarting the host, the command dmesg | grep -i vfio verifies whether the driver successfully claims the graphics card.

VM configuration: QEMU machine type, UEFI, and PCIe flags

Within the Proxmox web interface or via the CLI utility qm we set up a virtual machine with specific hardware properties suited to enterprise AI workloads. A wrong configuration of the virtual BIOS or machine type can lead to error codes such as the notorious Code 43 in drivers, or to initialization errors in the CUDA runtime.

The VM must be configured with machine type q35 and BIOS OVMF (UEFI). The modern q35 chipset model simulates a native PCIe bus architecture, which is essential for advanced functionality such as PCIe Extended Configuration Space and Resizable BAR. We preferably configure the processor as host to pass all instruction sets (AVX2, AVX-512, BMI2) straight through to the guest.

In the virtual machine's configuration file (for example /etc/pve/qemu-server/100.conf) we declare the PCI device explicitly:

bios: ovmf
machine: q35
cpu: host
numa: 1
memory: 32768
balloon: 0
hostpci0: 0000:01:00,pcie=1,x-vga=0

Two settings are crucial here for AI servers:
1. balloon: 0: Memory ballooning must be disabled without exception. VFIO requires that the VM's full allocated RAM be locked (pinned) directly and contiguously in physical host memory to prevent DMA corruption.
2. pcie=1: This attaches the device to a virtual PCIe root port instead of a legacy PCI bus, which is necessary for maximum PCIe throughput.

Resizable BAR (ReBAR) and HugePages for LLM workloads

When loading models with 70 billion parameters or more, the model server moves tens of gigabytes of weights from system RAM and NVMe storage directly into video memory. Traditional PCIe access works with a window of just 256 MB (the Base Address Register). Resizable BAR (known on AMD platforms as Smart Access Memory) allows the CPU to address the entire VRAM in one contiguous address space.

For maximum throughput within Proxmox, ReBAR must be enabled both in the host's UEFI motherboard BIOS (Above 4G Decoding and ReBAR Support) and initialized correctly within the QEMU configuration. Without ReBAR, the initial load time of model files (via mmap or safetensors) is noticeably slower, and micro-stutters can occur during context swapping in multi-user batching.

In addition, using HugePages on the Proxmox host and in the VM is a proven method for minimizing Translation Lookaside Buffer (TLB) overhead. By using 1 GB or 2 MB HugePages instead of the default 4 KB memory pages, CPU load during intensive KV cache manipulation drops dramatically. This optimization pattern aligns with the trends we analyzed earlier around local LLMs and Ollama signal analysis, where memory latency is the primary bottleneck.

# Reserveren van 32GB aan 2MB HugePages op de host
echo 16384 > /proc/sys/vm/nr_hugepages
# Toevoegen aan VM-configuratie (/etc/pve/qemu-server/100.conf)
hugepages: 2

Model server deployment: vLLM, Aphrodite, and Ollama in a VM

Once the VM boots with Ubuntu 24.04 LTS or Debian 12 and the official Nvidia Data Center or CUDA drivers are installed, we validate functionality with nvidia-smi. From this point on we can roll out the inference stack. The choice of model server depends on the intended use case:

For production environments with multiple concurrent calls, vLLM offers superior performance thanks to PagedAttention, dynamic continuous batching, and optimized CUDA kernels. We start a vLLM container via Docker inside the VM with full GPU support:

docker run --gpus all \
  -v /root/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-7B-Instruct \
  --gpu-memory-utilization 0.95 \
  --max-model-len 8192

For light, local development scenarios, Ollama remains popular for its simple model management. For anyone who wants advanced container isolation and sandboxing at the level of model servers and agent executions, the overview on sandboxing LLM tools and Docker isolation offers a detailed look at establishing safe container boundaries within virtualized environments.

When we then integrate these local models into broader application networks, we can expose them via a central gateway. See the architecture guide on self-hosting local models behind your own API and hybrid routing for methods to combine internal model servers seamlessly with public cloud fallbacks.

Indicative reference values and measuring for yourself: PCIe bandwidth, token latency, and multi-tenant stress

The table below shows indicative reference values rather than our own measurements. To verify whether virtual passthrough actually approaches maximum hardware performance, you should check this yourself on your own hardware on two fronts: PCIe memory throughput via bandwidthTest, and actual token generation with your own inference test.

Via the Nvidia CUDA sample program bandwidthTest we measure Host-to-Device (H2D) and Device-to-Host (D2H) speeds across the virtual PCIe bus. On a physical PCIe 4.0 x16 slot, throughput should sit consistently above 25 GB/s. A drop to ~6-7 GB/s indicates that the slot has electrically fallen back to PCIe 3.0 or x4 mode, which mainly hurts initial model load time and prompt evaluation (Time To First Token).

The reference values show that the pure generation process (where weights circulate entirely in VRAM) performs almost identically between bare metal and Proxmox VFIO. The minimal deviation in Time To First Token (TTFT) arises from IOMMU address translation while loading the initial prompt tokens into video memory, but in practice this difference is negligible.

Pitfalls, troubleshooting, and limitations

Although GPU passthrough on Proxmox is a mature technology, the system carries specific operational risks and limitations that must be weighed in advance:

1. No Proxmox live migration
Because the physical PCIe hardware is bound directly to the hardware registers of that specific host, it is not possible to live-migrate a running VM to another Proxmox node in a cluster. Migration always requires the VM to be shut down completely, after which the configuration on the target host must have identical hardware and PCIe mappings.

2. PCIe reset bugs and FLR (Function Level Reset)
Some consumer graphics cards (notably certain AMD Radeon RX series and older Nvidia GTX cards) do not support Function Level Reset correctly. When the virtual machine restarts, the GPU can get stuck in an undefined power state (D3 state). This requires a full restart of the physical Proxmox host to make the GPU functional again. Enterprise cards (such as the Nvidia A100, L4, L40S, or RTX 6000 Ada) have robust FLR support and do not suffer from this.

3. VRAM fragmentation and host OOM
Because VFIO requires guest memory to be locked, the Proxmox host must never be overcommitted on RAM. If ZFS memory caching (ARC) or other containers on the host unexpectedly cause memory pressure, the Linux Out-Of-Memory (OOM) killer can terminate arbitrary processes, including the QEMU process driving the GPU, leading to sudden VM crashes.

Conclusion and best practices

GPU passthrough on Proxmox VE offers the ideal balance between enterprise manageability and bare-metal AI performance. By setting up the hypervisor correctly with isolated IOMMU groups, q35chipsets, HugePages, and Resizable BAR, we build a resilient model server that gives up almost nothing to a dedicated bare-metal installation. For anyone managing scalable local AI workloads within a virtualized infrastructure, this setup forms a future-proof standard.