Host Configuration for PCI Passthrough and SR-IOV

This guide describes the common host configuration required to enable PCI passthrough in OpenNebula. These requirements apply independently of the type of PCI device being assigned to virtual machines. Device-specific configuration is covered in the corresponding guides. The guide assumes a Linux host running the OpenNebula KVM hypervisor. While the examples use common Linux utilities and a GRUB-based bootloader, equivalent procedures can be used with other Linux distributions and bootloaders.

Prerequisites

Before configuring PCI passthrough, verify that the host satisfies the following requirements:

• Hardware-assisted virtualization enabled in the system firmware.
• An IOMMU implementation supported by the processor:
  • Intel VT-d
  • AMD-Vi
  • Arm SMMUv3
• OpenNebula host packages installed, including KVM, QEMU, and libvirt support.

Step 1. Enable the IOMMU

PCI passthrough requires the Input-Output Memory Management Unit (IOMMU) to isolate DMA accesses performed by assigned devices.

Enable the IOMMU in the Firmware

Access your system’s BIOS or UEFI setup and enable the corresponding IOMMU option. Typical firmware options include:

• Intel VT-d
• AMD IOMMU
• Arm SMMU

Save the changes and reboot the system.

Enable the IOMMU in the Kernel

• Arch:
• Intel
• AMD
• ARM

intel_iommu=on iommu=pt

amd_iommu=on iommu=pt

Verification

Display the kernel command line:

cat /proc/cmdline

Verify that the IOMMU has been initialized:

dmesg | grep -Ei "iommu|dmar|smmu"

Typical output:

DMAR: IOMMU enabled

or:

AMD-Vi: Initialized

or:

arm-smmu-v3 ...

Step 2. Configure VFIO Device Binding

PCI devices assigned to virtual machines must be managed by a VFIO driver instead of their native host driver. OpenNebula supports any mechanism that binds devices to VFIO. The recommended approach is to use the driverctl utility, which provides persistent per-device driver overrides using the Linux kernel driver_override interface.

For Linux Containers, the PCI device driver shouldn’t be changed. Skip this section entirely if using passthrough in Linux Containers.

Install driverctl

Install the driverctl package using your Linux distribution package manager.

Load the VFIO Modules

Load the required kernel modules:

modprobe vfio
modprobe vfio-pci

To load them automatically after reboot, create:

/etc/modules-load.d/vfio.conf

With contents:

vfio
vfio-pci

Identify the PCI Device

List the available PCI devices:

lspci -nn

Example output:

43:00.0 3D controller: NVIDIA Corporation GH100 ...
43:00.1 Audio device: NVIDIA Corporation ...

Display the current driver:

lspci -nnk -s 43:00.0

Example output:

Kernel driver in use: nvidia

Bind the Device

Assign the device to the vfio-pci driver:

driverctl set-override 0000:43:00.0 vfio-pci

Many PCI devices expose multiple functions. For example, GPUs often expose both a graphics controller and an audio controller. Repeat the command for every PCI function that will be assigned to the virtual machine. The override is persistent and is automatically restored after reboot.

Verification

Verify that the device is managed by VFIO:

lspci -nnk -s 43:00.0

Expected output:

Kernel driver in use: vfio-pci

Verify that libvirt can detect the device:

virsh nodedev-list | grep pci

List the configured driver overrides:

driverctl list-overrides

Removing an Override

To restore the default host driver:

driverctl unset-override 0000:43:00.0

VFIO Device Ownership

For OpenNebula to manage the VFIO device files in /dev/vfio/, the files must be owned by user root and group kvm (root:kvm). This is achieved by creating a udev rule.

1. Identify the IOMMU group for your GPU using its PCI address:

   find /sys/kernel/iommu_groups/ -type l | grep 43:00.0
   
   /sys/kernel/iommu_groups/85/devices/0000:43:00.0
   

   In this example, the IOMMU group is 85. The IOMMU group is the minimum device-ownership unit used by VFIO. All devices in the same group must be configured for passthrough and assigned together to the same virtual machine. Modern server platforms usually provide suitable per-device or per-slot isolation, but verify the group membership before continuing:

   ls -1 /sys/kernel/iommu_groups/85/devices/
   

2. Create a udev rule: Create the file /etc/udev/rules.d/99-vfio.rules with the following content:

   SUBSYSTEM=="vfio", GROUP="kvm", MODE="0660"
   

3. Reload udev rules:

   udevadm control --reload
   udevadm trigger
   

4. Verify ownership: Check the ownership of the device file corresponding to your GPU’s IOMMU group.

   ls -la /dev/vfio/
   crw-rw---- 1 root kvm 509, 0 Oct 16 10:00 85
   

Step 3. Configure Huge Pages (Optional)

Huge Pages reduce memory translation overhead and are recommended for memory-intensive workloads such as GPUs and high-performance networking.

Edit the bootloader configuration and append the appropriate Huge Page parameters to the kernel command line.

Example:

hugepages=1024 default_hugepagesz=1G hugepagesz=1G

The appropriate Huge Page size depends on the processor architecture and workload. On x86 platforms, common Huge Page sizes are 2 MB and 1 GB.

Regenerate the bootloader configuration using the appropriate command for your Linux distribution and reboot the host.

Notes:

• Ensure sufficient free memory is available before reserving Huge Pages.
• The selected Huge Page size must be supported by both the processor and the kernel.
• Display the supported Huge Page size:

  grep Hugepagesize /proc/meminfo
  

OpenNebula allows virtual machines to request Huge Pages directly from the VM Template.

Verification

Display the current Huge Page configuration:

grep Huge /proc/meminfo

Step 4. Configure PCI Monitoring

OpenNebula discovers PCI devices through the host monitoring subsystem. PCI monitoring is disabled by default to avoid exposing unnecessary hardware resources.

Before PCI devices can be assigned to virtual machines, configure the monitoring subsystem to discover the devices intended for passthrough.

Configure System-wide PCI Monitoring

The default monitoring configuration is stored on the Front-end:

/var/lib/one/remotes/etc/im/kvm-probes.d/pci.conf

The configuration supports three filtering mechanisms:

• filter — Select devices using PCI vendor, device and class identifiers.
• short_address — Restrict discovery to specific PCI addresses.
• device_name — Select devices matching one or more regular expressions.

The filters are applied sequentially.

Example:

filter:

  - "10de:*"

short_address: []

device_name: []

To monitor a specific PCI device:

filter:

  - "10de:*"

short_address:

  - "43:00.0"

device_name: []

To discover SR-IOV Virtual Functions:

filter:

  - "*:*"

device_name:

  - "Virtual Function"

Override Monitoring at Cluster or Host Level

The system-wide configuration can be overridden for individual clusters or hosts using the corresponding template attributes:

This allows different clusters to expose different classes of PCI devices while sharing the same frontend configuration.

Synchronize the Monitoring Probes

After modifying pci.conf, synchronize the updated monitoring probes to all hosts (run as oneadmin):

onehost sync --force

Then trigger a new monitoring cycle:

onehost forceupdate <host>

Alternatively, wait for the next scheduled monitoring cycle.

Step 5. Verify PCI Device Discovery

After the monitoring cycle completes, verify that OpenNebula has discovered the expected PCI devices. Display the monitored devices:

onehost show <host>

Alternatively, inspect the PCI Devices tab from the Sunstone interface.

Verify that:

• The expected PCI devices are listed.
• Vendor and device identifiers are correct.
• Driver information is correct.
• NUMA information is reported.
• SR-IOV Virtual Functions are detected, when applicable.
• Device availability matches the host configuration.

If a device does not appear:

• Verify that it matches the configured monitoring filters.
• Verify that pci.conf has been synchronized to the hosts.
• Verify that the host has completed a monitoring cycle.
• Verify that the device is visible using lspci.
• Verify that the device is bound to the appropriate VFIO driver.
• Verify that the OpenNebula monitoring service is running.

Configuration Checklist

Before deploying virtual machines using PCI passthrough, verify that the following requirements have been completed:

• ✔ Hardware virtualization enabled
• ✔ IOMMU enabled
• ✔ VFIO modules loaded
• ✔ PCI devices bound to VFIO
• ✔ Huge Pages configured (if required)
• ✔ SR-IOV configured (if required)
• ✔ PCI monitoring configured
• ✔ PCI devices discovered by OpenNebula

Once these checks are complete, the host is ready to assign PCI devices to virtual machines.

Next Steps

After completing the host configuration:

• Continue with the Generic PCI Devices Guide to assign storage controllers, USB controllers, audio devices, and other generic PCI devices.
• Continue with the Network Interfaces Guide to configure PCI passthrough or SR-IOV network devices.
• Continue with the NVIDIA GPUs Guide for NVIDIA GPU passthrough, mediated devices (vGPU), and platform-specific GPU configuration.
• Continue with the NVIDIA Fabric Manager Guide when deploying supported NVSwitch-based systems.