Open vSwitch DPDK

Open vSwitch (OVS) can use the Data Plane Development Kit (DPDK) to move packet processing from the kernel to dedicated userspace polling threads. This provides an accelerated, shareable network path when assigning a physical NIC directly to each Virtual Machine is not appropriate.

The physical NIC is owned by ovs-vswitchd, normally through vfio-pci; it is not passed through to the VM. OpenNebula connects each VM to the userspace bridge with a dpdkvhostuserclient port. QEMU creates the corresponding vhost-user socket under /var/run/one/vhost-socks/.

Although OpenNebula can create a missing OVS bridge with datapath_type=netdev, the recommended workflow is to create the bridge during Host provisioning, either manually or with OneDeploy. The physical NIC must be bound and its DPDK port or bond must be added by the administrator, so configuring the bridge at the same time makes it possible to verify the complete physical data path before deploying VMs. OpenNebula then manages the VM-side ports.

Requirements

Before starting, verify that:

• The Host is an OpenNebula KVM node.
• The physical NIC is supported by a DPDK Poll Mode Driver (PMD).
• IPMI access is available when using management links as DPDK interfaces.

This guide walks you through configuring the remaining requirements; they do not need to be prepared before starting:

• An OVS build with DPDK support and a compatible DPDK version. The guide uses distribution packages and references the OVS and DPDK compatibility table.
• Huge Pages for OVS and for every VM connected through vhost-user.
• Dedicated CPU threads for the OVS PMDs and their reservation in OpenNebula.
• IOMMU and VFIO when the selected PMD requires the NIC to be bound to vfio-pci. Bifurcated PMDs are an exception and retain their kernel driver.
• The OVS userspace bridge and its physical DPDK port or bond.
• A virtio NIC, shared memory and NUMA-aware placement in the VM Template.

Automated Configuration

The OneDeploy Open vSwitch role can install OVS-DPDK, configure kernel parameters and modules, bind PCI devices, and create OVS ports, bonds and bridges. Use it when Host networking is managed with OneDeploy.

The following sections describe the equivalent manual configuration and the OpenNebula objects required to consume it.

Step 1. Install OVS with DPDK Support

Use the packages provided for the Host operating system:

• Ubuntu and Debian: install openvswitch-switch-dpdk. Select /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk with update-alternatives if the distribution does not select it automatically. The service is openvswitch-switch.service.
• Red Hat Enterprise Linux: enable the Fast Datapath repository for the installed RHEL major version, then install one of its versioned Open vSwitch packages and dpdk-tools. The service is openvswitch.service.
• AlmaLinux: enable the NFV Open vSwitch repository with centos-release-nfv-openvswitch, then install a versioned Open vSwitch package and dpdk-tools. The service is openvswitch.service.

Start and enable the service. Use the command for the Host distribution:

# Ubuntu and Debian
systemctl enable --now openvswitch-switch.service

# Red Hat Enterprise Linux and AlmaLinux
systemctl enable --now openvswitch.service

Verify that ovs-vswitchd reports both OVS and DPDK versions:

ovs-vswitchd --version

Do not continue if the output does not contain a DPDK version.

Step 2. Plan the Host Resources

Use one consistent topology throughout the configuration. The examples in this guide use the following values; replace them with the values from each Host:

Proper resource allocation when planning a DPDK based deployment can be a challenging task. Besides what is mentioned in this guide, we also recommend checking the OVS performance tuning recomendations as well as Red Hat NFV OVS-DPK deployment planning guide for additional tunings and example configurations.

Identify the NIC and its NUMA Node

Determine the PCI address used by the uplink and verify its current driver:

ethtool -i enp1s0f0
lspci -Dnnk -s 0000:01:00.0

Display the NUMA node associated with the device:

cat /sys/bus/pci/devices/0000:01:00.0/numa_node

A value of -1 means that the platform did not report a NUMA association for the device.

When using VFIO, also inspect the complete IOMMU group:

readlink -f /sys/bus/pci/devices/0000:01:00.0/iommu_group
ls -1 /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/

The IOMMU group is the minimum device-ownership unit. For the group to be usable through VFIO, every member must be detached from its native host driver and bound to a VFIO-compatible driver, or otherwise left unbound. Inspect the group before changing any driver and do not detach a member that the Host still requires.

Allocate Huge Pages

OVS uses a shared memory model by default. Ports on the same NUMA node with the same MTU share a mempool; memory is not allocated independently for every NIC. Enabling other_config:per-port-memory=true changes this behavior and requires a separate calculation for each port.

The following values from the OVS DPDK memory model are useful starting points for the default shared model:

An additional pool may be created for every distinct MTU on the same NUMA node. Size the final allocation from the configured MTUs, queue counts and OVS memory model rather than multiplying these values by the number of NICs.

Allocate OVS memory only on NUMA nodes that will run DPDK physical or vhost-user ports. For example, the reference Host uses 1024 MB on node 0 and none on node 1:

dpdk-socket-mem=1024,0

VM memory is separate from dpdk-socket-mem. The reference Host needs at least five 1 GB Huge Pages on node 0 to start OVS and one 4 GB VM: one page for the OVS mempool and four pages for the guest. Add capacity for concurrent VMs and operational headroom.

Configure the required pages as described in Host Configuration for PCI Passthrough and SR-IOV. Huge Pages are allocated per NUMA node, so verify their placement:

grep -i '\<Huge' /sys/devices/system/node/node*/meminfo
mount | grep hugetlbfs

The directory passed to dpdk-hugepage-dir must be a mounted hugetlbfs with the intended page size. This guide uses /dev/hugepages with 1 GB pages. Do not assume that libvirt created or mounted it.

Reserve PMD CPUs

PMD threads continuously poll the Rx queues assigned to them. Start with at least one PMD CPU on every NUMA node that runs DPDK ports, then scale the number of PMDs and queues from measured traffic and packet-loss requirements. OVS assigns Rx queues to PMDs automatically unless an explicit affinity is configured; see the OVS PMD documentation.

Inspect the CPU topology:

lscpu -e=CPU,CORE,SOCKET,NODE,ONLINE

Prefer dedicated physical cores. Using both SMT siblings of one core can reduce the number of reserved cores, but usually provides less predictable maximum throughput. Keep PMDs, physical ports, VM vCPUs and VM memory on the same NUMA node whenever possible.

other_config:pmd-cpu-mask is a hexadecimal bit mask of logical CPU IDs. For the reference CPU ID 2, calculate the mask with:

python3 -c 'cpus=(2,); print(hex(sum(1 << cpu for cpu in cpus)))'
0x4

Reserve the same CPUs in the OpenNebula Host template so that the scheduler does not pin VM vCPUs to them. Update the Host through Sunstone or run:

onehost update <host-id> -a

Add the following attribute in the editor:

ISOLCPUS = "2"

The OpenNebula ISOLCPUS attribute is required for scheduler accounting. The kernel isolcpus parameter is optional CPU scheduling and performance tuning; it is not a security control.

Step 3. Prepare the Physical NIC

The required binding procedure depends on the DPDK PMD used by the NIC.

Conventional VFIO PMDs

Enable the IOMMU and configure persistent vfio-pci binding as described in Host Configuration for PCI Passthrough and SR-IOV. For the reference NIC:

driverctl set-override 0000:01:00.0 vfio-pci
dpdk-devbind.py --status-dev net

Repeat the binding operation for every other IOMMU-group member that must use vfio-pci. The status command should show drv=vfio-pci for 0000:01:00.0.

VFIO ownership must match the process consuming the group. On distributions where ovs-vswitchd runs as a non-root user, the root:kvm and 0660 rule from the generic passthrough guide is sufficient only when that service user belongs to the kvm group. Otherwise, use a suitable supplementary group or distribution-provided policy. Check both the service identity and the VFIO group device:

ps -o user,group,comm -C ovs-vswitchd
ls -l /dev/vfio/

Bifurcated PMDs

Some NICs, including devices using the mlx5 PMD, use a bifurcated driver. The device remains bound to its native kernel driver, such as mlx5_core, while DPDK accesses the data path. Do not bind such a device to vfio-pci. Check the NIC-specific section of the DPDK driver documentation before changing its driver.

Step 4. Configure OVS-DPDK

Set the initialization parameters together so that only one restart is required. The following values implement the reference topology:

ovs-vsctl --no-wait set Open_vSwitch . \
    other_config:dpdk-init=true \
    other_config:dpdk-hugepage-dir=/dev/hugepages \
    other_config:dpdk-socket-mem=1024,0 \
    other_config:pmd-cpu-mask=0x4

Restart the service for the initialization parameters to take effect:

# Ubuntu and Debian
systemctl restart openvswitch-switch.service

# Red Hat Enterprise Linux and AlmaLinux
systemctl restart openvswitch.service

Verify initialization before creating ports:

ovs-vsctl get Open_vSwitch . dpdk_initialized
ovs-vsctl get Open_vSwitch . dpdk_version
ovs-vsctl get Open_vSwitch . other_config:dpdk-socket-mem
ovs-vsctl get Open_vSwitch . other_config:pmd-cpu-mask
ovs-vsctl get Open_vSwitch . other_config:dpdk-hugepage-dir

dpdk_initialized must return true. If it does not, inspect the service log:

# Ubuntu and Debian
journalctl -u openvswitch-switch.service --since "10 minutes ago"

# Red Hat Enterprise Linux and AlmaLinux
journalctl -u openvswitch.service --since "10 minutes ago"

Step 5. Configure the Physical Uplink

Create the DPDK bridge as part of Host provisioning, either manually as shown below or with OneDeploy. OpenNebula can create a missing bridge when the first VM NIC is attached, but it cannot bind or add the physical DPDK uplink. Creating the bridge explicitly keeps the bridge and uplink configuration together and allows the physical data path to be verified before it is exposed to VMs:

ovs-vsctl --may-exist add-br ovsbr0 \
    -- set Bridge ovsbr0 datapath_type=netdev

Add the reference NIC as a DPDK port:

ovs-vsctl --may-exist add-port ovsbr0 dpdk0 \
    -- set Interface dpdk0 type=dpdk \
       options:dpdk-devargs=0000:01:00.0

The OVS interface name dpdk0 is local to OVS. A VFIO-bound device has no kernel interface name; options:dpdk-devargs identifies the device by PCI address.

Verify the bridge and physical port:

ovs-vsctl show
ovs-vsctl get Interface dpdk0 error
ovs-vsctl get Interface dpdk0 link_state

An empty error value and link_state of up indicate that OVS attached the port successfully.

If the Host needs an IP address on this network, configure it on the bridge internal interface, ovsbr0, rather than on the physical NIC. Make this change persistently with the distribution network renderer. Migrating a management address requires out-of-band access and a coordinated change to addresses, routes and DNS.

DPDK Bonds and Jumbo Frames

For redundancy, create a DPDK bond instead of the single physical port. This corrected example uses the same interface names in add-bond and set Interface:

ovs-vsctl --may-exist add-bond ovsbr0 bond0 dpdk0 dpdk1 \
    -- set Port bond0 bond_mode=balance-slb \
    -- set Interface dpdk0 type=dpdk \
       options:dpdk-devargs=0000:01:00.0 mtu_request=9000 \
    -- set Interface dpdk1 type=dpdk \
       options:dpdk-devargs=0000:81:00.0 mtu_request=9000

Do not run both the single-port and bond examples without first removing dpdk0 from its existing OVS port. If the second NIC is on another NUMA node, allocate Huge Pages and a PMD CPU on that node and update dpdk-socket-mem, pmd-cpu-mask and OpenNebula ISOLCPUS accordingly.

For jumbo frames, use the same MTU on the physical DPDK interfaces, the OVS internal interface when used, the OpenNebula Virtual Network, and the guest. Recalculate the OVS mempool before changing the MTU.

Step 6. Configure OpenNebula

The network features available for OVS with DPDK are the same ones as regular OVS. However, the bridges with accelerated datapath are consumed differently than regular bridges. Therefore there is a nuance to consider in terms of Virtual Networks and VM Templates.

Create the Virtual Network

Use the ovswitch driver and set BRIDGE_TYPE to openvswitch_dpdk. Do not define PHYDEV; the physical DPDK port is already attached to the bridge outside OpenNebula.

NAME        = "dpdk-net"
VN_MAD      = "ovswitch"
BRIDGE      = "ovsbr0"
BRIDGE_TYPE = "openvswitch_dpdk"

Normal Open vSwitch network configuration rules apply, including the Open vSwitch Security Group limitations.

Configure the VM

The VM NIC must use the virtio model. Libvirt will create Vhost-user interfaces for each NIC backed by a virtual network with BRIDGE_TYPE = "openvswitch_dpdk". These interfaces require hugepages and shared memory in the VM. In OpenNebula, configure Huge Pages and set MEMORY_ACCESS to shared. Add the following resources to a normal VM Template:

MEMORY = "4096" # must be a multiple of the HUGEPAGE_SIZE

NIC = [
  NETWORK = "dpdk-net",
  MODEL   = "virtio"
]

TOPOLOGY = [
  HUGEPAGE_SIZE = "1024",
  MEMORY_ACCESS = "shared"
]

For improved performance, you can use CPU pinning and/or NUMA node affinity, so the VM can use resources available in the node where the DPDK polling occurs.

CPU    = "2"
VCPU   = "2"
MEMORY = "4096"

NIC = [
  NETWORK = "dpdk-net",
  MODEL   = "virtio"
]

TOPOLOGY = [
  SOCKETS       = "1",
  CORES         = "2",
  THREADS       = "1",
  PIN_POLICY    = "THREAD",
  NODE_AFFINITY = "0",
  HUGEPAGE_SIZE = "1024",
  MEMORY_ACCESS = "shared"
]

For each NIC, QEMU creates a server socket in /var/run/one/vhost-socks/ and OpenNebula adds an OVS dpdkvhostuserclient interface that connects to it. The socket name matches the VM NIC target, normally one-<vm-id>-<nic-id>. Initially the VM is created in PAUSED state in libvirt. It will be set to RUNNING once the dpdkvhostuserclient ovs port connects to the socket. This connection process requires proper permissions as the OVS daemon needs to be able to connect and unlink the socket, whereas qemu needs to be able to create it. Depending on the Linux distribution used in the KVM node, the OVS daemon might be running as a dedicated user, and security management tools like AppArmor or SELINUX might block these connections.

NODE_AFFINITY="0" matches the NIC, PMD and Huge Pages in the reference topology. Change it when the physical path is on another NUMA node. See CPU and NUMA Pinning for more complex topologies.

VM Migrations, both live and offline, require the target host to have a compatible topology. That is, the KVM host must be able to provide the necessary resources for the VM topology to be satisfied. There must be huge page availability, CPUs and NUMA nodes used by the VM. The simplest approach if to have identical hosts used for migrations.

Step 7. Verify the Data Path

Verify the Host configuration:

ovs-vsctl get Open_vSwitch . dpdk_initialized
ovs-vsctl show
ovs-appctl dpif-netdev/pmd-rxq-show
ovs-appctl dpif-netdev/pmd-perf-show

pmd-rxq-show displays the physical and vhost-user Rx queues assigned to every PMD. Use it after adding ports and starting a VM; the number of PMDs is not derived from the number of interfaces.

Verify the VM-side socket and libvirt interface:

find /var/run/one/vhost-socks/ -maxdepth 1 -type s -ls
virsh dumpxml one-<vm-id>

The domain XML must contain an interface similar to:

<interface type='vhostuser'>
  <source type='unix' path='/var/run/one/vhost-socks/one-42-0' mode='server'/>
  <model type='virtio'/>
</interface>

The corresponding OVS interface must use client mode:

Interface "one-42-0"
    type: dpdkvhostuserclient
    options: {vhost-server-path="/var/run/one/vhost-socks/one-42-0"}

Troubleshooting

Most of the issues arise from bad or missing configuration, either in the deployment or the usage through VMs. You can use the one-deploy openvswitch ansible role for a cofiguration procedure reference if in doubt.

DPDK Does Not Initialize

Check:

• ovs-vswitchd --version reports DPDK support and the OVS/DPDK versions are compatible.
• The Huge Page mount exists and has sufficient free pages on the requested NUMA nodes.
• dpdk-socket-mem matches the Host NUMA topology; use 0 for unused nodes.
• The OVS service user can open the required VFIO group.
• The PCI device is using the driver required by its PMD.

Review the distribution-specific Open vSwitch service log for DPDK EAL errors after every initialization change.

Unexpected PMD CPU Usage

Polling PMDs with assigned Rx queues normally consume a complete logical CPU even when traffic is low. This is expected. In fact, it is a good way to validate the CPU mask was properly configured. For example, the following processes correlate to 4 PMD threads (2 per core) pinned to CPUs 0,1,64 and 65.

ps -eLo pid,tid,psr,pcpu,comm | grep -E "pmd"
   6267    7164   1 99.7 pmd-c01/id:392
   6267    7165   0 99.7 pmd-c00/id:393
   6267    7247  65 99.5 pmd-c65/id:394
   6267    7248  64 99.5 pmd-c64/id:395

Confirm the queue assignment with ovs-appctl dpif-netdev/pmd-rxq-show before adding PMDs.

Recent OVS versions support other_config:pmd-sleep-max to reduce idle polling. Sleeping trades CPU consumption for wake-up latency and possible packet loss during bursts; validate it with the target workload before using it in production.

Vhost-user Socket Permission Errors

Permission issues manifest as blocked VM operations. QEMU is the socket server and OVS is the client.

If the VM gets stuck on boot, it can happen that the DPDK VM NIC port created in OVS is trying to connect to the qemu socket and gets continuously denied. The VM will remain paused in libvirt until the connection is succesful.

The VM might also fail to be powered off, restarted, live migrated or snapshotted. It can happen that the unlink operation from the socket fails to be performed.

Check directory traversal, socket permissions and the service identities:

namei -l /var/run/one/vhost-socks/
ps -o user,group,comm -C ovs-vswitchd
ps -eo user,group,comm,args | grep '[q]emu-system'

On SELinux or AppArmor systems, inspect the audit or kernel log for a mandatory-access-control denial before changing policy. The required policy depends on the distribution packages and their service domains. Prefer an updated distribution policy or a reviewed local policy.

Do not use chcon as a permanent fix because its label can be lost during filesystem relabeling. If a custom SELinux file context is required and has been validated for the distribution, make it persistent with semanage fcontext and apply it with restorecon. Apply equivalent profile changes when AppArmor is in use.

No TCP traffic in VMs

A VM might have operational UDP and ICMP traffic, but specifically, TCP traffic fails. This issue arise when mixing configurations. That is, a VM was deployed as a regular OVS based VM on a DPDK bridge. In this case you have a mixed datapath, and while OVS allows it as a valid configuration, it is effectively misconfiguration as the traffic is getting both polling and kernel interruptions, the worst of both worlds.

If for whatever reason, this configuration is intended, TCP checksum offloadin has to be disabled on the Guest OS NIC ethtool --offload 'eth0' tx off rx off .