Testing SR-IOV Switchdev Without a SmartNIC

Developing SR-IOV switchdev support usually means keeping a hardware lab close by. A policy change can touch PCI device discovery, VF creation, driver binding, representor discovery, OVS configuration, and CNI namespace moves. Unit tests help, but they cannot tell us whether those pieces work together when two pods send traffic.

switchdev-mock-driver gives us that integration environment inside QEMU/KVM. It combines genuine emulated PCI SR-IOV devices with a Linux kernel driver that exposes switchdev interfaces and executes supported TC flower rules in guest memory.

The goal is repeatable testing of the offload integration contract. There is no physical acceleration: the guest CPU still handles packets. That distinction makes the driver useful for development and regression testing while keeping performance validation in the hardware lab.

Why build this?

Consider a regression in ovs-cni: a VF reaches the pod, but its representor is attached to the wrong OVS port. The pod may start successfully, yet traffic fails. Or an operator update creates the right VFs but loses their intended driver binding after a reboot. Neither problem is fully exercised by checking generated YAML.

We need a test that follows the same boundaries as the real deployment:

1. The operator discovers a PF and configures VFs and switchdev mode.
2. The device plugin advertises VF resources to Kubernetes.
3. Multus and ovs-cni allocate an endpoint, move it into a pod, and attach its representor to OVS.
4. OVS installs TC rules through the driver’s offload callbacks.
5. Packets execute those rules, and their counters increase.

The mock lets us reproduce these failures on disposable virtual workers without reserving a physical SmartNIC for every iteration. It also lets regular SR-IOV tests and switchdev tests run in the same virtual cluster, on separate test PFs.

Why not use veth pairs or netdevsim? A veth pair can test software forwarding, but it does not provide the PCI PF/VF lifecycle that the operator and CNI inspect. The project’s architecture uses netdevsim as a useful reference, but its simulated bus is not a drop-in PCI SR-IOV topology. We need actual PCI relationships, including virtfn*, physfn, and each device’s net directory.

Real PCI topology, a software switch

The starting point is QEMU’s emulated Intel 82576 igb device. It supplies the PCI SR-IOV capability and VF devices. The mock replaces the guest driver for one explicitly selected test PF; it does not invent sysfs directories or impersonate another NIC vendor.

One module, mock_smartnic.ko, registers two PCI drivers:

• mock_smartnic_pf claims the configured emulated PF.
• mock_smartnic_vf claims only VFs belonging to that PF.

The management NIC keeps its normal driver. With two VFs, the mock exposes five network devices: one PF/uplink, two VF endpoints, and two representors. Each endpoint has its real emulated VF PCI device as its parent. Representors are separate software netdevices associated with devlink ports.

The PCI devices come from QEMU. The switchdev behavior and local packet forwarding come from the guest kernel module.

The PF exposes devlink eswitch mode changes between legacy and switchdev. In switchdev mode, devlink port identity gives userspace the information it needs to associate each representor with its VF. VF endpoints can move into pod network namespaces while their representors stay on the worker.

The synthetic uplink allows the operator to discover and attach the PF to OVS. It is not an external wire: the mock does not use native igb DMA queues to reach QEMU’s network backend. Frames transmitted toward that uplink are counted and consumed. The validated traffic topology is two pods on the same worker.

Implementing the packet path

The driver is split along its existing responsibilities: pci_pf.c and pci_vf.c handle PCI lifecycle, netdev.c handles endpoint delivery, devlink.c and eswitch.c expose switch configuration, and tc.c implements flower parsing, actions, and statistics.

The subtle part is representor direction. A packet sent by VF A appears as ingress on representor A when it needs software forwarding. Sending a packet out of representor B delivers it into VF B. An offloaded redirect to representor B must therefore deliver to VF B’s receive path. These directions follow the Linux representor contract.

Both paths reach the same endpoint. A successful terminal rule executes once and does not also send a copy through OVS.

When OVS asks the driver to offload a flower rule, the driver validates its match fields, masks, flags, and actions, then copies the supported information into its own flow state. It keeps no callback-owned flow_rule pointers. Supported terminal actions include redirect and drop; unsupported combinations are rejected with an extended error message rather than silently accepted.

On VF transmit, the mock evaluates the installed rules. A redirect hit delivers the packet to the peer VF, and a drop hit consumes it. A miss enters the source representor’s receive path so ordinary software TC and OVS can forward it. The implementation also has to preserve packet ownership, Ethernet header state, and namespace-safe references during delivery and teardown.

TC reports an accepted rule through its normal offload machinery. The driver supplies execution statistics; it does not manually forge in_hw bookkeeping. Here, in_hw means the kernel accepted the driver’s offload implementation. It does not mean physical silicon processed the packet.

Deploying the virtual test environment

The simplest complete entry point is the SR-IOV Network Operator’s virtual conformance runner, hack/run-e2e-conformance-virtual-cluster.sh.

Run it from a compatible operator checkout on a dedicated test hypervisor. The runner requires its existing virtualization and build tools, including kcli, libvirt utilities, Podman, Go, make, Python, and timeout. It deletes and recreates the cluster named by CLUSTER_NAME before provisioning. Choose a disposable name and ensure its API address and networks do not conflict with another lab.

# From the sriov-network-operator checkout, on the test hypervisor.
export CLUSTER_NAME=switchdev-lab
export MOCK_SMARTNIC_REPO=https://github.com/SchSeba/switchdev-mock-driver.git

# Keep the newly created cluster after the run for inspection.
SKIP_DELETE=TRUE ./hack/run-e2e-conformance-virtual-cluster.sh

SKIP_DELETE prevents cleanup at exit; it does not prevent the initial deletion of an existing cluster with that name. The runner uses its default ovs-cni image; the ovs-cni fix is available upstream. Record the deployed image digest for repeatable runs.

With MOCK_SMARTNIC_REPO set, each worker gets three emulated igb NICs: management, a native secondary PF for regular conformance, and a third PF for mock switchdev. The runner checks the configured MACs against libvirt XML, resolves the mock NIC to an exact guest PCI address, clones the driver’s main branch, and records the installed commit.

Before deploying the operator, it invokes the driver’s scripts/setup-worker.sh. That script installs missing build prerequisites and OVS, builds against the worker’s running kernel, and binds only the selected PF. It saves the native binding state and persists the module and binding before kubelet and OVS start. This matters because operator reconciliation can restart OVS or reboot a worker.

The installer finishes with zero VFs and legacy mode. It creates no final bridge, representor ports, policy, network, or pods. Those remain owned by the operator and CNI. The runner also enables the operator’s existing DEV_MODE for the emulated NICs.

For an already provisioned mutable worker, the direct deployment command is:

# On the worker, from the driver checkout.
# Replace this BDF with the exact dedicated PF verified in the VM XML.
sudo --preserve-env=SSH_CONNECTION ./scripts/setup-worker.sh \
  --pf 0000:29:00.0 --emulated-pf

Use a disposable guest with console or snapshot recovery, matching kernel headers, and an idle test PF. The --emulated-pf flag is an acknowledgement, not proof of emulation. The installer checks the selected PF and refuses management/used interfaces. Kernel upgrades require rebuilding the module; enforced signing requires an authorized signed module delivery process.

A small operator example: two VFs, two pods

For a focused test on an existing compatible cluster, the driver repository already provides a manifest renderer and verification harness. The cluster needs the operator, Multus, the SR-IOV device plugin, and ovs-cni installed. Label exactly one dedicated worker through the harness configuration so both pods use the same local mock switch.

The central policy looks like this; the BDF is illustrative:

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: mock-smartnic-switchdev
  namespace: sriov-network-operator
spec:
  resourceName: mock_smartnic
  nodeSelector:
    mock-smartnic.test/target: dut
  priority: 10
  numVfs: 2
  nicSelector:
    rootDevices:
      - "0000:29:00.0"
  deviceType: netdevice
  isRdma: false
  linkType: eth
  eSwitchMode: switchdev
  mtu: 1500
  externallyManaged: false
  bridge:
    ovs: {}

This asks the operator to create two kernel netdevice VFs, put the selected PF in switchdev mode, and manage its OVS bridge. The companion configuration enables manageSoftwareBridges in SriovOperatorConfig and supplies a scoped SriovNetworkPoolConfig with hw-offload: "true" and tc-policy: "none".

An OVSNetwork named mock-ovs exposes the resource to the workload namespace and uses host-local IPAM on 198.19.0.0/24, without a secondary default route. Each pod requests and limits one openshift.io/mock_smartnic resource and attaches mock-ovs as net1 through its Multus annotation. The harness renders both UDP echo pods and these supporting objects, so there is no need to maintain a second copy of the full manifest set in this post.

# From the switchdev-mock-driver checkout, with access to the lab cluster.
cp config/lab.env.example config/lab.env
# Edit lab.env: SSH_TARGET, PF_BDF, NODE_NAME, KUBE_CONTEXT,
# and the documented acknowledgements after checking the dedicated lab.

./scripts/lab.sh operator-preflight
./scripts/kube.sh preflight
./scripts/kube.sh render
# Inspect rendered/ for the intended node, PF, namespaces, and images.
./scripts/kube.sh apply
./scripts/kube.sh verify
./scripts/kube.sh collect

The Kubernetes runbook explains the configuration and ownership boundaries. This focused example uses two VFs; the operator’s shared switchdev conformance test uses five.

What counts as success?

Two Ready pods and successful ping are necessary, but software OVS can also deliver packets. Setting hw-offload=true only proves that offload was configured.

The harness checks bidirectional ping and UDP, actual TC in_hw rules, OVS offloaded flows, and increasing mock-engine counters. Inspect the worker’s discovered representor and bridge names rather than guessing them:

# On the worker; use the names discovered in this lab.
sudo devlink port show
sudo ovs-vsctl show
sudo tc -s -d filter show dev <source-representor> ingress
sudo ovs-appctl dpctl/dump-flows type=offloaded

Compare counters before and after traffic. A rule existing with zero packets is weaker evidence than the same rule recording execution. Preserve allocated VF BDFs, TC/OVS output, component revisions, and the driver commit alongside the result.

The project’s validation record reports a full operator conformance run with 47 passed, 0 failed, and 17 skipped, including the shared switchdev test. It also records two same-worker pods passing bidirectional ping with increasing OVS offloaded IPv4 and TC hardware packet counters. These are recorded lab results, not a promise that arbitrary component versions will work: the original integration required operator and ovs-cni fixes.

The validated environment includes QEMU 8.2.0, CentOS Stream 9, kernel 5.14.0-427.el9.x86_64, OVS 3.5.3, and Kubernetes 1.34.2. Additional kernel builds and runtime checks are documented separately. In particular, the operator’s virtual OpenShift runner implements Driver Toolkit/MachineConfig delivery, but runtime validation of that mock-driver delivery path is still pending.

Cleanup and the limits of the mock

For the focused harness, remove its owned Kubernetes resources first:

./scripts/kube.sh cleanup

Wait for the operator to release the bridge, ports, and VFs. When retiring the lab, remove its owned pool configuration as described in the runbook. Then restore the selected worker PF:

sudo --preserve-env=SSH_CONNECTION ./scripts/setup-worker.sh \
  --pf 0000:29:00.0 --emulated-pf --restore

Restore refuses live VFs or used interfaces. Keep the saved binding state and console recovery available if an installation or reconciliation fails.

The current scope excludes external uplink traffic, cross-worker forwarding, CT/NAT, tunnels, and VFIO/DPDK. It cannot validate throughput, latency, firmware behavior, or the capacity of a hardware flow table.

What it gives us is a repeatable way to test the chain from an operator policy to PCI VFs, pod interfaces, representors, and executed TC rules. That brings more integration failures into a disposable virtual lab and leaves the hardware lab for the behaviors that require a real NIC.