dra-multi-network: DRA + NRI MVP Implementation Guide

Status: Implementation specification for github.com/SchSeba/dra-multi-network. The new CRDs, controller, effect runner, and plugin contracts described here require implementation. Examples of project resources are proposed API contracts, not a claim that they are installed today.

Source baseline: Extend the existing dra-driver-sriov code and borrow selected helpers from dranet and the CNI projects.

Required MVP: Official project CRDs, a topology controller, immutable topology revisions, attachment status, a generic effect language, shared macvlan/ipvlan allocations, VFIO, RDMA, InfiniBand, rebinding, PF resets, and switchdev/representor management. Plugins execute inside the node agent without an internal gRPC boundary.

Additional switchdev plugins: Include internal ovs-cni and accelerated-bridge-cni implementations, using their upstream projects as baselines and existing sriovnet helpers for representor lookup. Their typed effects use the same ownership journal as the other plugins.

GA boundary: All plugins will run outside the DRA process through gRPC in GA. Keep a shared, transport-neutral plugin request/result contract now. MVP invokes it as Go calls; allocation, grants, validation, execution authority and the journal remain in the framework.

First POC review: Reboot recovery and broader hardware restoration/composition acceptance, including the final regression repeat, are deferred until after the user’s manual review. Runtime-binary tests and upgrade/migration acceptance remain excluded. Track these separately from delivery of the current POC.

Current acceptance waivers: The user excluded unit tests that launch containerd/CRI-O and upgrade/migration acceptance. RDMA/InfiniBand/switchdev hardware acceptance is waived on the current VM; their implementation and software checks remain required. Keep live verification, including feature-gate checks, separate from deployment hooks.

Companion designs: Composable Networking, Device Discovery, and Framework and Plugin SDK. This guide defines the implementation scope and project identity. It adapts those designs to an in-process runtime.

1. Required outcome and scope

The project must provision secondary networking through DRA and NRI while addressing two core problems:

1. Allocation compatibility: PF handoff excludes its child VFs and shared parent uses. Multiple distinct VFs can coexist. Multiple macvlan shares can coexist, but incompatible macvlan/ipvlan/direct-handoff modes cannot be allocated together on the same endpoint.
2. Validation before mutation: Validate each plugin configuration, typed composition, actual allocation, effect permissions, and hardware dependencies before changing networking. Valid JSON or a successfully parsed CNI configuration does not establish a valid topology.

The deliverable includes a control-plane controller, admission validation, a node agent, generated CRD manifests, builtin plugin descriptors, native effect executors, examples, deployment packaging, and runnable acceptance tests. An offline compiler may support development, but the topology controller is the production compiler and publisher.

1.1 MVP support matrix

Adding shared macvlan/ipvlan requires two additional logical plugins beyond the original four. They share the same process, effect runner, inventory authority, and journal. A shared allocation means multiple claims refer to one parent persona; it does not mean one claim is shared between Pods.

The two switchdev bridge plugins consume a verified host representor separately from the VF endpoint. accelerated-bridge-cni manages approved Linux bridge membership and port VLAN state; ovs-cni manages approved OVS bridge ports through OVSDB. The existing SR-IOV/VFIO allocation path continues to own endpoint handoff and device access.

Every requested feature is an MVP completion gate. Implementation stages deliver them incrementally. A kernel-Ethernet-only implementation is an intermediate milestone, not the completed MVP.

1.2 Valid combinations remain explicit

Broad feature coverage does not mean every combination is meaningful. A VFIO endpoint is not a Linux interface and cannot be a bond member or VLAN parent. InfiniBand/RDMA handles are not Ethernet handles. A PF reset cannot run while another consumer in its reset domain is active. Shared parents remain in the host namespace, and one share cannot rebind or reset the parent.

Publish supported combinations in locked plugin descriptors and policy contracts. Reject invalid combinations during admission/compilation and revalidate hardware-dependent facts at the node. This is validation of the requested features, not permission to omit them.

Primary Pod networking stays under the existing primary CNI. The project must protect its actual dependency closure from all effect types, including reset/rebinding operations that can disrupt more than one visible interface.

2. Repository bootstrap and project identity

2.1 Source baseline and target

The reviewed public source baseline is:

source repository: https://github.com/k8snetworkplumbingwg/dra-driver-sriov
source commit:     f9e0bd8587fa05590bc6d331bedd4c2e71f00965
source module:     github.com/k8snetworkplumbingwg/dra-driver-sriov
source Go:         1.26.0
source Kubernetes modules: 0.36.3
source NRI module:        0.12.3

target repository/module: github.com/SchSeba/dra-multi-network
node binary:              dra-multi-network-node
controller binary:        dra-multi-network-controller
DRA driver name:          dra-multi-network.schseba.github.io
project API group:        networking.dra-multi-network.schseba.github.io
project API version:      v1alpha1
NRI plugin name:          dra-multi-network
default system namespace: dra-multi-network

The new API/driver domains are this guide’s selected project identifiers under the account’s GitHub Pages namespace. Use them consistently in code, CRDs, DeviceClasses, ResourceSlices, status, labels, and tests. This does not make the project APIs upstream Kubernetes APIs.

Source dependency versions are recorded in the pinned go.mod. Preserve useful code and history, then adopt the target module and driver identity. Do not continue publishing the old driver’s name from the new project.

For an uninitialized target repository, bootstrap locally from upstream:

git clone https://github.com/k8snetworkplumbingwg/dra-driver-sriov.git dra-multi-network
cd dra-multi-network
git checkout -b mvp/crd-inprocess f9e0bd8587fa05590bc6d331bedd4c2e71f00965
git remote rename origin upstream
git remote add origin https://github.com/SchSeba/dra-multi-network.git
git status --short
go version
go mod download
make build
make test

If the target already contains work, clone https://github.com/SchSeba/dra-multi-network.git, inspect its instructions/history, and compare it with the source baseline. Do not overwrite it or reset it to upstream. Configuring a remote in the bootstrap instructions is not an instruction to push changes without the project’s normal workflow.

After establishing baseline results:

go mod edit -module github.com/SchSeba/dra-multi-network
rg -n 'github.com/k8snetworkplumbingwg/dra-driver-sriov' \
  --glob '*.go' --glob '*.mk' --glob 'Makefile' --glob '*.yaml'

Update local Go import prefixes, Makefile module references, generators, binary names, image/chart names, and project identifiers. Preserve original license notices. Run code generation, go mod tidy, build, and tests after the rename. Do not let unchanged local imports resolve against the old remote module by accident.

A changed driver identity is a migration boundary: existing claims allocated to sriovnetwork.k8snetworkplumbingwg.io remain owned by that driver. Drain them before moving the same hardware to dra-multi-network. Do not run both authorities over the same devices or publish alias identities to make old claims appear compatible.

2.2 Kubernetes target

The expanded sharing model requires compatibility groups. Target a pinned Kubernetes 1.37 build with:

DynamicResourceAllocation=true
DRAPartitionableDevices=true
DRAConsumableCapacity=true
DRADeviceCompatibilityGroups=true

In the inspected v1.37 source, compatibility groups are alpha and disabled by default; partitionable devices and consumable capacity are beta from 1.36. Verify the actual API-server and scheduler configuration, API round trips, and scheduler behavior. Sources: feature registration, compatibility-group proposal.

Upgrade the source’s Kubernetes Go modules coherently to the matching v0.37.x line, its k8s.io/kubernetes dependency to the matching v1.37.x, and envtest assets to that release. Review controller-runtime compatibility and pin compatible tools. The inspected Kubernetes v1.37.0 go.mod declares Go 1.26.0. Record the exact working dependency/toolchain set in the implementation.

No silent fallback may remove compatibility groups while exposing conflicting modes. A cluster without these prerequisites does not support this full MVP. List-valued plugin attributes are optional; the initial implementation can publish one scalar plugin/mode per persona without requiring DRAListTypeAttributes.

2.3 Reference source snapshots

Clone references alongside the target, outside its Go module:

git clone https://github.com/kubernetes-sigs/dranet.git
git clone https://github.com/k8snetworkplumbingwg/sriov-cni.git
git clone https://github.com/k8snetworkplumbingwg/bond-cni.git
git clone https://github.com/containernetworking/plugins.git cni-plugins

Pin reviewed commits or record the reviewed replacement versions. Ported files retain applicable Apache-2.0 headers and a provenance comment with repository, path, commit, and adaptations. Do not copy another driver’s entire go.mod or import a CLI’s package main as a library.

3. Architecture and ownership

Administrators
  DeviceExposurePolicy + NetworkPlugin + NetworkTopology CRDs
                                |
                                v
Control plane: dra-multi-network-controller + admission
  strict validation + typed DAG + effect authorization
  -> immutable NetworkTopologyRevision
  -> root DeviceClasses
  -> topology readiness and protected Pod binding

Node plane: dra-multi-network-node
  PCI/netlink/RDMA/devlink inventory + policy reconciliation
  -> personas/capacity/compatibility domains -> ResourceSlices
  kubelet DRA -> validate allocation/revision -> prepare allocation roots
  NRI -> verified sandbox -> execute effect programs -> readiness gate
  journal/recovery -> restore owned state and reconcile incomplete effects
  -> NetworkAttachmentStatus + scoped ResourceClaim observations

One process hosts all internal plugins and native effect executors.
Kubelet DRA and NRI keep their normal upstream protocols.

The controller owns desired plans and immutable revisions. The scheduler owns allocation. The node agent owns hardware discovery, local preparation, attachment, device injection, durable outcomes, and recovery. API status reports those outcomes; it is not a substitute for the journal.

The controller and node agent may be built from one repository and image, with separate commands. Use controller-runtime from the base for reconciliation and admission. The controller Deployment uses leader election. The node DaemonSet has one hardware authority and one state directory per node.

3.1 Reuse existing code

Keep the source’s kubelet registration, client setup, resource publisher helper, host providers/mocks, health endpoints, CDI patterns, policy reconciliation scaffolding, and image/chart/test workflows. Rework the raw-CNI branch into an opt-in INTERNAL mode. Legacy CNI execution may remain isolated for migration tests; the new topology path never executes CNI binaries or loads a NAD chain.

The base already has kernel binding, VFIO device-node/CDI, and RDMA helper paths. Wrap useful mechanics in authorized, journaled effects rather than copying their entire preparation flow. See base device preparation and host helpers.

Borrow strict decoding and bbolt transactions from dranet, plus scoped host-device/RDMA operations. Its saved configuration database is not already a write-ahead effect journal. Its CreateContainer callback is not already this project’s readiness gate. Sources: validation, storage, RDMA namespace helpers.

Use existing netlink, sriovnet, strict JSON, and CDI dependencies first. Add bbolt for synchronous local transactions and compatible CEL/schema tooling where required. Native effects use Go/netlink/sysfs/devlink APIs; no shell-command effect executor.

3.2 Initial package organization

cmd/dra-multi-network-node/
cmd/dra-multi-network-controller/
cmd/network-topology-compile/         # optional offline validator using shared compiler
pkg/api/networking/v1alpha1/         # project CRD types
pkg/controller/                     # topology/plugin/policy/GC reconcilers
pkg/admission/                      # config, revision immutability, protected Pod binding
pkg/topology/                       # strict config, typed graph, canonical compilation
pkg/pluginapi/                      # versioned transport-neutral calls/effects; no Kubernetes or DRA imports
pkg/effects/                        # language types, validation, runner, native executors
pkg/networkplugins/                 # eight in-process implementations/descriptors
pkg/devicestate/                     # retained PCI-first inventory and personas
pkg/nri/                            # attach/remove, Synchronize, device access/readiness
pkg/journal/                        # preparations, attachments, effects, shared references
config/crd/bases/                    # generated official project CRD YAML
examples/                           # actual CRs, workloads, negative examples

Adapt the source layout where practical. These are responsibilities, not a requirement to create empty scaffolding. A local SDK defines typed plugin calls and effect programs; it has no transport adapters or service discovery in this MVP.

Plugin implementations receive one normalized configuration and explicitly granted opaque input/prepared references. They return effect programs or an original-execution cleanup declaration. They cannot query ResourceClaims, publish devices, choose allocations, access clientsets, write the journal or manufacture PCI/namespace authority. The framework binds and checks references, declared output types, schemas, descriptor versions, budgets and live ownership before execution. Keep builtin planning outside the DRA packages and keep the SDK free of Kubernetes dependencies. GA replaces the invocation transport for every plugin with gRPC; it preserves these responsibilities and uses the same versioned contract. Do not add the GA transport or service scaffolding to MVP.

Use this boundary for all four lifecycle hooks, including compensation after a failed operation. Cleanup carries an opaque reference to the saved execution; it cannot supply fresh device targets or deletion effects. Resolve it against the framework-owned revision, allocation fingerprint and journal. Enforce the separation with a dependency check: the shared API and internal plugin planners must not import DRA, Kubernetes, netlink, runtime or journal packages.

4. Official project CRDs

All project resources use networking.dra-multi-network.schseba.github.io/v1alpha1. Ship generated structural CRDs, typed Go APIs/deepcopy code, status schemas, admission rules, RBAC, and Helm installation. Example custom resources alone do not satisfy this requirement.

Use status subresources and standard conditions with observedGeneration. Define list-map keys for steps (id), root bindings (rootStep), conditions (type), and status outputs (step,port). Preserve meaningful ordering inside effect programs. Enforce bounded arrays and payloads.

4.1 Schema and admission requirements

Generate apiextensions.k8s.io/v1 CRD manifests from project Go types and check them into config/crd/bases. Use structural schemas with enums, required fields, defaults, ranges, and x-kubernetes-validations for local invariants. Enable status subresources and printer columns for readiness, revision, and attachment phase.

Plugin config and effect payloads vary by descriptor. Preserve unknown fields only within those explicitly bounded JSON-object payload fields, then validate them against the locked descriptor/executor schema in fail-closed admission and compilation. Do not enable unknown-field preservation for the entire CRD. Unknown fields outside the payloads must be rejected with strict request validation; payload validation rejects unknown and duplicate fields before compilation. Do not rely on API-server pruning as semantic validation.

Admission verifies cross-resource plugin references, typed graph semantics, effect budgets, and immutable revision/status identity. Compilation repeats those checks so bypassed or previously stored objects cannot become executable. Node preparation repeats allocation/hardware-dependent checks.

Set webhook failurePolicy: Fail for topology/plugin validation, revision/class immutability, and required runtime binding. Ship its Service, certificates/rotation instructions, deployment readiness, and narrowly scoped rules. Use the project’s existing certificate mechanism if available; otherwise include an operator-managed certificate setup. An unavailable webhook must not admit an unvalidated new plan.

Revision spec is immutable after creation. Its digest must match canonical spec content. The topology controller is the only revision/DeviceClass writer. Plugin descriptors referenced by active revisions are retained or fully captured in those revisions. Descriptor updates create a new contract digest and revision, never reinterpret existing preparations.

4.2 Required resource relationships

DeviceExposurePolicy -> eligible personas and permitted effect grants
NetworkPlugin        -> strict config/ports/effect/state contracts
NetworkTopology      -> compiled NetworkTopologyRevision
Revision             -> one DeviceClass per allocation root
DeviceClass          -> class-origin TopologyRootConfiguration
ResourceClaim        -> actual allocated identities/shares/capacity
Pod + sandbox        -> NetworkAttachmentStatus
Local journal        -> authoritative effect progress and recovery

A namespaced attachment status can have a Pod owner reference for API cleanup. Pod deletion and garbage collection of that status do not delete local recovery evidence. The node must complete cleanup from its journal even if all reporting objects disappear.

The source SriovResourcePolicy/DeviceAttributes implementation supplies migration and reconciliation code. DeviceExposurePolicy is the canonical new project policy API. Do not apply both policy systems to the same hardware. Document an explicit migration/import procedure and preserve legacy APIs only in isolated legacy deployments.

4.3 Complete plugin descriptors

Ship one builtin NetworkPlugin CR per logical plugin. Generate their schemas/contracts from the same Go descriptor definitions used by the node so the installed contract cannot drift from its implementation. A builtin’s immutable handler/version and descriptor digest must match the node’s compiled descriptor; a manifest cannot grant a builtin effects it does not implement.

This VLAN descriptor illustrates the project shape:

apiVersion: networking.dra-multi-network.schseba.github.io/v1alpha1
kind: NetworkPlugin
metadata: {name: vlan-cni}
spec:
  implementation: {type: Builtin, handler: vlan-cni}
  contractVersion: v1
  stepKinds: [Operation]
  configSchema:
    type: object
    additionalProperties: false
    required: [ifName, mtu, up, vlanId]
    properties:
      ifName: {type: string, minLength: 1, maxLength: 15}
      mtu: {type: integer, minimum: 576, maximum: 9000}
      up: {type: boolean}
      vlanId: {type: integer, minimum: 1, maximum: 4094}
  inputs:
    parent:
      type: KernelInterface/v1
      linkType: ethernet
      minItems: 1
      maxItems: 1
      access: CreateChild
  outputs:
    interface:
      type: KernelInterface/v1
      linkType: ethernet
      minItems: 1
      maxItems: 1
      ownership: Created
  allowedEffects: [linux/create-vlan/v1, linux/link-configure/v1]
  stateContract: {version: v1, cleanup: RecordedEffects}

Common framework validation also enforces the name restrictions and graph/ownership checks in section 5. Native executors validate the same fields after binding; a broad descriptor schema never overrides a narrower executor or hardware contract.

For implementation.type: EffectProgram, spec.hooks contains versioned effect programs for the supported hooks. The same config/port/effect/state descriptor fields are required. Operation descriptors cannot implement root preparation hooks. Allocation descriptors must define supported preparation and cleanup behavior and their exclusive/shared allocation contract. Cleanup references recorded effects rather than accepting a caller-written list of delete targets.

Implement descriptor condition rules for mode-dependent ports: kernel mode exports interface, VFIO exports device, and an approved RDMA kernel mode additionally exports rdma. Evaluate conditions against strictly validated config during compilation; record the resolved port contract in the revision. A descriptor cannot advertise all outputs unconditionally and leave callers to discover missing ports at runtime.

5. Topology API, typed composition, and validation

5.1 Example NetworkTopology

apiVersion: networking.dra-multi-network.schseba.github.io/v1alpha1
kind: NetworkTopology
metadata:
  name: dual-vf-bond-vlan
spec:
  effectBudget:
    allow:
      - linux/link-move/v1
      - linux/link-configure/v1
      - linux/create-bond/v1
      - linux/attach-master/v1
      - linux/create-vlan/v1
  steps:
    - id: left
      kind: Allocation
      pluginRef: {name: sriov-cni}
      config:
        mode: kernel
        linkType: ethernet
        ifName: vfleft
        mtu: 1500
        up: true
        rdmaAccess: false
    - id: right
      kind: Allocation
      pluginRef: {name: sriov-cni}
      config:
        mode: kernel
        linkType: ethernet
        ifName: vfright
        mtu: 1500
        up: true
        rdmaAccess: false
    - id: uplink
      kind: Operation
      pluginRef: {name: bond-cni}
      inputs:
        members:
          - {from: left.interface}
          - {from: right.interface}
      config:
        ifName: bond0
        mtu: 1500
        up: true
        mode: active-backup
        monitorIntervalMs: 100
        failOverMac: active
    - id: tenant
      kind: Operation
      pluginRef: {name: vlan-cni}
      inputs:
        parent: {from: uplink.interface}
      config: {ifName: tenant100, mtu: 1500, up: true, vlanId: 100}

Allocation roots have no topology input ports: their input authority is the DRA allocation. Derived operations consume explicit inputs.<port>.from references. The array order is presentation order; references determine graph order. Step IDs are unique lower-case DNS-style identifiers without dots, up to 40 characters. The compiler rejects cycles and unresolved references.

Allow up to 8 roots, 32 topology steps, and 256 effects per hook program initially. Bound the normalized topology plus locked contracts to 256 KiB. These are project validation limits, not Linux maximums. Required-field presence must be retained during typed decoding, including explicitly false booleans; do not mistake Go zero values for supplied values.

5.2 Typed outputs and grants

Handles include stable inventory identity, allocation/share provenance, owner step/scope, node/boot/namespace identity, and allowed effect grants. Names, BDFs, ifindexes, and paths are observations, not authority. VFIO mode exports device; it does not export interface. RDMA-capable kernel roots may export both interface and rdma after their association is verified.

A descriptor defines ports/cardinality and accesses such as Read, CreateChild, ExclusiveConfigure, and ExclusiveMaster. A shared-parent grant permits child creation, not parent configuration. Effects can only use the intersection of descriptor permissions, topology budget, exposure-policy grants, verified allocation, and live ownership.

5.3 Validation rules

Use stable reason codes and field paths, for example E_PORT_TYPE, E_ROOT_COVERAGE, E_EFFECT_DENIED, E_FAMILY_CONFLICT, E_SHARED_PARENT_MUTATION, and E_RESET_DOMAIN_BUSY. Return meaningful errors from admission, topology status, DRA preparation, and attachment status.

5.4 Topology controller algorithm

Watch NetworkTopology and its referenced NetworkPlugin contracts, plus policy changes relevant to validation. Reconcile deterministically:

1. Fetch the topology by UID and current generation; strictly validate config and references.
2. Resolve complete plugin descriptors and schemas. Lock contract versions/digests and native executor contract versions.
3. Type-check the DAG, root allocation requirements, sharing/capacity expectations, allowed effects, and workload constraints.
4. Normalize defaults and compute a semantic SHA-256 digest over the source topology UID, normalized spec, locked descriptors, effect contracts, and compiler contract version. Exclude mutable status, timestamps, and resourceVersion. Preserve arrays whose order is meaningful; sort maps and sets.
5. Create or verify an immutable revision named from the topology and digest. Detect abbreviated-name collisions using the full digest.
6. Generate one root DeviceClass per revision. Its selectors require this driver, appropriate persona/plugin/mode, allocation readiness, and any administrator hardware constraints. Its opaque config binds the exact revision UID/digest and root.
7. Verify every generated class exists with the expected content, then mark the revision Ready and update topology status with observedGeneration, revision ref, and root/class list.
8. A subsequent semantic change creates a new revision/classes. Existing allocated claims and sandboxes remain pinned to their original revision.

An invalid update does not rewrite a Ready revision or its classes. Report the invalid current generation while retaining earlier revisions needed by existing allocations. Reconciliation retries can adopt identical previously created objects; conflicting objects are errors, not silently overwritten.

The offline network-topology-compile command uses the same library to validate CRs and preview normalized revisions. It is not the production publisher and does not emit plan ConfigMaps. Examples must work by applying NetworkTopology CRs and waiting for controller readiness.

5.5 Revision retention and deletion

Put a project finalizer on revisions. Retain a revision while a ResourceClaim allocation references it or any node reports a live/prepared/cleanup-pending attachment. Retain its locked plugin/effect contracts as well. Do not use an age-based TTL to remove active cleanup contracts.

Garbage collection requires verified absence of allocations and healthy node reconciliation confirming no relevant local reservations. An unreachable node is unknown, not proof of release. Topology deletion stops new publication but cannot revoke the saved cleanup authority. DeviceClasses remain protected against edits; delete unused generated classes only through the controller’s drain/GC logic.

6. Generic effect language and native runner

The effect language is a versioned, serializable program interpreted by the node runner. Builtin plugins produce these programs; administrator-defined NetworkPlugin objects may supply EffectProgram hook templates using the same registered opcodes. Both paths receive identical validation and permissions. No arbitrary Go closures, shell commands, unchecked sysfs paths, or runtime code loading appear in an effect payload.

6.1 Program contract

{
  "apiVersion": "effects.dra-multi-network.schseba.github.io/v1alpha1",
  "effects": [
    {
      "id": "create",
      "kind": "linux/create-vlan/v1",
      "inputs": {"parent": {"from": "call.inputs.parent"}},
      "parameters": {"ifName": "tenant100", "vlanId": 100, "mtu": 1500},
      "requires": [{"kind": "same-sandbox"}, {"kind": "create-child-grant"}],
      "postconditions": [{"kind": "vlan-parent-and-tag-match"}],
      "recovery": {"strategy": "ObserveAndReconcile"}
    },
    {
      "id": "state",
      "kind": "linux/link-configure/v1",
      "inputs": {"interface": {"from": "effects.create.interface"}},
      "parameters": {"up": true},
      "requires": [{"kind": "owned-link"}],
      "postconditions": [{"kind": "link-state-match"}],
      "recovery": {"strategy": "ObserveAndReconcile"}
    }
  ],
  "exports": {"interface": {"from": "effects.create.interface"}}
}

This JSON describes a local hook program, not another Kubernetes API object. An effect’s inputs are typed references to verified call grants or earlier effect outputs. Parameters are literals or typed template references to validated config fields. Validate schemas before substitution and after binding. Exported values retain ownership/provenance when later effects update the same resource.

Use explicit parameterBindings for templates, separate from literal parameters. For example, parameterBindings.ifName.from: call.config.ifName binds the validated config string to the executor’s ifName field. A field cannot appear in both maps. Type-check bindings against the config and opcode schemas, then validate the fully bound payload. Never interpolate a string into a path, command, or JSON fragment.

The reference environment is closed: call.allocation, call.prepared, call.inputs.<port>, call.config.<field>, call.sandbox, and effects.<earlier-id>.<port>. Availability depends on the hook; NodePrepare has no guaranteed sandbox/Pod value. Handle references carry grants and provenance rather than exposing a raw PCI/namespace path as mutation authority. Preconditions and postconditions also use registered versioned predicate contracts.

Require unique effect IDs, known versioned opcodes, backward-only local references, bounded execution, and declared exports. Use deterministic sequential effect execution within a hook initially. The topology coordinator orders hooks across steps. Do not introduce arbitrary loops, unbounded expressions, or commands to make the language generic.

6.2 Executor contract

Every opcode implementation supplies:

The runner cannot invent an inverse for arbitrary hardware actions. A reset has no inverse. Binding/eswitch transitions may recreate interfaces, so restoration requires rediscovery rather than replaying obsolete ifindexes. Executors declare these facts, and the compiler rejects programs with no supported cleanup/recovery contract.

6.3 Required effect vocabulary

Restore/delete behavior belongs to the original effect’s recorded contract. A cleanup program references the original operation ID and completed/attempted effects; it cannot supply fresh arbitrary delete targets.

6.4 Plugin hook facade

Keep exactly four mutating lifecycle hooks, as ordinary Go calls returning effect programs:

Planning is side-effect-free. The coordinator verifies the complete topology and hook program budgets before any mutation. For hardware discovery or runtime-resolved parameters, bind verified call values and validate the complete bound local program before executing it. Persist the exact program, contract digests, authorizations, and before-images.

Read-only Describe/Validate/Observe operations and container readiness checks are separate from these hooks. No protobuf plugin API is needed for in-process execution. Use the larger SDK design’s typed plan/runner concepts with one journal authority rather than coordinator/plugin RPC journals.

7. Inventory, policies, shared capacity, and exclusion

7.1 PCI-first inventory and protection graph

Discover PCI functions even when bound to VFIO, unbound, or absent from the host netdevice namespace. Join optional netdevice, RDMA, devlink/eswitch and representor observations. Track parent PF/VF index, software lower/master relationships, IOMMU groups, reset domains, and eswitch/controller scopes distinctly.

Persist stable inventory identity and current locators. A BDF alone is not replacement-proof identity; combine node UID, hardware identity, parent/controller observations and verified incarnation. Missing queries are unknown, not evidence of freedom. Moving a netdevice or rebinding a function must not erase an active allocation from publication.

Protect management/primary-CNI interfaces and their dependency closures. A bridge member, VF sibling, RDMA HCA port, representor or reset-domain peer can be affected even if it is not the literal effect target. Policy grants cannot override protected or externally owned dependencies.

7.2 DeviceExposurePolicy example

apiVersion: networking.dra-multi-network.schseba.github.io/v1alpha1
kind: DeviceExposurePolicy
metadata:
  name: pf-macvlan
spec:
  nodeSelector:
    matchLabels:
      dra-multi-network.schseba.github.io/node-pool: secondary
  priority: 200
  selector:
    kind: PF
    pciAddresses: ["0000:18:00.0"]
  action: Expose
  exposure:
    persona: macvlan
    plugin: macvlan-cni
    mode: bridge
    allowMultipleAllocations: true
    capacity:
      children: {value: "64", default: "1", validValues: ["1"]}
    compatibility:
      - {scope: Interface, domain: interface-mode, groups: [macvlan]}
      - {scope: SriovFamily, domain: pf-access, groups: [host-retained]}
    effectPermissions:
      - linux/create-macvlan/v1

Policy selectors operate on normalized inventory facts and support explicit kind/PCI/parent/vendor/driver/link-type fields. Optional CEL selectors can follow the companion vocabulary, but must be compiled/type-checked against a documented environment. Conflicting equal-priority persona policies are errors; exclusions prevent new exposure without erasing active declarations.

A hardware policy separately authorizes allowed drivers, reset scopes, VF settings, eswitch modes, representor host-bridge targets, and RDMA access/namespace profiles. These are typed fields, not arbitrary paths or strings passed to a command. Define corresponding status conditions for unsupported hardware and changes pending drain.

Policies are administrator-owned. A workload topology can request only permitted operations and personas. Reconcile the desired hardware baseline during approved quiescent maintenance; apply allocation-specific changes only within an exclusively owned scope.

7.3 Persona model

Publish one canonical DRA identity for each distinct allocation contract on an inventory resource. A persona is an alternative view of the same hardware, not extra hardware capacity. Examples are PF kernel handoff, PF VFIO handoff, PF macvlan parent, PF ipvlan parent, VF kernel handoff, and VF VFIO handoff.

Use reserved scalar attributes under dra-multi-network.schseba.github.io:

pciAddress, pfPciAddress, inventoryID, plugin, mode, linkType,
rdmaCapable, allocationReady, policyDigest

Names derive deterministically from inventory identity and persona and remain pinned while active. DeviceClasses select exact plugin/mode/link type and allocationReady=true, plus approved hardware selectors. Custom attributes cannot override identity, capacity, groups, or effect eligibility.

7.4 Consumable capacity for shared children

A shared macvlan/ipvlan persona has allowMultipleAllocations: true and a local children capacity. Default and valid request amount are one unit per allocated result. Each root request still has count 1; that result carries an actual share ID and consumed capacity.

Preparation keys include (claim UID, allocation digest, driver, pool, device, shareID, rootStep). Attachment records include the exact share. One Pod’s cleanup deletes its own child and releases its own entitlement, never the common parent or another share’s state.

Persist parent baseline/profile and share references centrally in the node journal. On restart, reconstruct references from live allocations and saved preparations; a zero in-memory counter is not proof that the parent is unused. Do not rebind, reset, move, or globally retune a shared parent from a per-share hook.

7.5 Compatibility groups replace the exclusive-only slot model

The earlier PF=N/VF=1 model covered exclusive endpoints only. Use compatibility-group intersections for this expanded MVP, together with consumable local capacity for repeated shares.

Separate domains scope separate constraints. PF shared-parent uses can coexist with suitable child VF allocations. PF handoff excludes all family host-retained uses. Macvlan and ipvlan on the same lower endpoint have disjoint interface groups. Shared aliases with conflicting kernel/VFIO modes must also be excluded.

Kernel and VFIO handoff personas use different interface groups even though both use the family pf-handoff group. Otherwise two exclusive DRA identities for one PF could be allocated together. The same distinction applies in a VF-local interface domain when both VF personas are exposed. Exclusive allocation of each persona alone does not exclude its physical aliases.

For each compatibility domain publish a bookkeeping counter members with total at least the number of published/retained persona identities referencing it. Each identity has fixed consumption one. This is a counter-set association, not a per-share quota; local children capacity accounts for repeated allocations. Pin declarations and totals until drained. Real-scheduler tests must establish repeated-share and release behavior on the selected build.

Kubernetes v1.37 represents groups in device.consumesCounters[].compatibilityGroups. The inspected API allows at most two consumption entries per device and two groups per entry. Do not flatten a third independent domain into group tokens unless the encoding is proven equivalent. Source: v1.37 ResourceSlice API.

IOMMU/reset/eswitch scopes can extend beyond a PF family. Apply quiescent execution checks for destructive management operations. If concurrent steady-state allocations require a third unrepresentable scheduler domain, reject that publication profile rather than accepting it and relying on routine prepare failures. Certify hardware profiles with a representable isolation model.

7.6 ResourceSlice shape

Counter-bearing and device-bearing slices are separate. This shortened example illustrates PF handoff and shared PF macvlan; production includes the remaining personas/VFs and verified hardware attributes.

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata: {name: worker-a-network-counters}
spec:
  driver: dra-multi-network.schseba.github.io
  nodeName: worker-a
  pool: {name: worker-a, generation: 1, resourceSliceCount: 2}
  sharedCounters:
    - name: pf0-interface
      counters: {members: {value: "4"}}
    - name: pf0-family
      counters: {members: {value: "8"}}
---
apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata: {name: worker-a-network-devices}
spec:
  driver: dra-multi-network.schseba.github.io
  nodeName: worker-a
  pool: {name: worker-a, generation: 1, resourceSliceCount: 2}
  devices:
    - name: pf0-kernel
      attributes:
        dra-multi-network.schseba.github.io/plugin: {string: host-cni}
        dra-multi-network.schseba.github.io/mode: {string: kernel}
        dra-multi-network.schseba.github.io/linkType: {string: ethernet}
        dra-multi-network.schseba.github.io/allocationReady: {bool: true}
      consumesCounters:
        - counterSet: pf0-interface
          counters: {members: {value: "1"}}
          compatibilityGroups: [kernel-handoff]
        - counterSet: pf0-family
          counters: {members: {value: "1"}}
          compatibilityGroups: [pf-handoff]
    - name: pf0-macvlan
      allowMultipleAllocations: true
      attributes:
        dra-multi-network.schseba.github.io/plugin: {string: macvlan-cni}
        dra-multi-network.schseba.github.io/mode: {string: bridge}
        dra-multi-network.schseba.github.io/linkType: {string: ethernet}
        dra-multi-network.schseba.github.io/allocationReady: {bool: true}
      capacity:
        dra-multi-network.schseba.github.io/children:
          value: "64"
          requestPolicy: {default: "1", validValues: ["1"]}
      consumesCounters:
        - counterSet: pf0-interface
          counters: {members: {value: "1"}}
          compatibilityGroups: [macvlan]
        - counterSet: pf0-family
          counters: {members: {value: "1"}}
          compatibilityGroups: [host-retained]

Use the source publisher helper for complete pool generations and correct slice counts/API limits. Pin live identities, memberships, and groups across rescan, movement, policy updates, driver restarts, and scheduler restarts. Inspect the selected allocator’s use of live slices and allocation state rather than assuming compatibility survives removal of active declarations.

For cleanup-pending released allocations, retain their declarations and set allocationReady=false on affected endpoints and incompatible peers. API publication and claim release are not atomic; node ownership checks must prevent reuse during lag or API outage. A missing claim is not permission to forget a still-attached share.

Policy/driver/VF-count/eswitch changes require a drain of their affected scopes. Block new workloads, retain active accounting, finish cleanup, verify all allocated-but-not-prepared claims and node reservations are gone, then reconcile hardware and publish a coherent generation.

8. Internal plugin behavior and hardware profiles

8.1 Shared configuration conventions

All configs are validated against the locked plugin schema. Interface names are 1–15 ASCII characters from [A-Za-z0-9_.-], excluding ., .., lo, eth0; require global plan uniqueness and live namespace collision checks. MTU bounds depend on Ethernet versus IPoIB/hardware profile. Require explicit MTU/up state for kernel interface creation; VFIO configs reject them.

Kernel roots use mode: kernel, linkType: ethernet|infiniband, optional approved driver, rdmaAccess, and hardware profile refs. VFIO roots use mode: vfio, an approved driver: vfio-pci, and accessBackend: group|cdev; they cannot carry Ethernet interface config. The descriptor defines the exact required/optional fields and profile support.

Hardware changes are typed policy-approved profile settings. A config cannot manufacture a host PCI address, namespace path, IOMMU peer, RDMA device name, representor, or bridge target. Resolve those from verified allocation/inventory/policy handles.

8.2 host-cni and sriov-cni

host-cni handles eligible non-VF/PF resources. sriov-cni handles VFs and verifies parent/index. Both support kernel Ethernet, certified kernel InfiniBand/RDMA, and VFIO modes through typed contracts. Keep shared-parent implementations in macvlan/ipvlan plugins.

NodePrepare validates all roots, reserves their scopes, snapshots original binding/VF/eswitch/device state, and builds permitted preparation effects. Kernel connect moves the allocated netdevice and applies its sandbox config. VFIO connect verifies device-access readiness without expecting a netdevice. Removal and unprepare restore the recorded effects in dependency order.

Adapt SR-IOV CNI state/setup. Its allocator/config loader is not pure validation, and temporary namespace movement needs persistent ownership before embedding. Do not call CmdAdd/CmdDel or reuse the CNI allocator alongside DRA.

8.3 bond-cni and vlan-cni

Start with active-backup bonds, 2–8 Ethernet members, explicit monitoring interval 1–10000 ms, and failOverMac: active. Validate exact member identities, shared/exclusive accesses, equal MTUs, and no existing unrelated master. Do not invent physical diversity: two VFs from one PF still share a failure domain. Administrative selectors can require distinct domains.

VLAN supports 802.1Q IDs 1–4094, an Ethernet parent already in the same sandbox, MTU no greater than parent, and unique tag/parent pairs. No host-name parent lookup or implicit IPAM. Remove VLAN descendants before bond/master teardown.

Port minimal native code from bond-cni and VLAN CNI. Wrap every mutation in effects and read-back; a successful opaque helper call alone is not recovery evidence.

8.4 macvlan-cni and ipvlan-cni

These are allocation roots backed by shared parent personas. NodePrepare holds one validated DRA share/capacity entitlement and records the parent baseline. Connect creates one owned child from that entitlement, configures it, and moves/creates it in the verified sandbox. The lower remains in the host namespace.

Initial modes are macvlan bridge and ipvlan L2. Validate Linux parent-mode restrictions, MTU, MAC behavior, and inherited profile. Children can become VLAN parents; bonding them requires an explicitly certified descriptor/access profile and is not assumed from an Ethernet type alone.

Read baseline mechanics from macvlan CNI and ipvlan CNI. Narrow CLI/IPAM/global-cache behavior to typed effects. Cleanup removes only this share’s child. Parent/profile restoration occurs only after the last verified allocation and pending child operation is gone, if the project changed the baseline at all.

8.5 VFIO and device rebinding

Discover prebound VFIO and unbound functions without a netdevice. Before binding, verify IOMMU group membership and ownership of every affected function. A group must be exclusively authorized; do not expose unallocated siblings or infer that an unbound sibling is free. Support isolated functions first, and multi-function groups only when the complete ownership/isolation contract is proven.

Snapshot original driver, driver_override, device identity, readable VF settings, and dependent endpoints. Journal unbind/bind operations separately, restrict target drivers to policy allowlists, wait for verified readiness, and rediscover nodes/interfaces. Restore original binding on unprepare after all sandbox/device users stop. Do not automatically bind “default” without saving what the original state was.

Reuse the base’s VFIO/CDI mechanics. Group backend needs only the authorized /dev/vfio/<group> and control node; cdev backend additionally requires a certified kernel/runtime/application and verified cdev/iommufd association. Validate actual character-device identity/major/minor. Never grant wildcard /dev, whole unrelated groups, or filesystem paths copied from config.

VFIO isolation and cdev/group semantics come from Linux VFIO documentation. A VFIO userspace application may itself reset a device; certify the reset/isolation implications of that access profile, not just the preparation syscall.

8.6 RDMA and InfiniBand

Join PCI functions to RDMA devices, ports, link layers, character devices and namespace capabilities. RDMA capability is not a guarantee that every interface sharing a PCI function can be moved independently. Discover and protect HCA-wide dependencies.

Kernel RDMA roots may export a verified network interface and RDMA handle. Associate the RDMA device with the sandbox where the kernel/profile supports exclusive namespace movement; use scoped NETLINK_RDMA handles and persist original association. Inject only allocated RDMA character-device access using CDI/runtime integration, and verify device nodes again at container creation.

For InfiniBand, distinguish verbs-only access from certified IPoIB netdevice handoff. Preserve typed IB link properties and ports/P_Key profile state where modified. Never apply Ethernet VLAN/bond/MAC semantics to an IB root. A hardware profile must identify the fabric dependencies and exact port/device ownership needed for namespace handoff.

If multiple ports/functions share one HCA that cannot be isolated by the requested namespace/access model, allocate the complete protected scope or reject the combination before mutation. Do not expose every uverbs/umad node simply because one VF is RDMA-capable. RoCE and InfiniBand acceptance tests are both required; IPoIB needs its own kernel/fabric test where supported.

8.7 PF resets, switchdev, and representors

Implement supported reset and eswitch operations, with explicit policy grants and effect contracts. Discover reset and controller domains separately from PF/VF ancestry. Before any destructive transition, prove no active allocation, preparation, sandbox, external owner, or primary-network dependency would be affected beyond this operation’s exclusive grant.

Reset is not reversible. Persist intent, quiescence evidence, expected rediscovery identity/profile and recovery strategy. A crash after an uncertain reset outcome must not trigger blind repeated resets; observe and reconcile the expected safe state, or retain an unresolved record for maintenance investigation. Interface names/ifindexes can change after reset or binding.

Use devlink APIs to reconcile supported eswitch mode/profile, save prior mode/settings, rediscover VF/representor mappings, and verify the result. Source semantics: Linux eswitch attributes. Transitions requiring reload/reset or VF reprovisioning run only under a fully drained approved scope. This functionality is part of MVP management, not a live per-share shortcut.

Map representors using controller/port/PF/VF identity from devlink/sysfs, never a name-pattern guess. The representor remains a host resource. Support approved link/MTU/VF-state configuration and association with an administrator-approved Linux bridge target. Record original membership/state and restore only owned changes. External OVS/OVN integration can later add adapters; do not silently treat a sandbox bond as a host representor bridge.

A kernel-VF, VFIO or whole-HCA RoCE allocation in switchdev mode may carry an associated HostRepresentor grant. Preserve the distinction between the VF endpoint and its representor. Certify dataplane forwarding under the selected host-bridge/fabric profile and retain mappings while the workload is active.

Implement accelerated-bridge-cni and ovs-cni as internal operations consuming HostRepresentor/v1. Their output remains a host representor; it cannot be used as a sandbox VLAN parent or bond member. Bridge targets and allowed port settings come from official exposure-policy CRD grants, never arbitrary workload names or database paths. One representor cannot be attached through both plugins or consumed twice for host-master configuration.

The initial Linux bridge implementation uses exposure.allowedHostBridges entries containing name, an administrator alias, and permitted vlans. The approved existing bridge has VLAN filtering, a fixed administrator-set MAC, an up link and the configured representor MTU. Record and recheck its incarnation and settings; do not change shared bridge attributes as a side effect of attaching a port. Record the representor’s original membership, MTU, state, alias and VLAN table. Keep the port down while installing access/trunk VLANs, reject foreign changes on replay/removal, and restore the original recorded state. The certified initial membership is absent or already on the approved target; migration between bridge targets, shared uplink VLAN changes and controller mode changes require additional explicit grants and before-images.

The implemented mode-management API is DeviceExposurePolicy.action: ManageEswitch with management.profile: ethernet-pf-vfs-v1, desired mode, both restoration allowedModes and explicit linux/eswitch-configure/v1 permission. It requires an exact PF selector, a whole-node DRA drain and cordon, and a publication fence before native mutation. The journal captures controller/function/VF before-images; mode completion is observed, never blindly repeated after an uncertain setter. Deletion or edits restore the original mode under a renewed drain. The initial local Ethernet profile requires default unowned representors and records their port before-images; customized representors are rejected before mode mutation. It rejects registered RDMA HCAs, auxiliary/SF/remote and sibling-controller domains; it does not implicitly reset, reload, rebind or reprovision VFs. Inline/encapsulation attributes must remain unchanged.

For shared Linux-uplink VLANs, an allowedHostBridges[].uplinks[] grant identifies an existing bridge member by name, administrator alias and canonical PCI address, with permitted VLAN IDs. accelerated-bridge-cni-uplink is a profile of the same plugin family and emits a separate linux/shared-uplink-vlan/v1 effect before representor attachment. Both the topology budget and exposure permissions must allow it. Per-VID before-images and attachment references live in the framework journal. Preserve preexisting administrator VLANs; remove a driver-created VID only after its last reference is released and no foreign bridge-port dependency remains. Lost acknowledgements reconcile saved intent; foreign flags or changed identities stay reserved. OVS shared uplinks remain administrator configured.

Reuse sriovnet representor lookup helpers, then independently verify the controller/PF/VF/port tuple, current netdevice and bidirectional PCI association. Use accelerated-bridge-cni pkg/manager as the Linux bridge/port-VLAN baseline, retaining original membership, MTU, link state and VLAN entries. Shared uplink VLAN changes require their own explicit grant and recorded reference ownership. Use ovs-cni pkg/sriov and pkg/ovsdb as the OVS baseline. Native OVSDB transactions must capture bridge/interface/port UUIDs and per-operation external IDs, reject foreign ports, recover lost acknowledgements by ownership identity and remove only the recorded owned rows/membership. OVSDB is an external dataplane API; it does not add an internal plugin gRPC boundary or a shell-command effect executor.

9. Bind revisions to DRA allocations

9.1 Opaque root configuration

Add a project TopologyRootConfiguration decoder type. It is a DeviceClass parameter object, not another CRD:

{
  "apiVersion": "networking.dra-multi-network.schseba.github.io/v1alpha1",
  "kind": "TopologyRootConfiguration",
  "revisionRef": {
    "name": "dual-vf-bond-vlan-<digest-prefix>",
    "uid": "<revision-uid>",
    "digest": "<full-sha256>"
  },
  "rootStep": "left",
  "pluginContractDigest": "<locked-contract-sha256>"
}

Angle-bracket values are explanatory placeholders; the controller emits real values. Revision references pin name, UID, digest, and contract. Recreated same-name objects cannot change the allocation’s authority. Save exact revision bytes and contracts locally before preparation effects.

Generated class shape for the Ethernet VF root:

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: dual-vf-bond-vlan-left-<digest-prefix>
spec:
  selectors:
    - cel:
        expression: >-
          device.driver == "dra-multi-network.schseba.github.io" &&
          device.attributes["dra-multi-network.schseba.github.io"].plugin == "sriov-cni" &&
          device.attributes["dra-multi-network.schseba.github.io"].mode == "kernel" &&
          device.attributes["dra-multi-network.schseba.github.io"].linkType == "ethernet" &&
          device.attributes["dra-multi-network.schseba.github.io"].allocationReady == true
  config:
    - opaque:
        driver: dra-multi-network.schseba.github.io
        parameters:
          apiVersion: networking.dra-multi-network.schseba.github.io/v1alpha1
          kind: TopologyRootConfiguration
          revisionRef:
            name: dual-vf-bond-vlan-<digest-prefix>
            uid: <revision-uid>
            digest: <full-sha256>
          rootStep: left
          pluginContractDigest: <locked-contract-sha256>

For shared roots, classes and request examples include the required local children capacity contract. Validate generated CEL against actual device attributes and missing-attribute cases. Preserve meaningful administrator hardware selectors without allowing them to replace mandatory plugin/mode constraints.

9.2 Allocation checks

Use upstream resourceclaim.ConfigForResult applicability rules. Accept root bindings only from allocation configuration with Source=FromClass. The source driver’s generic claim-over-class VfConfig precedence must not become topology authority.

Before any preparation effect, verify all driver-owned results:

1. They belong to this node/pool and retained inventory and have applicable class-origin bindings.
2. Bindings reference one Ready revision and cover every allocation root exactly once.
3. Actual persona/plugin/mode/capacity/share semantics match the locked contract and policy.
4. Distinct roots do not alias the same exclusive function or duplicate one shared entitlement. Separate legal shares on a shared parent have distinct share IDs and reserved units.
5. PF/child, kernel/VFIO, macvlan/ipvlan and hardware isolation constraints hold across concurrent allocations and retained cleanup records.
6. The exact allocation, revision, policy grants and executor contracts are supported by this agent build.

Support Exactly/count 1 per root, including one unit of shared capacity where applicable. Reject adminAccess, All, firstAvailable, claim-origin root overrides, legacy raw CNI config, and mixed topology revisions in this initial profile. Other drivers’ results may coexist in the claim; leave their configuration and status intact.

Hash a canonical allocation fingerprint including request/result identities, driver/pool/device/shareID, consumed capacities, revision/root bindings, and access semantics. Do not omit shareID when indexing preparation or status. Multiple roots may share a permitted physical lower only when their allocation contracts and entitlements explicitly allow it.

Kubelet preparation has no guaranteed Pod metadata. Do not require a resolved Pod UID to complete NodePrepare. Validate the claim/allocation and record reservations; verify the actual consumer, Pod UID, hostNetwork policy, and sandbox association at NRI attachment. Reject multiple consumers for this project’s one-Pod-per-claim profile without assuming the first reservation entry is the Pod.

9.3 Workload and admission

Users apply NetworkTopology and wait for its Ready revision/root classes, then reference those generated classes from a ResourceClaimTemplate. This sample uses the controller’s actual class names in place of placeholders:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata: {name: dual-vf-network, namespace: default}
spec:
  spec:
    devices:
      requests:
        - name: primary
          exactly:
            deviceClassName: dual-vf-bond-vlan-left-<digest-prefix>
            allocationMode: ExactCount
            count: 1
        - name: backup
          exactly:
            deviceClassName: dual-vf-bond-vlan-right-<digest-prefix>
            allocationMode: ExactCount
            count: 1
---
apiVersion: v1
kind: Pod
metadata: {name: network-mvp, namespace: default}
spec:
  resourceClaims:
    - name: secondary
      resourceClaimTemplateName: dual-vf-network
  containers:
    - name: app
      image: registry.k8s.io/pause:3.10
      resources:
        claims: [{name: secondary}]

Request names may differ from root IDs; binding configuration resolves them. Admission resolves actual claims/templates/classes, requires one consistent project revision, injects/protects required NRI participation, and adds revision-capable node affinity. Node readiness comes from verified agent contract/hardware profiles, not a tenant annotation. Direct nodeName binding cannot bypass those checks.

The node reports a bounded protected capability annotation on its Node containing node/boot/agent-build identity, supported descriptor/executor contract digests, NRI connection/recovery readiness, and certified hardware-profile digests. The controller verifies the agent Pod’s identity/readiness and publishes protected per-revision capability labels for admission’s affinity. Stale/unavailable agents lose eligible labels; tenant writes are denied. These reports are scheduling hints with a defined refresh/expiry policy, while the live node and required-runtime gates remain authoritative during propagation lag.

Implement protected admission for init, application and ephemeral containers. Workload users cannot remove runtime requirements, choose unapproved device-access injection, or set bootstrap toleration. Existing live workloads stay pinned to their original revision after topology updates.

10. Runtime lifecycle, CDI, readiness, and cleanup

10.1 NodePrepare

Validate the full revision and driver-owned allocation before mutating any root. Under the node mutation fence, persist preparation intent, exact revision/contracts, bindings, permissions, before-images, and share/domain reservations. Plan all root hooks and validate their combined hardware effects.

Execute root preparation programs in deterministic root order. Binding/reset/eswitch dependencies can impose an additional validated order. Persist outcomes synchronously and return successful kubelet device results/CDI IDs only after all required preparation and access descriptors are verified. A retry with the same fingerprint observes/reuses existing work; a changed live allocation is an error.

On partial failure, reverse attempted reversible effects. Non-reversible/ambiguous effects remain reserved pending inspection. Never update the in-memory cache to Prepared before its transaction commits.

10.2 RunPodSandbox

Resolve the actual Pod by runtime identity and verify UID/node, claim references (including generated claims), single-consumer relationship and hostNetwork policy. Distinguish a Pod with no project claims from one whose required preparation is missing. Only the former may receive a no-op.

Open and verify the runtime-supplied network namespace, reject the host namespace, pin it durably, and persist (node UID, boot ID, Pod UID, sandbox ID) attachment identity. A missing namespace is an error when the contract needs one. VFIO-only workloads still require verified sandbox/consumer identity and device readiness.

Preflight the entire plan against live links, parent profiles, RDMA associations, device-node identities and representors. Execute roots and derived steps topologically through the effect runner. Commit Ready synchronously only after all required exports, access descriptors and postconditions are verified.

A new sandbox ID for the same Pod is a new incarnation. It cannot inherit Ready or steal exclusive roots from an old live sandbox. Share child ownership is per attachment. A repeated callback for the same sandbox inspects/resumes the original operation rather than creating duplicate links or device access.

10.3 Container device access and readiness

Use the base’s CDI integration for prepared VFIO/RDMA device access, narrowed to verified allocated resources and intended containers. NRI finalizes/validates container-specific namespace/access requirements before start. Never trust a remembered device path after rebinding or reset; verify current device identity and major/minor.

CreateContainer requires a Ready attachment for this exact sandbox, boot, Pod, claim fingerprint, revision and contract. Include init, ordinary, and ephemeral containers. When supported, StartContainer repeats the gate to catch readiness invalidation between creation and start. ResourceClaim status, attachment CR status, Pod annotations, and a PodUID-only cache are not readiness authority.

If the NRI agent is absent, its callbacks cannot reject creation. Configure the pinned runtime’s required-plugin validator for dra-multi-network and verify rejection when the plugin is killed. Sources: NRI mandatory plugin checks, containerd configuration. Record exact runtime/build/configuration and callback ordering in the implementation acceptance evidence.

10.4 Stop, Remove, Unprepare

StopPodSandbox durably invalidates Ready and records teardown intent. Return/reset/rebind resources only after application/device users are stopped. RemovePodSandbox, or recovery with verified quiescence, runs reverse attachment cleanup: derived children/VLANs, bond membership/bond, RDMA association/representor changes, exclusive root movement or share child removal.

NodeUnprepare records the claim-release request. If any sandbox/device user or uncertain effect remains, return a retryable error and retain reservations. After attachment cleanup, restore preparation effects, remove owned CDI artifacts, and release only this root/share. Parent restoration and mode changes require all relevant shares and dependent records to be gone.

Cleanup targets attempted effects even when their acknowledgement was lost. It uses saved programs/contracts/before-images if topology, revision, plugin, Pod or claim objects disappear. Deletion or timeout is never by itself proof that hardware is free.

11. Durable journal and recovery

11.1 One node authority

Use a synchronous bbolt store in a root-owned persistent host directory:

/var/lib/dra-multi-network/state.db
/var/lib/dra-multi-network/netns/<attachment-id>

Mode 0600, synchronous commits, required directory durability, no NoSync, no ephemeral filesystem. A single-writer database lock and deployment fencing prevent two node agents from controlling the same inventory. Also fence DRA publication/NRI registration; separate database directories do not authorize duplicate authorities.

One context-aware node mutation lock can serialize preparation, effects, cleanup, recovery and eligibility transitions initially. Do not hold it while waiting for runtime synchronization or unrelated status updates. Shared parent/domain reservations live in the store, not an unprotected per-plugin map.

11.2 Required records

Use small explicit preparation and attachment states (Preparing, Prepared, Unpreparing, Released; Connecting, Ready, Removing, Removed) plus effect progress and failure/recovery action. Schema/contract upgrades must preserve old cleanup or require a drain before incompatible changes.

11.3 Effect transaction protocol

validate program and bound authority
commit operation/program intent
for each effect:
  capture and commit complete before-image + Applying intent
  recheck identity, ownership and protected scopes
  perform native action
  observe/read back actual state
  commit outcome and typed output
verify all exports/postconditions
commit completed hook result / whole-topology Ready

Kernel mutation and database commit are not one transaction. Recovery observes a syscall with a lost outcome rather than assuming failure. Use unique ownership aliases for created links, restore original aliases for moved roots, and verify namespace/boot/PCI relationships. An ifindex/name alone cannot authorize adoption or deletion.

For non-replayable reset and eswitch transitions, save quiescence and expected rediscovery state. If outcome is uncertain, do not apply the action again automatically. Reconcile observable postconditions under a renewed quiescent fence or leave an explicit unresolved record. Rebinding/reset can invalidate previous RDMA/representor/CDI observations; rebuild them from verified identity.

11.4 Namespace retention

Open FDs survive only the process lifetime. Before moving hardware or children, bind-mount the verified namespace at the persistent pin path and record mount intent and (device,inode) identity. Include staging namespaces if an adapted helper requires them. A bare pin file is not a namespace mount.

Make pins visible after driver-container replacement using a tested shared mount-propagation arrangement or equivalent host mount-namespace setup. hostPID alone does not provide that guarantee. Retain pins while any owned link/RDMA association needs them; unmount/remove only after verified cleanup.

11.5 Restart and reboot

Before new eligibility/readiness, load the journal and reconstruct reservations, register with NRI, obtain current sandboxes/containers through Synchronize, and reconcile retained namespaces, links, shares, device bindings, RDMA nodes and representors. Unavailable runtime/API state is unknown, not an empty consumer set.

Revalidate surviving Ready attachments before gates accept them. A moved interface absent from the host can belong to a live sandbox. A shared parent with no current in-memory refs can still have allocated shares.

After reboot, invalidate old Ready/namespace/ifindex observations. Rediscover hardware, driver/eswitch/RDMA state and actual runtime users; use saved identity and baselines to reconcile safely. Do not replay old ifindexes or assume volatile mount paths survived. Keep uncertain scopes unavailable until ownership is established.

Status retries and CR deletion must not lose the journal. Existing saved cleanup can continue during API outage; new preparations requiring unavailable authoritative revision/allocation information fail safely.

12. Status, RBAC, deployment, and operations

12.1 Attachment status contract

The node creates one NetworkAttachmentStatus per attachment identity in the Pod namespace. Immutable spec includes Pod UID, sandbox ID, node UID/name, boot ID, claim UID/fingerprint, and revision ref. Node-owned status contains phase/conditions, sanitized output descriptions, last error, and cleanup progress.

VFIO/RDMA outputs report identities and readiness without exposing arbitrary host filesystem details. Shared output status includes shareID. Derived interfaces are reported as topology outputs, not fictitious allocated DRA devices. Also reuse ResourceClaim device-status observations keyed by exact driver/pool/device/shareID, preserving other drivers’ entries.

Status is asynchronous and retryable. Local committed outcomes govern readiness and cleanup. The controller cannot mark a node attachment Ready merely because the topology revision is Ready.

12.2 RBAC and admission separation

Use separate controller and node service accounts. The controller manages project plans/revisions/descriptors/generated classes and admission. Node agents read their required policies/revisions/claims/Pods, publish ResourceSlices, and report owned attachment/device status. Workload users reference approved classes/topologies but cannot edit descriptors, effect permissions, revisions, classes, node readiness, bootstrap tolerations, or journals.

Protect generated DeviceClass config/selector fields through admission. Authorize status ownership against claim/Pod node identity and trusted agent identity; do not use a user-supplied nodeName as a write grant. Keep admission rules off unrelated Kubernetes resources.

12.3 Packaging

Ship:

• Generated CRD bundle and migration notes.
• Controller Deployment with leader election and webhook Service/certificate setup.
• Privileged node DaemonSet with kubelet/NRI/sysfs/CDI access and durable state/pin mounts.
• Builtin NetworkPlugin CRs and hardware DeviceExposurePolicy examples.
• Topology/workload examples for kernel, shared, VFIO, RDMA, IB, switchdev profiles.
• Runtime required-plugin configuration and protected bootstrap admission.
• Diagnostics, drain checks, and exact certified runtime/hardware versions.

No internal plugin sidecars or CNI binary installation are required in INTERNAL mode. Keep kubelet DRA/NRI runtime transports. Rename the chart/image/binaries to dra-multi-network while preserving useful base deployment mechanics.

For a compatible pinned containerd build, the runtime template is:

[plugins.'io.containerd.nri.v1.nri']
  disable = false
  disable_connections = false
  plugin_request_timeout = "10s"

[plugins.'io.containerd.nri.v1.nri'.default_validator]
  enable = true
  required_plugins = ["dra-multi-network"]
  tolerate_missing_plugins_annotation = "dra-multi-network.schseba.github.io/bootstrap-nri-toleration"

Verify these keys, actual annotation propagation, and failure behavior on the certified runtime. Executor deadlines must be below callback timeout; incomplete effects stay journaled. Allow bootstrap toleration only for operator-controlled driver/static/required system Pods and block it for tenants through admission. Otherwise use a host-service bootstrap proven to start the agent before workloads.

12.4 Install and drain

Install on dedicated hardware/node pools. Establish primary-network protection and supported IOMMU/reset/eswitch/RDMA profiles. Install CRDs, webhook/controller, runtime enforcement and node agent. Wait for synchronization/recovery/prerequisite readiness before publishing policies. Apply plugin CRs, policies and topologies; wait for Ready revisions and generated classes before running claims.

During initial cordoned-node smoke tests, grant an administrator-controlled unschedulable toleration rather than bypassing DRA scheduling with nodeName. Uncordon ordinary workloads only after tests pass.

Before destructive policy/driver/eswitch/VF-count/contract updates, block new workloads in affected scopes, retain active declarations, drain claims including allocated-but-not-prepared ones, complete journal cleanup, verify no share/device users or retained pins, then reconcile hardware and republish. Revision/descriptor GC follows verified release. Do not uninstall the agent or delete state while effects remain.

13. Source change map

Paths here start from the pinned dra-driver-sriov tree and are renamed/adapted into the target module.

Add the topology/CRD/effect packages from section 3 without rewriting reusable Linux helpers. Trace the base’s prepare → raw NAD/binding/CDI → PodManager checkpoint and NRI → CNI → async updater flow before replacing it. INTERNAL mode must bypass raw CNI execution and make its synchronous journal the authority.

14. Implementation stages and completion gates

Stage 0 — Bootstrap the target

Establish source baseline build/test evidence, preserve ancestry/licenses, adopt github.com/SchSeba/dra-multi-network, rename identifiers/binaries, and pin the Kubernetes 1.37-compatible dependency/runtime set.

Gate: Reproducible build/tests and explicit source/target migration boundary.

Stage 1 — CRDs, admission, compiler, controller

Implement five official CRDs and schemas, builtin plugin descriptors, strict config/typed graph validation, immutable revision hashing, generated root classes, status and retention. Add controller/admission envtests and actual CR examples.

Gate: Applying a NetworkTopology creates a Ready immutable revision and correct classes; invalid updates never change active contracts. No plan ConfigMaps exist.

Stage 2 — Generic effect language and journal

Implement opcode schemas/types/grants, pure hook planning/templates, native runner protocol, before-images/recovery, persistent namespaces and typed exports. Start with link/bond/VLAN effects, then add hardware/device executors through the same contract.

Gate: Injected crashes between intent/syscall/commit can be reconciled without deleting foreign resources or losing ownership.

Stage 3 — Personas, sharing, scheduler constraints

Implement PCI-first inventory/protection, DeviceExposurePolicy translation, compatible persona domains/capacity, share-aware keys, retained accounting and feature checks. Add real scheduler tests for PF/VF and shared macvlan/ipvlan lifetimes.

Gate: Concurrent compatible shares allocate; incompatible modes never co-allocate; releasing one of multiple shares does not unblock an incompatible mode.

Stage 4 — Kernel and shared runtime networking

Complete six internal plugins’ kernel/shared behavior, NRI callbacks/readiness, controller-protected admission/runtime binding and reverse cleanup. Certify containerd required-plugin behavior.

Gate: Root/bond/VLAN/shared examples work before first init container, including agent restart and sandbox recreation.

Stage 5 — VFIO, rebinding, RDMA and InfiniBand

Implement approved driver transitions, IOMMU ownership, CDI/access backend checks, RDMA namespace/device scoping and IB/IPoIB profiles. Add hardware acceptance and restore/crash tests.

Gate: Actual VFIO, RoCE and InfiniBand workloads receive only their allocated resources; wrong type/isolation configurations fail before mutation; original binding/association restores safely.

Stage 6 — Reset, switchdev and representor management

Implement destructive-operation grants/quiescence, reset recovery/rediscovery, eswitch profile transitions, representor identity/membership and saved restoration. Exercise scopes with protected/external/live peers.

Include the internal OVS and accelerated-bridge operations: validate their typed representor inputs and policy bridge grants, exercise Linux bridge/VLAN and OVSDB lifecycle/restoration, and test lost acknowledgements and foreign membership/row preservation. Hardware-independent checks remain required when actual switchdev forwarding acceptance is waived.

Gate: Approved operations work on certified hardware; busy/protected domains are rejected; crashes never trigger unbounded resets or stale-handle mutation.

Stage 7 — Release packaging and full acceptance

Complete diagnostics, state/CRD migrations, retention/GC, operator drain/install instructions, webhook/runtime bootstrap, generated manifests and all acceptance scenarios below.

Gate: Every required feature has passing evidence on its certified environment. Report unavailable environments explicitly; do not label the expanded MVP complete based only on compiler or kernel smoke tests.

15. Tests and acceptance evidence

15.1 Unit, schema, and controller tests

15.2 Real scheduler tests

Use actual ResourceSlices, classes, claims and scheduler on the supported build:

1. PF handoff blocks child VFs and shared PF users, across claims and within one claim.
2. Distinct VFs coexist; different families remain independent.
3. Multiple macvlan shares allocate until capacity is exhausted.
4. Any active macvlan share blocks ipvlan/direct handoff on that lower endpoint; equivalent tests apply in the reverse direction.
5. Releasing the first of two shares still blocks incompatible mode; releasing the last and completing cleanup allows it.
6. Scheduler/driver restart preserves those constraints and share/capacity accounting.
7. Allocated-but-not-prepared claims still participate in exclusion.
8. Kernel/VFIO personas on one function cannot co-allocate; group/isolation profiles are enforced or publication is rejected.
9. Active policy withdrawal/change retains declarations and requires drain.
10. Disabled/stripped prerequisite fields keep full-MVP publication unavailable.
11. Third unrepresentable domains/tokens are rejected instead of flattened incorrectly.
12. Cleanup-pending released resources remain ineligible; publication-lag allocations fail node checks before mutation.

Synthetic inventory exercises scheduler behavior without physical NICs. It does not certify hardware namespace/reset/isolation behavior.

15.3 Runtime and hardware matrix

Run traffic/device workloads, not just link listings. Disposable namespaces/veth can test mechanics; physical VFIO/RDMA/IB/eswitch/reset features require their real environments. Assign test-only addresses outside this L2 topology in controlled test workloads until IPAM is added.

15.4 Validation commands and release evidence

After implementation, provide working repository commands for:

make generate
make build
make test
make check

Add documented CRD/webhook/controller envtests, scheduler integration, privileged effect tests, and runtime/hardware suites. Run race checks on journals/share reservations/callback coordination. Render/lint the Helm chart and validate generated CRDs/CRs with strict server-side admission on the supported Kubernetes build.

Record source/target commits, dependency lock, compiler/effect contracts, kernel/NIC/driver/firmware, IOMMU/reset/eswitch/RDMA profile, Kubernetes build/gates, containerd config, and image digest. Report what actually ran and any unavailable hardware. Proposed tests are not evidence until executed.

16. Copyable implementation-agent handoff

Implement the full dra-multi-network MVP defined by this guide.

Target repository/module: github.com/SchSeba/dra-multi-network.
Bootstrap from dra-driver-sriov commit
f9e0bd8587fa05590bc6d331bedd4c2e71f00965 if the target is uninitialized;
otherwise inspect and extend existing target work. Preserve source history and
licenses. Adopt the guide's target API, DRA, binary, image and chart identity.
Read repository instructions and establish baseline results before editing.

Implement official DeviceExposurePolicy, NetworkPlugin, NetworkTopology,
NetworkTopologyRevision and NetworkAttachmentStatus CRDs, structural schemas,
admission, a topology controller, immutable revision/descriptor locking,
root DeviceClass generation, status and safe retention. Do not store plans in
ConfigMaps. An offline compiler is only a shared-library development tool.

Implement a generic versioned effect language and native runner with typed
references, schemas, permissions, before-images, postconditions, explicit
compensation/recovery and durable effect progress. In-process plugins return
pure programs; no internal gRPC services or shell-command effects.

Implement host-cni, sriov-cni, bond-cni, vlan-cni, macvlan-cni and ipvlan-cni.
Reuse existing CNI Linux behavior and selected dranet/base helpers. Complete
exclusive and shared networking, VFIO, RDMA, InfiniBand, device rebinding,
PF/function resets and switchdev/representor management in this MVP.

Use shared consumable capacity and scheduler compatibility groups for persona
exclusion on a certified Kubernetes 1.37-compatible build. Pin active
identities/groups, include shareID in ownership, reject unsupported domain
encodings, and never silently downgrade incompatible publication.

Implement NRI sandbox attachment/removal and container readiness/device-access
gates, required-plugin runtime enforcement, persistent namespace pins, one
node journal authority, shared-parent references, hardware-domain protection,
reverse cleanup and restart/reboot recovery. No Pod annotation or async
status write is proof of readiness.

Follow stages 0–7 and finish all feature gates. Add runnable examples,
CRD/controller/webhook/deployment packaging, diagnostics, drain/migration
instructions and the unit/scheduler/runtime/hardware tests in this guide.
Report exact validation evidence and unavailable environments. A compiler or
kernel-only milestone is not the completed expanded MVP.