Cluster Baseline​

Before initiating any node removal, verify that the cluster state is healthy and that all active Kubernetes node objects align 1:1 with their corresponding Cluster API machine objects. If any stale or redundant machine objects exist from previous failed deployments, delete them before proceeding.

The sample cluster has two nodes, and worker node harv21 will be removed.

harv31:/home/rancher # kubectl get nodes -A
NAME     STATUS   ROLES                AGE   VERSION
harv21   Ready    <none>               15m   v1.36.3+rke2r1
harv31   Ready    control-plane,etcd   56m   v1.36.3+rke2r1

harv31:/home/rancher # kubectl get machines -A
NAMESPACE     NAME                  CLUSTER   NODE NAME   FAILURE DOMAIN   READY   AVAILABLE   UP-TO-DATE   PHASE     AGE   VERSION
fleet-local   custom-2b108e6cb5d9   local     harv21                       True    True                     Running   15m   
fleet-local   custom-d9d1ba8f7563   local     harv31                       True    True                     Running   51m

Removing a Control-Plane Node in 3+ Node Clusters​

If your cluster has 3 or more control-plane nodes (e.g., a High-Availability setup with harv11, harv21, harv31), deleting a control-plane node is supported, but extra precautions apply:

• Quorum Health: etcd requires a strict majority to maintain quorum ($N/2 + 1$). Ensure the remaining control-plane count after removal will form an odd number (or maintain quorum requirements) before taking a node down.

• Proactive etcd Leadership Transfer: If the control-plane node scheduled for removal currently holds the active etcd leader role, you MUST explicitly transfer leadership to a surviving control node before executing uninstallation scripts.

Control Plane & ETCD Leadership Transfer (Control-Plane Nodes Only)​

If the target node to be removed holds the control-plane / etcd role, perform leadership verification and manual handover before draining or uninstalling software.

1. Check Current ETCD Leadership

   Log in to any active control-plane node and query the member list inside the static etcd pod:

   harv31:/home/rancher # ETCDCTL_API=3 kubectl exec -n kube-system etcd-harv31 -- etcdctl \
       --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt \
       --cert=/var/lib/rancher/rke2/server/tls/etcd/server-client.crt \
       --key=/var/lib/rancher/rke2/server/tls/etcd/server-client.key \
       endpoint status --cluster -w table
   

   Copy

   The following example output displays the endpoint status for a three-node control plane cluster:

   

   Locate the target node by its IP address in the ENDPOINT column and evaluate the following:

   • If the target control-plane node has IS LEADER set to false, proceed directly to Prepare the Target Node.

   • If the target control-plane node has IS LEADER set to true, proceed next step to transfer leadership.

2. Proactively Move ETCD Leadership

   1. Select a surviving member node from the endpoint status --cluster table that has IS LEADER set to false and IS LEARNER set to false. Record its hexadecimal ID.

   2. Transfer leadership away from the target host to the surviving member ID:

      harv31:/home/rancher # ETCDCTL_API=3 kubectl exec -n kube-system etcd-harv31 -- etcdctl \
          --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt \
          --cert=/var/lib/rancher/rke2/server/tls/etcd/server-client.crt \
          --key=/var/lib/rancher/rke2/server/tls/etcd/server-client.key \
          move-leader <TARGET_SURVIVING_MEMBER_ID>
      

      Copy

   3. Re-run endpoint status --cluster -w table to confirm that IS LEADER has successfully shifted to the surviving node before proceeding.

Prepare the Target Node​

Check the official Harvester Deleting a Node Documentation steps before 5. Evict workloads from the node to be removed.

Whenever possible, enable Maintenance mode on the target node directly from the Harvester UI via Hosts > Action > Enable Maintenance Mode.

If UI maintenance mode is not accessible/not allowed, prepare the node manually using kubectl

# Cordon the target node to prevent new pod scheduling
kubectl cordon harv21

# Verify that the node status reflects 'SchedulingDisabled'
kubectl get node harv21

# Drain active workloads and local data storage from the node
kubectl drain harv21 --ignore-daemonsets --delete-emptydir-data

Verify Post-Maintenance Cluster Health​

Once the target node successfully enters Maintenance Mode (or completes its kubectl drain cycle), do not proceed directly to node uninstallation. Workload migration places extra CPU, memory, network, and disk I/O pressure on surviving nodes. You must verify that the cluster has stabilized before taking irreversible deletion steps.

Perform the following system-level checks on an active control-plane node:

1. Node Readiness & Resource Overhead: Verify that all remaining nodes report Ready and have sufficient CPU and memory headroom to handle the re-located workloads:

   harv31:/home/rancher # kubectl get nodes
   harv31:/home/rancher # kubectl top nodes
   

   Copy

2. Pod & Workload Status: Ensure no evicted pods or virtual machine instances are stuck in Pending, CrashLoopBackOff, or unschedulable states:

   harv31:/home/rancher # kubectl get pods -A | grep -v -E 'Running|Completed'
   

   Copy

3. Abnormal Cluster Events: Check for recent scheduling warnings, resource exhaustion alerts, or failed volume mounts triggered by the eviction:

   harv31:/home/rancher # kubectl get events -A --field-selector type=Warning --sort-by='.metadata.creationTimestamp'
   

   Copy

4. Longhorn Storage Health or Third-Party Storage: In the Harvester UI (or Longhorn dashboard / third-party storage management console), verify that all storage volume replicas have finished rebuilding across the remaining hosts and that no volumes remain degraded, degraded-syncing, or in a Faulted state.

5. Critical System Services & Business Workloads: Confirm that core cluster add-ons and critical business applications have successfully re-established quorum and connectivity:

   • Core Infrastructure: Verify that ingress controllers, CNI networking components (e.g., Canal/Flannel/Cilium), and monitoring stacks are fully functional across surviving nodes.
   • VIP / Ingress Traffic: Ensure VIP services or external load balancer targets have updated and are actively serving production traffic without packet drop or high latency.
   • Stateful Workloads: Validate that stateful applications (e.g., databases, message queues, key-value stores) have re-attached their persistent volumes and recovered cluster synchronization.

6. VM Auto-Balance & Migration Thrashing Check: If the Virtual Machine Auto-Balance addon is enabled, monitor active migrations in the Harvester UI. Pay close attention to ensure that workload shifts do not trigger cascading, hyper-frequent VM re-balancing migrations across the remaining hosts due to tight CPU/memory thresholds.

7. Observation & Stability Window: Allow an observation window (e.g., 30 minutes) before initiating permanent node uninstallation or deletion. Monitor real-time metrics, node load averages, and VM responsiveness to guarantee cluster stability under sustained operational load.

Execute Node Uninstallation Script​

Log in to harv21 via SSH/console as the root user and execute the pre-installed script:

harv21:/home/rancher # hostname
harv21

harv21:/home/rancher # /opt/rke2/bin/rke2-uninstall.sh
++ id -u
...
...
...
+ echo -e '\e[31mCleanup didn'\''t complete successfully\e[0m'
Cleanup didn't complete successfully
+ log 'Removing uninstall script'
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '[2026-08-05 13:25:09] Removing uninstall script'
[2026-08-05 13:25:09] Removing uninstall script
+ rm -f -- /opt/rke2/bin/rke2-uninstall.sh

Verification:

Once the script finishes, verify that RKE2 services and virtual interfaces have been stopped.

1. Check process termination:

   harv21:/home/rancher # ps aux | grep kubelet
   # Expected output: Only the grep process itself should return
   

   Copy

2. Check network interfaces:

   harv21:/home/rancher # ip link
   # Expected output: Only physical interfaces and management bridges remain active
   # no output like `56: cali05b22ce82b3@if2:`
   

   Copy

3. Check Harvester UI.

   The host harv21 displays a red Maintenance status with the warning message Kubelet stopped posting node status.

Remove the Node Object​

To complete the node removal from the cluster, you must explicitly delete the Kubernetes node object using one of the following methods:

Option A: Via Harvester UI (Recommended)

1. Navigate to the Hosts page in the Harvester UI.

2. Locate the target node (harv21).

3. Select ⋮ > Delete.

Option B: Via kubectl CLI

1. Verify the current node list:

   kubectl get nodes
   

   Copy

2. Delete the node object:

   kubectl delete node harv21
   

   Copy

3. Confirm that harv21 is no longer listed in the cluster.

Known Issue 1: Node stuck in "Draining" state during removal​

Symptom

During node deletion, the host status on the UI normally shows Kubelet stopped posting node status.

If the UI displays Node is draining due to kubelet/node not ready, or toggles between Kubelet stopped posting node status and Node is draining due to kubelet/node not ready for more than 5 minutes without removing the host, you have encountered a known issue. Refer to the Node Draining Workaround Guide for resolution steps.

Known Issue 2: Longhorn node resource persists after host removal​

Symptom

After a node is removed from the Harvester cluster, its corresponding Longhorn node resource (nodes.longhorn.io) may persist in the longhorn-system namespace.

When you attempt to manually delete the resource, the Longhorn admission webhook blocks the deletion with an error:

$ kubectl get nodes.longhorn -n longhorn-system
NAMESPACE         NAME     READY   ALLOWSCHEDULING   SCHEDULABLE   AGE
longhorn-system   harv21   False   true              False         13d
longhorn-system   harv31   True    true              True          13d

$ kubectl delete nodes.longhorn -n longhorn-system harv21
The request is invalid: : could not delete node harv21 with node ready condition is False, reason is KubernetesNodeGone, node schedulable true, and 0 replica, 0 engine running on it

Attempts to force-delete the object by manually removing its finalizers (metadata.finalizers)` will also fail, as the admission webhook continues to intercept and reject the update request.

Furthermore, if you re-add the host to the cluster using the same node name while this stale object remains, Longhorn may report that the node's disks are not ready or fail to provision storage pools for the newly added node.

Cause

Longhorn's validation webhook prevents node resource deletion while allowScheduling is set to true (even if the underlying Kubernetes node is gone and no replicas or engines are running on it).

Workaround

1. Disable scheduling on the stale Longhorn node by setting allowScheduling to false:

   $ kubectl patch nodes.longhorn.io harv21 -n longhorn-system --type=merge -p '{"spec":{"allowScheduling":false}}'
   

   Copy

2. Delete the Longhorn node resource:

   $ kubectl delete nodes.longhorn.io harv21 -n longhorn-system
   

   Copy

Post-Removal Disk Cleanup​

The uninstallation script tears down container runtimes and Kubernetes configurations, but it does not wipe local storage drives or bootloader partitions.

If harv21 is rebooted without disk formatting, it will load leftover installation data locally (as shown on its physical or virtual console below). However, this is strictly a local artifact—the Harvester cluster is not affected by it.

• CVE-2026-53359

The mitigation steps involves disabling the nested virtualization feature of the KVM kernel module on your Harvester hosts.

On your Harvester hosts, use the following commands to confirm that the KVM kernel module is loaded with nested virtualization enabled:

lsmod | grep -iE "kvm_amd|kvm_intel"

sudo grep -H '' /sys/module/{kvm_amd|kvm_intel}/parameters/* 2>&1 |grep nested

If the above commands return module and parameter information about the KVM kernel module, then your host is affected by this vulnerability.

Deploy the following CloudInit configuration to disable the nested virtualization feature of the KVM kernel module on all your Harvester hosts:

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: disabled-nested-virtualization
spec:
  matchSelector:
    harvesterhci.io/managed: "true"
  filename: 99-disabled-nested-virtualization
  contents: |
    stages:
      initramfs:
      - name: "disable nested virtualization in kvm modules"
        files:
        - path: "/etc/modprobe.d/99-disabled-nested-virtualization.conf"
          content: |
            options kvm_amd nested=0
            options kvm_intel nested=0

Once the configuration is applied, reboot your Harvester hosts for the changes to take effect.

Once you have upgraded to a fixed version of Harvester, you can re-enable the nested virtualization feature by deleting the CloudInit configuration and rebooting your Harvester hosts:

kubectl delete cloudinit disabled-nested-virtualization

References​

• https://support.scc.suse.com/s/kb/Security-Vulnerability-x86-64-kvm-Guest-to-Host-escape-aka-Januscape-CVE-2026-53359
• https://www.suse.com/security/cve/CVE-2026-53359.html

What is KubeVirt? Harvester uses KubeVirt as its core virtualization engine. KubeVirt is an open-source technology that allows Kubernetes to run and manage traditional Virtual Machines inside standard containers, translating VM specifications directly into Pod configurations.

This article explores why these OOM events occur in a Kubernetes-native virtualization environment and how Harvester provides granular tools to eliminate them.

Anatomy of a VM OOM Event​

When a VM is terminated due to insufficient memory at the host level, the Linux kernel logs specific keywords that help pinpoint the fault. In Harvester, these logs generally fall into two distinct categories depending on which process triggered the exhaustion.

Example 1: virt-launcher invoked oom-killer

The virt-launcher process runs inside the dedicated Kubernetes Pod backing the VM. If this component or its direct sub-processes run out of the memory allocated to their cgroup, the kernel triggers a memory cgroup (memcg) OOM event.

Feb 03 19:57:08 ** kernel: virt-launcher invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=986
Feb 03 19:57:08 ** kernel: CPU: 40 PID: 40785 Comm: virt-launcher Tainted: G          I    X    5.14.21-150400.24.60-default #1 SLE15-SP4 9096397fa6646928cc6d185ba417f2af65b536f1
...

Feb 03 19:57:08 ** kernel: memory: usage 17024340kB, limit 17024340kB, failcnt 1243
Feb 03 19:57:08 ** kernel: memory+swap: usage 17024340kB, limit 9007199254740988kB, failcnt 0                                                                 
Feb 03 19:57:08 ** kernel: kmem: usage 143556kB, limit 9007199254740988kB, failcnt 0
Feb 03 19:57:08 ** kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice:
...
Feb 03 19:57:08 ** kernel: Tasks state (memory values in pages):
Feb 03 19:57:08 ** kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Feb 03 19:57:08 ** kernel: [  38886]     0 38886      243        1    28672        0          -998 pause
Feb 03 19:57:08 ** kernel: [  38917]     0 38917   310400     6921   192512        0           986 virt-launcher-m
Feb 03 19:57:08 ** kernel: [  38934]     0 38934  1200940    25126   954368        0           986 virt-launcher
Feb 03 19:57:08 ** kernel: [  38951]     0 38951   386525     8247   466944        0           986 libvirtd
Feb 03 19:57:08 ** kernel: [  38952]     0 38952    33619     3940   290816        0           986 virtlogd
Feb 03 19:57:08 ** kernel: [  39079]   107 39079  4457263  4201766 34439168        0           986 qemu-system-x86
Feb 03 19:57:08 ** kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice/cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,task=qemu-system-x86,pid=39079,uid=107

Example 2: CPU X/KVM invoked oom-killer

This occurs when a vCPU execution thread inside qemu-system-x86_64 attempts a memory operation that pushes the entire container beyond its Kubernetes memory limit.

[Thu May  9 14:52:38 2024] CPU 11/KVM invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=830
[Thu May  9 14:52:38 2024] CPU: 60 PID: 70888 Comm: CPU 11/KVM Not tainted 5.3.18-150300.59.101-default #1 SLE15-SP3
...
[Thu May  9 14:52:38 2024] memory: usage 67579904kB, limit 67579904kB, failcnt 67391
[Thu May  9 14:52:38 2024] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[Thu May  9 14:52:38 2024] kmem: usage 633636kB, limit 9007199254740988kB, failcnt 0
...
[Thu May  9 14:52:38 2024] Tasks state (memory values in pages):
[Thu May  9 14:52:38 2024] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Thu May  9 14:52:38 2024] [  70675]     0 70675      243        1    28672        0          -998 pause
[Thu May  9 14:52:38 2024] [  70728]     0 70728   310400     5467   188416        0           830 virt-launcher-m
[Thu May  9 14:52:38 2024] [  70746]     0 70746  1242373    25104  1073152        0           830 virt-launcher
[Thu May  9 14:52:38 2024] [  70762]     0 70762   455279    14110   770048        0           830 libvirtd
[Thu May  9 14:52:38 2024] [  70763]     0 70763    37704     3916   339968        0           830 virtlogd
[Thu May  9 14:52:38 2024] [  70870]   107 70870 18302464 16718510 135278592        0           830 qemu-system-x86
[Thu May  9 14:52:38 2024] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,mems_allowed=0-3,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice/cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,task=qemu-system-x86,pid=70870,uid=107
[Thu May  9 14:52:38 2024] Memory cgroup out of memory: Killed process 70870 (qemu-system-x86) total-vm:73209856kB, anon-rss:66852088kB, file-rss:21948kB, shmem-rss:4kB
[Thu May  9 14:52:38 2024] oom_reaper: reaped process 70870 (qemu-system-x86), now anon-rss:0kB, file-rss:132kB, shmem-rss:4kB

Root Cause: Native Hypervisors vs. Harvester Architecture​

Traditional Linux Host (e.g., Virtual Machine Manager)​

On a standard Linux host, a VM managed via QEMU/KVM runs inside a systemd machine.slice. The hypervisor process (qemu-system-x86_64) has access to the host's wider pool of resources, managed loosely unless strict cgroup limits are manually added.

$ systemd-cgls
Control group /:
-.slice
├─1173 bpfilter_umh
├─system.slice

└─machine.slice
  └─machine-qemu\x2d1\x2dharv41.scope 
    └─8632 /usr/bin/qemu-system-x86_64 -name guest=harv41,debug-threads=on -S -…

Harvester (Kubernetes/KubeVirt Engine)​

In Harvester, every VM is encapsulated inside a Kubernetes Pod. This introduces a strict cgroup boundary (kubepods.slice).

As shown below, multiple helper processes must live alongside the primary qemu-system-x86_64 process within the same tightly limited container memory budget:

-.slice
└─kubepods.slice
  │ ├─kubepods-burstable-pod99ee3a64_645b_4699_9384_5a3875d78b41.slice
  │ │ ├─cri-containerd-0c316eb8a4711bff1ce968b46ddb658e49378897454b4caf2a20704c808f33f1.scope …
  │ │ │ └─ 8505 /pause
  │ │ ├─cri-containerd-eb5deef29f064adfd2456d9f9c535674ac4d0c95c81b13afbdab5a89dc6a774b.scope …
  │ │ │ └─ 8590 /usr/bin/virt-tail --logfile /var/run/kubevirt-private/2ce151aa…
  │ │ └─cri-containerd-fd57a5cfc2b9b1f53eaf7b575c3273e6784f4c56a04a17d502ecfbd19e55b066.scope …
  │ │   ├─ 8542 /usr/bin/virt-launcher-monitor --qemu-timeout 301s --name vm2 -…
  │ │   ├─ 8558 /usr/bin/virt-launcher --qemu-timeout 301s --name vm2 --uid 2ce…
  │ │   ├─ 8591 /usr/sbin/virtqemud -f /var/run/libvirt/virtqemud.conf
  │ │   ├─ 8592 /usr/sbin/virtlogd -f /etc/libvirt/virtlogd.conf
  │ │   └─ 8823 /usr/bin/qemu-system-x86_64 -name guest=default_vm2,debug-threa…

The "Hidden" Memory Overhead​

Breaking Down the Memory Overhead Buffer​

When you define a virtual machine, for example, a VM configured with 4 vCPUs, 2 GiB of memory, and 1 Ethernet interface, KubeVirt does not just allocate exactly 2 GiB of memory to the container.

Instead, KubeVirt calculates an additional baseline memory overhead required to operate the virtualization stack. This overhead budget covers:

• CPU Simulators: Thread pools tracking guest state and handling context switches.
• Memory Management: Tracking structures such as QEMU page tables mapping guest RAM.
• Auxiliary Devices: Buffers for virtual network interfaces (NICs), storage queues, and video devices.

The High Stakes of VM OOM Kills​

Depending on the guest OS type, specific kernel workloads, and heavy storage/network I/O spikes, the memory consumed by these helper tasks can quickly exceed KubeVirt's default calculations. Because Kubernetes enforces a strict hard ceiling on the Pod container, the entire container triggers a CONSTRAINT_MEMCG OOM kill the moment this boundary is breached.

Unlike traditional, stateless Kubernetes workloads where a container crash is quickly mitigated by a rapid pod restart, an OOM kill on a VM pod carries severe operational consequences:

• Prolonged Downtime: A virtual machine is a stateful workload. It does not instantaneously serve traffic upon a container restart; it must undergo a full operating system boot cycle, run init scripts, and re-initialize services, drastically extending your Recovery Time Objective (RTO).

• Risk of Data Corruption: Sudden terminations during flight can abruptly cut off active storage queues. If the guest OS or database engine is in the middle of a critical write operation when the host terminates the qemu process, it can result in uncommitted journals, filesystem degradation, or severe data corruption on your persistent volumes.

The Solution: Tunable Memory Architectures​

To address this, Harvester introduced dual-layer configurations that give administrators full flexibility over how overhead buffers are calculated.

Global Adjustment: additional-guest-memory-overhead-ratio​

This cluster-wide setting functions as a multiplier for KubeVirt's automatically calculated memory overhead. For deep structural details, refer to the Harvester Advanced Documentation.

• Definition: Scales the calculated overhead buffer to accommodate heavy I/O or virtualization tasks.
• Default Value: 1.5 (Provides a 50% safety cushion above baseline calculations).
• Valid Range: 0 or 1.0 to 10.0.

💡 Important Operational Notes:

• Lifecycle Impact: Changes to this setting only apply to newly created virtual machines or existing VMs after they undergo a migration or a full power cycle.
• System Overhead: A higher ratio increases the host container's memory allocation, guaranteeing safety for heavy workloads but scaling up the overall system resource reservation footprint.
• Resource Allocation Trade-off: Setting this ratio excessively high can lock up unneeded host memory blocks, leading to predictable underutilization and significant memory waste across your compute nodes.

Per-VM Optimization: Reserved Memory​

For specific virtual machines running intensive or non-standard workloads, a global multiplier might not offer the precision required. Harvester allows administrators to define a dedicated Reserved Memory value directly on individual VMs. For complete configuration steps, see the Harvester VM Management Documentation.

⚠️ Under the Hood Memory Carving: When you configure this setting, Harvester explicitly scales down the available memory presented to the Guest OS inside the VM. For example, if a VM is configured with 2 GiB of memory and you set a Reserved Memory value of 256 MiB, the Guest OS will only see and utilize 1.75 GiB (2 GiB - 256 MiB).

Why Use Per-VM Reserved Memory?​

• Guaranteed Overhead Headroom: By restricting the Guest OS from consuming the top slice of its configured allocation, you guarantee an isolated, un-evictable memory runway for host helper tasks.

• Targeted Safety for Heavy Workloads: This mechanism is highly practical for mission-critical, high-performance, or special-purpose workloads (such as nested virtualization layers or intensive database engines). It effectively prevents the VM from running into host-level cgroup OOM termination by proactively limiting its internal usage boundaries, removing the risk of unexpected node-level kills.

• Optimized Cluster Usability: Using per-VM reservations eliminates the major disadvantage of cranking up the global additional-guest-memory-overhead-ratio for the whole cluster. Instead of forcing a massive, wasteful memory overhead reservation across every idle or lightweight VM on your hosts, you can maintain a lean global default and surgically protect only the heavy workloads—striking an ideal balance between system density and ironclad stability.

Outcome: Guaranteed Workload Stability​

By leveraging this dual-layer tunable memory architecture, Harvester fundamentally alters how host-level overhead is calculated, moving from rigid, generalized defaults to a precise, tiered enforcement model:

Total Memory Overhead = Auto-calculated Overhead * Ratio + Reserved Memory

Best Practices & Configuration Matrix​

The following matrix showcases how combinations of Reserved Memory and the Overhead Ratio change the actual layout of the Guest OS space versus what Kubernetes reserves as a hard boundary.

When optimizing your Harvester cluster to eliminate host-level container OOM events, use the following operational checklist to tailor your memory strategies:

• For General Workloads:

  • Stick to the default ratio of 1.5, or configure a slightly higher value of 2.0. This ensures that standard Guest operating systems receive exactly the memory requested while scaling out a stable, predictable background overhead buffer across the cluster.

• For High I/O and Storage-Heavy VMs:

  • If you observe periodic KVM OOM events during massive backup windows, large-scale data syncs, or intensive disk read/write cycles, increase the individual VM's allocation or implement a targeted Reserved Memory configuration to safely expand the helper overhead pool.

• For GPU Passthrough Workloads:

  • Virtual machines utilizing direct hardware acceleration or GPU passthrough are prime candidates for explicit Reserved Memory carving. The underlying host-side device drivers and memory-mapped I/O (MMIO) windows for high-performance graphics hardware require a significantly higher, specialized memory footprint outside the guest OS space. Allocating dedicated per-VM reserved memory prevents driver-instigated cgroup allocation breaches, keeping both the hardware pipeline and the hypervisor completely stable.

Quick Summary​

• The Problem: In Harvester's Kubernetes-native architecture, every virtual machine is bound by a strict Pod container limit. While this rigid cgroup boundary is essential for security, ensuring a single rogue or leaking VM can never starve neighboring workloads or crash the bare-metal host, it means heavy storage/network I/O, device drivers, or GPU passthrough can cause internal helper processes to breach this hard ceiling, triggering a sudden host-level OOM kill.

• The Solution: Harvester eliminates these crashes without losing secure resource control using a dual-layer memory tuning strategy:

  • Globally: The additional-guest-memory-overhead-ratio scales out a safety cushion cluster-wide for newly created or migrated VMs.
  • Per-VM: The Reserved Memory setting surgically carves out a chunk of the VM's configured RAM exclusively for background helper tasks—preventing wasteful memory reservations across the cluster while safely anchoring high-performance, mission-critical workloads.

Appendix: Lab Simulation — Manually Triggering the Host-Level OOM​

For engineers looking to validate this behavior safely in a staging environment, you can replicate this multi-process cgroup breach. A detailed script and case study can be found in the Harvester Development Summary: OOM Investigation.

The simulation process highlights a fundamental truth about modern virtualization boundaries:

• The Guest OS is Trustworthy: Testing shows that modern guest operating systems handle internal resource limits reliably. If a runaway application inside the guest OS eats up all available RAM, the guest kernel safely steps in and kills that specific process internally. The VM itself survives, and from the host's perspective, the virtual machine continues running normally.

• The Host Cgroup Boundary is the Weak Link: The true host-level crash only happens if processes inside the host cgroup expand unexpectedly. If an infrastructure task or helper process inside the Pod container balloons, it consumes the memory buffer that KubeVirt set aside, causing the entire cgroup, the VM's carrier, to slam into the hard Kubernetes ceiling and trigger a host-level OOM kill.

• CVE-2026-42945
• CVE-2026-42946
• CVE-2026-40701
• CVE-2026-42934

You can confirm the version of the RKE2 ingress-nginx pods by running this command on your Harvester cluster:

kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'

If the command returns one of the affected versions, perform the following mitigation steps.

The primary resolution is to upgrade Harvester to one of these versions:

• 1.7.2 or newer
• 1.8.1 or newer

If upgrade is not possible, apply the following mitigation to protect your clusters.

All ingress resources with the nginx.ingress.kubernetes.io/rewrite-target annotation containing ? in the annotation value are at risk.

By default, Harvester does not include any ingress resources with this annotation. Run the following command on your clusters to identify affected custom ingress resources:

kubectl get ingress -A -o json | jq '.items[] | select(.metadata.annotations["nginx.ingress.kubernetes.io/rewrite-target"] // "" | contains("?")) | {namespace: .metadata.namespace, name: .metadata.name, rewrite: .metadata.annotations["nginx.ingress.kubernetes.io/rewrite-target"]}'

Any ingress resources reported by the above command are vulnerable. They should be updated to either remove the vulnerable annotation or change the annotation value to not contain a question mark ?.

The following validating admission policy can be applied to your cluster to reject ingress resources with the vulnerable configuration:

cat<<EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: ingress-nginx-annotation-validation-20260514
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["networking.k8s.io"]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["ingresses"]
  validations:
    - expression: |
        !has(object.metadata.annotations) ||
        !object.metadata.annotations.exists(k, k == 'nginx.ingress.kubernetes.io/rewrite-target') ||
        !object.metadata.annotations['nginx.ingress.kubernetes.io/rewrite-target'].contains('?')
      message: "Ingress resources with 'nginx.ingress.kubernetes.io/rewrite-target' annotation containing '?' in the annotation value are not allowed, due to the following CVEs: CVE-2026-42945, CVE-2026-42946, CVE-2026-40701, CVE-2026-42934"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: ingress-nginx-annotation-validation-20260514
spec:
  policyName: ingress-nginx-annotation-validation-20260514
  validationActions: [Deny]
EOF

The policy can be removed once you upgrade to Harvester 1.7.2, 1.8.1 or newer:

kubectl delete validatingadmissionpolicy ingress-nginx-annotation-validation-20260514

kubectl delete validatingadmissionpolicybinding ingress-nginx-annotation-validation-20260514

References​

• https://depthfirst.com/research/nginx-rift-achieving-nginx-rce-via-an-18-year-old-vulnerability
• https://www.suse.com/security/cve/CVE-2026-42945.html
• https://www.suse.com/security/cve/CVE-2026-42946.html
• https://www.suse.com/security/cve/CVE-2026-40701.html
• https://www.suse.com/security/cve/CVE-2026-42934.html

The admin password for the Harvester GUI can be reset if you can still login via ssh as the rancher user, or if you have the Harvester cluster's kubeconfig file saved locally. For details, see How can I reset the administrator password? in the documentation.

You have lost the rancher user's ssh/console login password​

But you still have the Harvester cluster's kubeconfig​

The rancher user's ssh/console login password can be reset by creating a CloudInit CRD to update the password. For details, see Password of user rancher in the documentation. Alternately you can create additional users with ssh access as described in How to create an SSH user for Harvester nodes.

You don't have a kubeconfig, but do have physical or remote console access​

In this case, the rancher user's ssh/console login password can be changed by editing /oem/90_custom.yaml on each host.

If you can boot the Harvester installer ISO​

Boot the Harvester installer, but don't proceed with the regular installation process. Instead, once the installer comes up, press CTRL-ALT-F2 to switch to VT2 and login as the rancher user with password rancher. Then proceed with the following steps:

1. Run sudo -i to become root
2. Mount the COS_OEM partition from the host:

   # mkdir /tmp/oem
   # mount -L COS_OEM /tmp/oem
   

   Copy
3. Run vim /tmp/oem/90_custom.yaml to edit 90_custom.yaml and change the password for the rancher user. You can specify either a plaintext password (not recommended) or a password hash generated with e.g. openssl passwd -6. Following is the section that you need to edit:

   users:
     rancher:
       passwd: <PASSWORD_GOES_HERE>
   

   Copy
4. Reboot the host. You should now be able to log in as the rancher user with the new password.

You can't boot the Harvester installer ISO, but can still reboot the host and access the boot menu​

If you have no other option, then during system boot, edit the grub config and add rd.break at the end of the kernel command line (the one that starts with linux). This will drop you into the dracut emergency shell, with the root partition mounted under /sysroot. Unfortunately, this shell does not contain any text editor programs. Trying to edit /oem/90_custom.yaml under the circumstances would be unwise. Instead though, you can run this command:

# sed -i 's%rancher.*%rancher:$6$j0.h3TQv8RZPHJkB$3SbV978JLT2Qeq4KSCBZitErNlZZGfrDxnGW5HS0wHzWexGyPzeQBoQmQJetUhLFfquv/X5VWL6odxtlEec1u/:20468::::::%' /sysroot/etc/shadow

Then, hit CTRL-D to continue, and once the system finishes booting, the rancher user's password will be set back to rancher for this boot only. You can then login on the console and use vim to update /oem/90_custom.yaml and permanently set the password to something more secure as described in the previous section.

• CVE-2025-15566
• CVE-2026-1580
• CVE-2026-24512
• CVE-2026-24513
• CVE-2026-24514

You can confirm the version of the RKE2 ingress-nginx pods by running this command on your Harvester cluster:

kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'

If the command returns one of the affected versions, perform one of the following mitigation steps.

The primary resolution is to upgrade to Harvester 1.7.1 or newer, which includes the fixed RKE2 ingress-nginx controller.

If upgrade is not possible, deploy the following validating admission policy to your cluster to reject ingress resources with the vulnerable configuration:

cat<<EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: ingress-nginx-annotation-validation
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["networking.k8s.io"]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["ingresses"]
  validations:
    - expression: |
        !('nginx.ingress.kubernetes.io/auth-proxy-set-headers' in object.metadata.annotations) &&
        !('nginx.ingress.kubernetes.io/auth-method' in object.metadata.annotations) &&
        (object.spec.rules.all(rule, rule.http.paths.all(path, path.pathType != 'ImplementationSpecific')))
      message: "Ingress resources with the vulnerable annotations are not allowed. Please remove the 'nginx.ingress.kubernetes.io/auth-proxy-set-headers' and 'nginx.ingress.kubernetes.io/auth-method' annotations, and avoid using the 'ImplementationSpecific' path type."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: ingress-nginx-annotation-validation
spec:
  policyName: ingress-nginx-annotation-validation
  validationActions: [Deny]
EOF

This validating policy rejects any ingress resources that contain the:

• nginx.ingress.kubernetes.io/auth-proxy-set-headers annotation
• nginx.ingress.kubernetes.io/auth-method annotation
• ImplementationSpecific path type

The policy can be removed once you upgrade to Harvester 1.7.1 or newer:

kubectl delete validatingadmissionpolicy ingress-nginx-annotation-validation

kubectl delete validatingadmissionpolicybinding ingress-nginx-annotation-validation

References​

• https://github.com/kubernetes/kubernetes/issues/136789
• https://github.com/kubernetes/kubernetes/issues/136677
• https://github.com/kubernetes/kubernetes/issues/136678
• https://github.com/kubernetes/kubernetes/issues/136679
• https://github.com/kubernetes/kubernetes/issues/136680
• https://support.scc.suse.com/s/kb/Upstream-Ingress-Nginx-CVEs-CVE-2025-15566-CVE-2026-1580-CVE-2026-24512-CVE-2026-24513-and-CVE-2026-24514

Node selector constraints can prevent the scheduler from live-migrating a virtual machine to a target node. This often indicates a mismatch between the virtual machine's requirements and the node's capabilities.

A node selector may require a specific CPU feature, but the target node lacks the corresponding label (for example, cpu-feature.node.kubevirt.io/fpu: "true"). This mismatch can occur when the host-model CPU models and features computed by KubeVirt change over time.

Solutions​

You can resolve this issue using four different approaches.

• Reboot the virtual machine.

  KubeVirt automatically adds node selectors (during a previous migration or initial start) that can restrict scheduling. You can clear these node selectors by rebooting the virtual machine.

• Reboot the virtual machine and set up a common CPU model.

  You can override KubeVirt's default host-model CPU configuration by setting up a common CPU model for virtual machine migration. The model is applied to the virtual machine as its domain CPU, and to the pod as its node selector configuration.

  This is the recommended approach for environments that can tolerate restarting of virtual machines.

• Modify the node labels.

  If rebooting the virtual machine is not an option, you can manually manipulate the target node's labels to satisfy the scheduling requirements.

  1. Add the node-labeller.kubevirt.io/skip-node="true" annotation to the target node.

     This annotation, which persists even after upgrades, prevents KubeVirt's node-labeller from automatically adding or removing CPU-related labels on this node.

     kubectl annotate node <node-name> node-labeller.kubevirt.io/skip-node="true"
     

     Copy

     important

     The annotation itself does not affect the pod's node selector. It only controls the presence of specific CPU-related labels on the node, which the node selector checks against. For more information, see the References section.

  2. Identify labels that are missing from the virtual machine's node selector and add them to the target node.

     You can add the missing labels using the following command:

     kubectl label node <node-name> <key>=<value>
     

     Copy

     This circumvents the standard scheduling restrictions, allowing the virtual machine to migrate to the target node.

  If a new node that lacks the required features is added to the cluster, you must repeat these steps to allow the virtual machine to live-migrate to that node.

• Remove the node labels.

  If you want to ensure that the virtual machine does not acquire specific node selector constraints after live migration, you can remove the relevant CPU labels from the target node.

  1. Add the node-labeller.kubevirt.io/skip-node="true" annotation to the target node.

     This annotation, which persists even after upgrades, prevents KubeVirt's node-labeller from automatically adding or removing CPU-related labels on this node.

     kubectl annotate node <node-name> node-labeller.kubevirt.io/skip-node="true"
     

     Copy

     important

     This method works only if the virtual machine's pod does not have an existing node selector that contains the labels listed in the References section. Otherwise, you must reboot the virtual machine to clear the constraints.

  2. Check if the pod has a node selector.

     kubectl get pod <pod-name> -o yaml | grep nodeSelector -A 5 -B 5
     

     Copy

  3. If no node selector exists, remove the relevant CPU labels from the node.

     Performing this action prevents the pod from acquiring new node selector constraints, thus enabling its future migration to other nodes. However, the successful outcome of that migration is not guaranteed.

References​

KubeVirt Node Labels​

The KubeVirt CPU node-labeller manages the following labels:

• cpu-feature.node.kubevirt.io/*
• cpu-model-migration.node.kubevirt.io/*
• cpu-model.node.kubevirt.io/*
• host-model-cpu.node.kubevirt.io
• host-model-required-features.node.kubevirt.io

External Links​

• Labeling nodes with CPU models, CPU features and machine types

Problem​

Symptoms​

In the testing environment, guest virtual machines experienced severely degraded download and transfer speeds, dropping as low as 100 bps. This extreme slowdown was particularly evident when apt-get update, curl, and scp were used to transfer data between virtual machines running on different nodes. In contrast, performance remained normal when the virtual machines were hosted on the same node.

Environment​

The issue was observed in a Harvester cluster hosted on Dell servers using Broadcom NetXtreme-E Series BCM57508 NICs (100 Gbps). mgmt, the built-in cluster network, was used for both management and virtual machine traffic.

Cause​

Root Cause​

Harvester relies on Linux’s bridge-based virtual networking to connect guest virtual machines to physical networks.

The NetXtreme-E BCM57508 NICs were connected to leaf switches configured to transmit jumbo frames. When the default MTU of 1500 is used, these frames should ideally be segmented to approximately 1450 bytes before reaching the Harvester host kernel. However, the packets actually arriving at the kernel were fragmented into unexpectedly small sizes. This forced the kernel to process a significantly higher volume of packets, leading to increased CPU overhead and reduced download throughput.

Packets captures collected using the following command confirmed the unexpectedly small size of the incoming packets.

tcpdump -xx -i <interface-name>

<interface-name> is the name of the physical interface on the host connected to the VMs.

GRO/GSO offload​

Generic Receive Offload (GRO) and Generic Segmentation Offload (GSO) are kernel-level software offloading mechanisms designed to optimize network performance. GRO aggregates multiple small incoming packets into larger ones before passing them to the network stack. GSO performs the opposite on transmission, splitting large packets into smaller frames before sending them to the NIC.

While these features are typically used to enhance performance, in this specific scenario, they interfered with the normal TCP segmentation process. This interference led to inefficient packet segmentation and the creation of an excessive number of small fragments, which ultimately degraded overall network performance.

When GRO and GSO were disabled, the Linux network stack automatically reverted to using standard transport-layer segmentation methods, specifically TCP Segmentation Offload (TSO) and Large Receive Offload (LRO). These mechanisms maintained efficient packet aggregation and segmentation at the appropriate layers, ensuring properly sized packets were presented to the kernel, which successfully restored expected network performance.

The NetXtreme-E BCM57508 NICs may experience suboptimal interaction with GRO and GSO due to a Broadcom driver bug. Enabling these offload mechanisms led to inefficient packetization, producing many small packets instead of fewer large ones, which ultimately reduced network throughput.

Solution​

• Option 1: Disable GRO and GSO on the affected Harvester host interfaces. This change does not persist across reboots.

  # Check current offload settings
  /usr/sbin/ethtool -k <interface-name>
  # Disable GRO and GSO
  /usr/sbin/ethtool -K <interface-name> gro off
  /usr/sbin/ethtool -K <interface-name> gso off
  

  Copy

• Option 2: Apply the following cloudinit resource and reboot the nodes. This change persists across reboots.

  Replace <interface-name> with the name of the physical interface the virtual machines are connected to.

  apiVersion: node.harvesterhci.io/v1beta1
  kind: CloudInit
  metadata:
    name: disable-offloads
  spec:
    matchSelector: {}
    filename: 99_disable_offloads.yaml
    contents: |
      stages:
        network:
          - commands:
              - /usr/sbin/ethtool -K <interface-name> gro off
              - /usr/sbin/ethtool -K <interface-name> gso off
  

  Copy

Verification​

• Run apt-get update or a curl command from a guest virtual machine. The download throughput should be normal (utilizing most of the link capacity).
• Transfer files between virtual machines running on different nodes using scp. The file transfers should be completed at the expected speed.
• Verify that no packets are dropped and no errors are reported using the following command:

  ip -s link show <interface-name>
  

  Copy

For additional context, see the official documentation at https://longhorn.io/docs/latest/advanced-resources/volumeattachment/

Workflow: How K8s and Longhorn VolumeAttachments Work Together​

When a Pod requires a Longhorn volume, two separate VolumeAttachment resources work together to complete the attachment process. The Kubernetes VolumeAttachment represents the CSI standard attachment request, while the Longhorn VolumeAttachment manages the actual attachment orchestration with ticket-based coordination.

The following diagram illustrates the complete flow from Pod scheduling to successful volume attachment:

┌─────────────────────────────────────────────────────────────┐
│                  Pod Scheduled to Node                      │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Kubernetes Attach/Detach Controller                         │
│ Creates K8s VolumeAttachment                                │
│   APIVersion: storage.k8s.io/v1                             │
│   Spec:                                                     │
│     Attacher: driver.longhorn.io                            │
│     NodeName: worker-node-1                                 │
│     Source.PersistentVolumeName: pvc-xxx                    │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ CSI External-Attacher (Longhorn)                            │
│ Watches K8s VolumeAttachment                                │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn CSI Plugin                                         │
│ Calls ControllerPublishVolume()                             │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn Manager API                                        │
│ Creates/Updates Longhorn VolumeAttachment                   │
│   APIVersion: longhorn.io/v1beta2                           │
│   Spec:                                                     │
│     Volume: my-volume                                       │
│     AttachmentTickets:                                      │
│       csi-attacher-<hash>:                                  │
│         ID: <pod-id>                                        │
│         Type: csi-attacher                                  │
│         NodeID: worker-node-1                               │
│         Parameters: {...}                                   │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn VolumeAttachment Controller                        │
│ 1. Evaluates all attachment tickets                         │
│ 2. Selects appropriate ticket to satisfy                    │
│ 3. Updates Volume.Spec.NodeID = worker-node-1               │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn Volume Controller                                  │
│ Performs actual volume attachment operation                 │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn VolumeAttachment Controller                        │
│ Updates ticket status: Satisfied = true                     │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn CSI Plugin                                         │
│ Returns attach success to external-attacher                 │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ CSI External-Attacher                                       │
│ Updates K8s VolumeAttachment.Status.Attached = true         │
└─────────────────────────────────────────────────────────────┘

The resulting Longhorn VolumeAttachment YAML:

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  name: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  namespace: longhorn-system
  labels:
    longhornvolume: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
spec:
  volume: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  attachmentTickets:
    # This CSI ticket was triggered by K8s VolumeAttachment (Pod binding)
    csi-3d3120f43480db87c91a6902d670c35899917c03f9f6f81db7bf26d9d66e45ec:
      id: csi-3d3120f43480db87c91a6902d670c35899917c03f9f6f81db7bf26d9d66e45ec
      type: csi-attacher
      nodeID: harvester-node-1
      parameters:
        disableFrontend: "false"
status:
  attachmentTicketStatuses:
    csi-3d3120f43480db87c91a6902d670c35899917c03f9f6f81db7bf26d9d66e45ec:
      id: csi-3d3120f43480db87c91a6902d670c35899917c03f9f6f81db7bf26d9d66e45ec
      satisfied: true  # Volume successfully attached
      conditions:
      - type: Satisfied
        status: "True"

Notice the csi-attacher ticket type - this confirms the attachment was triggered by Kubernetes VolumeAttachment through the CSI flow, not by Longhorn internal operations.

Trigger Points​

Understanding when each VolumeAttachment is created or modified is crucial for troubleshooting attachment issues:

1. K8s VolumeAttachment Creation: Triggered when Pod is scheduled to a node requiring a PVC

   • Managed by Kubernetes Attach/Detach (AD) Controller
   • One VolumeAttachment per PV-node combination
   • Represents Kubernetes' intent to attach the volume

2. Longhorn VolumeAttachment Ticket Addition: Triggered by various Longhorn components based on operation needs:

   • CSIAttacher - when CSI ControllerPublishVolume is called
   • SnapshotController - when creating snapshots of volumes
   • BackupController - when backing up volumes
   • LonghornAPI - when users manually attach volumes via Longhorn UI
   • VolumeCloneController - when managing source volume during clone
   • VolumeRestoreController - when restoring data from backups
   • VolumeExpansionController - when expanding volume size
   • ShareManagerController - for RWX volume sharing
   • SalvageController - for volume salvage operations

Attachment Ticket Priority and Coordination​

When multiple operations require volume attachment simultaneously, Longhorn uses a ticket-based priority system to coordinate access intelligently. This ensures critical operations take precedence while allowing background tasks to coexist when possible.

How Priority Works​

Each ticket type has an assigned priority level that determines selection order when the volume is detached:

• Priority 2000 (Highest):
  • VolumeRestoreController
  • VolumeExpansionController
• Priority 1000:
  • LonghornAPI
• Priority 900:
  • CSIAttacher
  • ShareManagerController
  • SalvageController
• Priority 800 (Lowest):
  • BackupController
  • SnapshotController
  • VolumeCloneController
  • VolumeEvictionController

When the volume is detached, the ticket with the highest priority is selected for attachment. If multiple tickets share the same priority, the first one (sorted by ID) is chosen.

Interruption Mechanism​

Priority levels alone don't tell the complete story. Longhorn also implements an interruption mechanism to handle cases where request arrives while the volume is already attached to a different node.

Interruptible operations (can be interrupted):

• BackupController
• SnapshotController
• VolumeCloneController - clone operations, but only when the volume is in VolumeCloneStateCopyCompletedAwaitingHealthy state

Workload operations (can trigger interruption):

• CSIAttacher - Pod workloads requiring the volume on a different node
• LonghornAPI - manual attachment requests via UI/API
• ShareManagerController - RWX volume sharing operations

The interruption only occurs when:

1. The volume's currently attached node has only interruptible tickets
2. A different node has a workload ticket requesting the volume

This design ensures background operations never block workload rescheduling, while protecting active workloads from being interrupted by other background tasks.

Real-World Scenarios​

Scenario 1: Backup During Active Pod Usage

• Pod is running on node-A with a CSIAttacher ticket
• BackupController creates a ticket for node-A (same node)
• Both tickets coexist peacefully - backup runs alongside the Pod
• CSI attachment and backup execution use the engine on the same node, avoiding a node transition.

Scenario 2: Backup Interrupted by Pod Workload

• BackupController is running on node-A (only ticket present)
• A Pod requiring this volume is scheduled to node-B, CSIAttacher creates a ticket for node-B
• VolumeAttachment Controller detects: interruptible ticket on node-A, workload ticket on node-B
• Volume detaches from node-A (backup interrupted), attaches to node-B (csi attacher)
• Backup will retry later automatically

Scenario 3: Detached Volume Snapshot

• Volume is detached, SnapshotController creates a ticket
• Volume attaches temporarily for snapshot creation
• After snapshot completes, ticket is removed
• Volume auto-detaches if no other tickets exist

Usage Examples​

The following examples demonstrate how VolumeAttachment resources behave in common scenarios. Each example shows the complete YAML resource state at different stages, helping you understand what to look for when troubleshooting or monitoring Longhorn operations.

Example 1: VolumeSnapshot Creation (Longhorn VolumeAttachment Only)​

VolumeSnapshot operations use only Longhorn VolumeAttachment without involving Kubernetes VolumeAttachment. This demonstrates that Longhorn VolumeAttachment can operate independently for internal operations.

┌─────────────────────────────────────────────────────────────┐
│           User Creates VolumeSnapshot via kubectl           │
│           kubectl apply -f volumesnapshot.yaml              │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn Snapshot Controller                                │
│ Detects new VolumeSnapshot resource                         │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Snapshot Controller Checks Volume State                     │
│ If Volume is detached → needs attachment for snapshot       │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Snapshot Controller Creates Attachment Ticket               │
│ Updates Longhorn VolumeAttachment:                          │
│   AttachmentTickets:                                        │
│     snapshot-<snapshot-name>:                               │
│       Type: snapshot-controller                             │
│       NodeID: <volume-owner-node>                           │
│       Parameters: {disableFrontend: "false"}                │
│                                                             │
│ ❌ No K8s VolumeAttachment created                          │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn VolumeAttachment Controller                        │
│ Selects snapshot ticket → Updates Volume.Spec.NodeID        │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn Volume Controller                                  │
│ Attaches volume → Starts Engine                             │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Snapshot Controller                                         │
│ Engine running → Creates snapshot via Engine API            │
│ Snapshot complete → Removes attachment ticket               │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Volume May Auto-Detach (if no other tickets exist)          │
└─────────────────────────────────────────────────────────────┘

The Longhorn VolumeAttachment YAML during snapshot creation:

During Snapshot Creation (ticket exists):

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  name: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  namespace: longhorn-system
  generation: 30
spec:
  volume: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  attachmentTickets:
    # Temporary ticket created by Snapshot Controller
    snapshot-controller-snapshot-a36bedf5-fb3b-4b30-a10d-ed98f9c0323a:
      id: snapshot-controller-snapshot-a36bedf5-fb3b-4b30-a10d-ed98f9c0323a
      type: snapshot-controller
      nodeID: harvester-node-1
      parameters:
        disableFrontend: any
status:
  attachmentTicketStatuses:
    snapshot-controller-snapshot-a36bedf5-fb3b-4b30-a10d-ed98f9c0323a:
      id: snapshot-controller-snapshot-a36bedf5-fb3b-4b30-a10d-ed98f9c0323a
      satisfied: false  # Snapshot in progress
      conditions:
      - type: Satisfied
        status: "False"

After Snapshot Completes (ticket removed):

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  name: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  namespace: longhorn-system
  generation: 31  # Incremented after ticket removal
spec:
  volume: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  attachmentTickets: {}  # Ticket removed after snapshot completes
status:
  attachmentTicketStatuses: {}

Key Observations:

• The snapshot-controller ticket type clearly identifies this as a Longhorn internal operation
• Unlike csi-attacher tickets (triggered by K8s), this ticket is created purely by Longhorn
• The ticket is temporary - it appears during snapshot creation and disappears when complete
• No corresponding Kubernetes VolumeAttachment exists for this operation

Example 2: VM Migration​

During VM migration, Harvester has two virt-launcher pods for the same VM: the original pod on the source node and a new pod on the target node. This multi-attach capability is enabled for RWX (ReadWriteMany) block mode volumes when the StorageClass has migratable: true parameter, which allows Longhorn to support live VM migration. In the following example, we migrate a VM from harvester-node-2 to harvester-node-0.

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  creationTimestamp: "2025-12-10T04:19:42Z"
  finalizers:
  - longhorn.io
  generation: 3
  labels:
    longhornvolume: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
  name: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
  namespace: longhorn-system
  ownerReferences:
  - apiVersion: longhorn.io/v1beta2
    kind: Volume
    name: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
    uid: 7cd2ed46-194f-4528-83f7-bbaa5945e7e3
  resourceVersion: "2736440"
  uid: b2492681-8fcb-4330-9ec6-496afa93e96b
spec:
  attachmentTickets:
    csi-5852f2d48d96311bb582eeeaad0e38361031d502899416c71cea10795748a84b:
      generation: 0
      id: csi-5852f2d48d96311bb582eeeaad0e38361031d502899416c71cea10795748a84b
      nodeID: harvester-node-2
      parameters:
        disableFrontend: "false"
        lastAttachedBy: ""
      type: csi-attacher
    csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d:
      generation: 0
      id: csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d
      nodeID: harvester-node-0
      parameters:
        disableFrontend: "false"
        lastAttachedBy: ""
      type: csi-attacher
  volume: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
status:
  attachmentTicketStatuses:
    csi-5852f2d48d96311bb582eeeaad0e38361031d502899416c71cea10795748a84b:
      conditions:
      - lastProbeTime: ""
        lastTransitionTime: "2025-12-10T04:19:49Z"
        message: ""
        reason: ""
        status: "True"
        type: Satisfied
      generation: 0
      id: csi-5852f2d48d96311bb582eeeaad0e38361031d502899416c71cea10795748a84b
      satisfied: true
    csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d:
      conditions:
      - lastProbeTime: ""
        lastTransitionTime: "2025-12-10T04:21:00Z"
        message: The migrating attachment ticket is satisfied
        reason: ""
        status: "True"
        type: Satisfied
      generation: 0
      id: csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d
      satisfied: true

After Migration Completes (ticket removed):

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  creationTimestamp: "2025-12-10T04:19:42Z"
  finalizers:
  - longhorn.io
  generation: 4
  labels:
    longhornvolume: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
  name: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
  namespace: longhorn-system
  ownerReferences:
  - apiVersion: longhorn.io/v1beta2
    kind: Volume
    name: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
    uid: 7cd2ed46-194f-4528-83f7-bbaa5945e7e3
  resourceVersion: "2736824"
  uid: b2492681-8fcb-4330-9ec6-496afa93e96b
spec:
  attachmentTickets:
    csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d:
      generation: 0
      id: csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d
      nodeID: harvester-node-0
      parameters:
        disableFrontend: "false"
        lastAttachedBy: ""
      type: csi-attacher
  volume: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
status:
  attachmentTicketStatuses:
    csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d:
      conditions:
      - lastProbeTime: ""
        lastTransitionTime: "2025-12-10T04:21:00Z"
        message: ""
        reason: ""
        status: "True"
        type: Satisfied
      generation: 0
      id: csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d
      satisfied: true

Key Observations:

• Two CSI attachment tickets coexist: One pointing to the source node (harvester-node-2) and another to the target node (harvester-node-0)
• Both tickets are csi-attacher type: Indicating they were both triggered by Kubernetes VolumeAttachment through the CSI flow
• Both tickets have satisfied: true status: This demonstrates Longhorn's support for attaching the same volume to multiple nodes simultaneously (RWX-like behavior for migration)
• Target node ticket has special message: "The migrating attachment ticket is satisfied" explicitly identifies this as a migration scenario
• Multi-attach is temporary: This dual-attachment state only exists during VM migration; the source node's ticket will be removed after migration completes

Summary​

Longhorn uses two different VolumeAttachment resources for different purposes:

Kubernetes VolumeAttachment (storage.k8s.io/v1) follows the standard CSI specification and is created only when Pods are scheduled to nodes. It represents Kubernetes' official attachment intent and is managed by K8s Attach/Detach Controller and CSI External-Attacher.

Longhorn VolumeAttachment (longhorn.io/v1beta2) extends beyond CSI to support Longhorn's advanced features. It's created for multiple scenarios, including Pod workloads, snapshots, backups, clones, and manual operations. It uses a ticket-based system to coordinate concurrent attachment requests and is managed collaboratively by multiple Longhorn controllers.

Why both are needed: K8s VolumeAttachment ensures CSI compliance with the Kubernetes ecosystem, while Longhorn VolumeAttachment enables automation for background operations without manual intervention. Importantly, not all Longhorn operations trigger K8s VolumeAttachment—for example, creating a VolumeSnapshot only creates a Longhorn VolumeAttachment ticket (snapshot-controller), not a K8s VolumeAttachment.

When troubleshooting: Check both resources. K8s VolumeAttachment shows the CSI standard workflow status, while Longhorn VolumeAttachment shows the complete picture, including all internal operations via attachment tickets. Look at the ticket type to identify the operation source: csi-attacher means triggered by the K8s VolumeAttachment (Pod workload), while snapshot-controller, backup-controller, etc. indicate Longhorn internal operations.

However, in certain scenarios, users may require the VM filesystem to be quiesced during Velero backup creation to prevent data corruption, especially when the VM is experiencing heavy I/O operations.

This article describes how to customize Velero Backup Hooks to implement filesystem freeze during Velero backup processing, ensuring data consistency in the backup content.

Background Knowledge​

KubeVirt's virt-freezer provides a mechanism to freeze and thaw guest filesystems. This capability can be leveraged to ensure filesystem consistency during VM backups. However, certain prerequisites must be met for filesystem freeze/thaw operations to function properly:

Prerequisites for Filesystem Freeze​

• QEMU Guest Agent must be enabled in the guest VM
  • Verify this by checking if the VMI has AgentConnected in its status
• Guest VM must be properly configured for related libvirt commands
  • When virt-freezer is triggered, KubeVirt communicates with the QEMU Guest Agent via libvirt commands such as guest-fsfreeze-freeze
  • The guest agent translates these commands to OS-specific calls:
    • Linux systems: Uses fsfreeze syscalls
    • Windows systems: Uses VSS (Volume Shadow Copy Service) APIs

Common Configuration Challenges​

Based on Harvester project experience, some guest operating systems require additional configuration:

• Linux distributions (e.g., RHEL, SLE Micro): May lack sufficient permissions for filesystem freeze operations by default, requiring custom policies
• Windows guests: Require the VSS service to be enabled for filesystem freeze functionality

Important: Filesystem freeze/thaw functionality depends on guest VM configuration, which is outside Harvester's control. Users are responsible for ensuring compatibility before implementing Velero backup hooks with filesystem freeze.

Verifying Filesystem Freeze Compatibility​

To confirm that your VM supports filesystem freeze operations:

1. Access the virtual machine's virt-launcher compute container:

   POD=$(kubectl get pods -n <VM Namespace> \
     -l vm.kubevirt.io/name=<VM Name> \
     -o jsonpath='{.items[0].metadata.name}')
   kubectl exec -it $POD -n default -c compute -- bash
   

   Copy

2. Test filesystem freeze using the virt-freezer application available in the compute container:

   virt-freezer --freeze --namespace <VM namespace> --name <VM name>
   

   Copy

3. Critical: Always verify the freeze operation result and thaw the VM filesystems before performing any other operations

Prerequisites​

All preparation steps outlined in External CSI Storage Backup and Restore With Velero are mandatory, including:

• Harvester installation and configuration
• Velero installation and setup
• S3-compatible storage configuration
• Proper networking and permissions

Implementing Filesystem Freeze Hooks for VM Backup Consistency​

Velero supports pre and post backup hooks that can be integrated with KubeVirt's virt-freezer to ensure filesystem consistency during VM backups.

Configuring VM Template Annotations​

For all VMs requiring data consistency, add the following annotations to the VM template:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-nfs
  namespace: demo
spec:
  template:
    metadata:
      annotations:
        # These annotations will be applied to the virt-launcher pod
        pre.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--freeze", "--namespace", "<VM Namespace>", "--name", "<VM Name>"']'
        pre.hook.backup.velero.io/container: compute
        pre.hook.backup.velero.io/on-error: Fail
        pre.hook.backup.velero.io/timeout: 30s
        
        post.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--unfreeze", "--namespace", "<VM Namespace>", "--name", "<VM Name>"]'
        post.hook.backup.velero.io/container: compute
        post.hook.backup.velero.io/timeout: 30s
    spec:
      # ...rest of VM spec...

These annotations will be propagated to the related virt-launcher pod and instruct Velero to:

• Freeze the VM filesystem before backup creation begins
• Thaw the VM filesystem after backup completion

Important: Replace <VM Namespace> and <VM Name> with the actual namespace and name of your VM.

Creating a Velero Backup with Filesystem Freeze​

After applying the Velero pre/post hook annotations to the VM manifest, follow the backup procedures described in External CSI Storage Backup and Restore With Velero.

Verifying Successful Hook Execution​

If the guest VM is configured correctly, the Velero backup will complete successfully with HooksAttempted indicating successful hook execution.

Check the backup status using:

velero backup describe [Backup Name] --details

Example output showing successful hook execution:

Name:         demo
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.33.3+rke2r1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=33

Phase:  Completed


Namespaces:
  Included:  demo
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          true
Data Mover:                  velero

....

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots:
    demo/vm-nfs-disk-0-au2ej:
      Data Movement:
        Operation ID: du-be5417aa-498e-4b93-b59f-e6498f95a6df.d7f97dab-3bb1-41e189381
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 5368709120
        Result: succeeded

  Pod Volume Backups: <none included>

HooksAttempted:  2
HooksFailed:     0

The output shows that Velero pre/post backup hooks completed successfully. In this case, the hooks are connected to guest VM filesystem freeze and thaw operations to ensure data consistency.

Restoring the Velero Backup​

Follow the restoration procedures described in External CSI Storage Backup and Restore With Velero to restore the namespace using Velero.

Troubleshooting​

If you encounter issues with filesystem freeze operations:

1. Verify QEMU Guest Agent status in the VMI
2. Check guest OS configuration for filesystem freeze support
3. Review Velero hook logs for specific error messages
4. Test virt-freezer manually as described in the verification section

Conclusion​

Implementing filesystem freeze hooks with Velero ensures data consistency during VM backups by quiescing the filesystem before snapshot creation. This approach is particularly valuable for VMs with high I/O activity or critical data that requires point-in-time consistency guarantees.

For Harvester to successfully migrate a virtual machine from one node to another, the source and target nodes must have compatible CPU models and features.

Harvester uses KubeVirt / QEMU / libvirt to manage and run virtual machines. When a VM starts, libvirt exposes a specific CPU feature set to the guest operating system. Live migration requires this CPU feature set to be identical on both source and target nodes.

Different CPU generations support different instruction sets and feature flags. If those differ, the live migration will be blocked to avoid instability or incorrect execution on the target node.

If the CPU model of a virtual machine isn't specified, KubeVirt assigns it the default host-model configuration so that the virtual machine has the CPU model closest to the one used on the host node.

KubeVirt automatically adjusts the node selectors of the associated virt-launcher Pod based on this configuration. If the CPU models and features of the source and target nodes do not match, the live migration may fail.

Let's examine an example.

When a virtual machine is first migrated to another node with the SierraForest CPU model, the following key-value pairs are added to the spec.nodeSelector field in the Pod spec.

spec:
  nodeSelector:
    cpu-model-migration.node.kubevirt.io/SierraForest: "true"
    cpu-feature.node.kubevirt.io/fpu: "true"
    cpu-feature.node.kubevirt.io/vme: "true"

The above nodeSelector configuration is retained for subsequent migrations, which may fail if the new target node doesn't have the corresponding features or model.

For example, compare the CPU model and feature labels added by KubeVirt to the following two nodes:

# Node A
labels:
  cpu-model-migration.node.kubevirt.io/SierraForest: "true"
  cpu-feature.node.kubevirt.io/fpu: "true"
  cpu-feature.node.kubevirt.io/vme: "true"

# Node B
labels:
  cpu-model-migration.node.kubevirt.io/SierraForest: "true"
  cpu-feature.node.kubevirt.io/vme: "true"

This virtual machine will fail to migrate to Node B due to the missing fpu feature. However, if the virtual machine doesn't actually require this feature, this can be frustrating. Therefore, setting up a common CPU model can resolve this issue.

How to Set Up a Common CPU Model​

You can define a custom CPU model to ensure that the spec.nodeSelector configuration in the Pod spec is assigned a CPU model that is compatible and common to all nodes in the cluster.

Consider this example.

We have the following node information:

# Node A
labels:
  cpu-model.node.kubevirt.io/IvyBridge: "true"
  cpu-feature.node.kubevirt.io/fpu: "true"
  cpu-feature.node.kubevirt.io/vme: "true"

# Node B
labels:
  cpu-model.node.kubevirt.io/IvyBridge: "true"
  cpu-feature.node.kubevirt.io/vme: "true"

If we set up IvyBridge as our CPU model in the virtual machine spec, KubeVirt only adds cpu-model.node.kubevirt.io/IvyBridge under spec.nodeSelector in the Pod spec.

# Virtual Machine Spec
spec:
  template:
    spec:
      domain:
        cpu:
          model: IvyBridge

# Pod spec
spec:
  nodeSelector:
    cpu-model.node.kubevirt.io/IvyBridge: "true"

With this configuration, your virtual machine can be migrated to any node that has the label cpu-model.node.kubevirt.io/IvyBridge.

Understanding CPU Model Options in Harvester​

Harvester allows the CPU model to be defined in the VM specification.

If host-passthrough is used, the VM exposes the exact host CPU to the guest. This provides maximum performance but completely prevents migration across different CPU generations.

If host-model is used, the VM derives a CPU model based on the host’s capabilities. This works only when all nodes expose identical CPU features. In mixed clusters, this setting commonly causes migration failures.

If an explicit CPU model is defined, the VM exposes a named CPU architecture defined by QEMU. This is the recommended approach for clusters that may contain different CPU generations. For mixed environments, an explicit CPU model should always be used.

Choosing a CPU Model for Live Migration​

When selecting a CPU model, the goal is to choose the highest common denominator. This means the most modern CPU architecture that every node in the cluster supports. Pick a model supported by all nodes in your cluster, including the oldest CPU generation. Use a server-grade model that provides a reasonable set of capabilities (e.g., vector instructions, security features) without tying VMs to host-specific features. Use the same model on all VMs that need live migration.

Below are practical examples for common enterprise hardware combinations.

For a cluster that mixes Cascade Lake and Sapphire Rapids, the recommended CPU model is Cascadelake-Server. Sapphire Rapids processors are backward compatible with Cascade Lake instructions, making Cascadelake-Server the highest common denominator between these two generations.

For a cluster that mixes Skylake and Cascade Lake nodes, the appropriate model is Skylake-Server. This is the common ground between both generations. Cascade Lake includes additional AVX-512 optimizations, but using Skylake ensures compatibility across the entire cluster.

If your cluster contains Broadwell nodes along with anything newer, the safest baseline is Broadwell. Broadwell serves as a stable and widely supported baseline for older enterprise hardware. Any newer CPUs will support it.

Set Up Cluster-Wide Configuration​

If your virtual machines run only on a specific CPU model, you can set up a cluster-wide CPU model in the kubevirt resource.

You can edit it with kubectl edit kubevirt kubevirt -n harvester-system, then add the CPU model you want in the following spec:

spec:
  configuration:
    cpuModel: IvyBridge

Then, when a new virtual machine starts or an existing virtual machine restarts, the cluster-wide setting will be applied. The system follows these priorities when using CPU models if you configure them in both locations:

1. CPU model in the virtual machine spec.
2. CPU model in the KubeVirt spec.

References​

• CPU Model Matching
• QEMU / KVM CPU model configuration

Public Key Authentication​

Create cloud-init.yaml​

Create cloud-init.yaml with the following content.

• Modify the matchSelector if you want to create an SSH user only on specific nodes.
• #cloud-config should be written exactly as shown.
• User must be in the admin group.
• Specify the public key in ssh_authorized_keys.
• Modify the contents according to your environment.

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: add-test-user
spec:
  matchSelector: {} # applies to all nodes
  filename: 99_add_test_user.yaml
  contents: |
    #cloud-config
    users:
      - name: test-user
        gecos: "admin_user"
        groups: [users, admin]
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
        ssh_authorized_keys:
        - ssh-rsa AAAA.... # <--insert full authorized key here, e.g. from your ~/.ssh/id_rsa.pub file

Password Authentication​

Create password hash​

Use the following command to create a password hash. Replace test with your actual password.

$ openssl passwd -6 'test'
$6$zF26pcXOS2eaivX8$6ySoTzQC2cToz29mGFC0DuG5cVWTv3Mktc3k/g1KXTtrG2BhsFh8xs3N0zBmNx0D/H4f1W48a45vI1RK8Rzs.0

Create cloud-init.yaml​

Create cloud-init.yaml with the following content.

• Modify the matchSelector if you want to create an SSH user only on specific nodes.
• #cloud-config should be written exactly as shown.
• User must be in the admin group.
• Specify the hash created earlier in passwd.
• Modify the contents according to your environment.

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: add-test-user
spec:
  matchSelector: {} # applies to all nodes
  filename: 99_add_test_user.yaml
  contents: |
    #cloud-config
    users:
      - name: test-user
        gecos: "admin_user"
        groups: [users, admin]
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
        lock_passwd: false
        passwd: $6$zF26pcXOS2eaivX8$6ySoTzQC2cToz29mGFC0DuG5cVWTv3Mktc3k/g1KXTtrG2BhsFh8xs3N0zBmNx0D/H4f1W48a45vI1RK8Rzs.0

Apply the YAML file​

Apply the YAML with this command.

kubectl apply -f cloud-init.yaml

Trailing file will be created.

$ cat /oem/99_add_test_user.yaml
#cloud-config
users:
  - name: test-user
    gecos: "admin_user"
    groups: users, admin
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ...

Reboot Harvester nodes​

1. Run Enable Maintenance Mode on the Harvester UI.
2. Wait until the state changes to Maintenance.
3. Reboot the Harvester node.
4. Run Disable Maintenance Mode on the Harvester UI.

After that, SSH login will be possible.

Tested on: Harvester 1.6.0, 1.5.1.

To mitigate this elapsed time, you can modify the vm-force-reset-policy setting, by reducing its period value. This enables Harvester to detect non-ready virtual machines on unreachable nodes sooner.

This setting can be found in the Advanced -> Settings page on the Harvester UI.

Additionally, while the current default is 5 minutes, we are considering reducing the default value [1].

References​

[1] https://github.com/harvester/harvester/issues/8971

To allow extra disks without WWNs to be visible to Harvester, perform either of the following workarounds:

Workaround 1: Create a filesystem on the disk​

When you create a filesystem on a disk (for example, using the command mkfs.ext4 /dev/sda), a filesystem UUID is assigned to the disk. Harvester uses this value to identify disks without a WWN.

In Harvester versions earlier than v1.6.0, you can use this workaround for only one extra disk because of a bug in duplicate device checking.

Workaround 2: Add a udev rule for generating fake WWNs​

You can add a udev rule that generates a fake WWN for each extra disk based on the device serial number. Harvester accepts the generated WWNs because the only requirement is a unique ID_WWN value as presented by udev.

A YAML file containing the necessary udev rule must be created in the /oem directory on each host. This process can be automated across the Harvester cluster using a CloudInit Resource.

1. Create a YAML file named fake-scsi-wwn-generator.yaml with the following contents:

   apiVersion: node.harvesterhci.io/v1beta1
   kind: CloudInit
   metadata:
     name: fake-scsi-wwn-generator
   spec:
     matchSelector: {}
     filename: 90_fake_scsi_wwn_generator.yaml
     contents: |
       name: "Add udev rules to generate missing SCSI disk WWNs"
       stages:
         initramfs:
           - files:
               - path: /etc/udev/rules.d/59-fake-scsi-wwn-generator.rules
                 permissions: 420
                 owner: 0
                 group: 0
                 content: |
                   # For anything that looks like a SCSI disk (/dev/sd*),
                   # if it has a serial number, but does _not_ have a WWN,
                   # create a fake WWN based on the serial number.  We need
                   # to set both ID_WWN so Harvester's node-disk-manager
                   # can see the WWN, and ID_WWN_WITH_EXTENSION which is
                   # what 60-persistent-storage.rules uses to generate a
                   # /dev/disk/by-id/wwn-* symlink for the device.
                   ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="sd*[!0-9]", \
                     ENV{ID_SERIAL}=="?*", \
                     ENV{ID_WWN}!="?*", ENV{ID_WWN_WITH_EXTENSION}!="?*", \
                     ENV{ID_WWN}="fake.$env{ID_SERIAL}", \
                     ENV{ID_WWN_WITH_EXTENSION}="fake.$env{ID_SERIAL}"
   

   Copy

2. Apply the file's contents to the cluster by running the command kubectl apply -f fake-scsi-wwn-generator.yaml.

   The file /oem/90_fake_scsi_wwn_generator.yaml is automatically created on all cluster nodes.

3. Reboot all nodes to apply the new udev rule.

Once the rule is applied, you should be able to view and add extra disks that were previously not visible on the Harvester UI.

References​

• Harvester: Issue 7173

This article demonstrates how to use Velero 1.16.0 to perform backup and restore of virtual machines in Harvester.

It goes through commands and manifests to:

• Back up virtual machines in a namespace, their NFS CSI volumes, and associated namespace-scoped configuration
• Export the backup artifacts to an AWS S3 bucket
• Restore to a different namespace on the same cluster
• Restore to a different cluster

Velero is a Kubernetes-native backup and restore tool that enables users to perform scheduled and on-demand backups of virtual machines to external object storage providers such as S3, Azure Blob, or GCS, aligning with enterprise backup and disaster recovery practices.

Harvester Installation​

Refer to the Harvester documentation for installation requirements and options.

The kubeconfig file of the Harvester cluster can be retrieved following the instructions here.

Install and Configure Velero​

Download the Velero CLI.

Set the following shell variables:

BUCKET_NAME=<your-s3-bucket-name>
BUCKET_REGION=<your-s3-bucket-region>
AWS_CREDENTIALS_FILE=<absolute-path-to-your-aws-credentials-file>

Install Velero on the Harvester cluster:

velero install \
  --provider aws \
  --features=EnableCSI \
  --plugins "velero/velero-plugin-for-aws:v1.12.0,quay.io/kubevirt/kubevirt-velero-plugin:v0.7.1" \
  --bucket "${BUCKET_NAME}" \
  --secret-file "${AWS_CREDENTIALS_FILE}" \
  --backup-location-config region="${BUCKET_REGION}" \
  --snapshot-location-config region="${BUCKET_REGION}" \
  --use-node-agent

• In this setup, Velero is configured to:

  • Run in the velero namespace
  • Enable CSI volume snapshot APIs
  • Enable the built-in node agent data movement controllers and pods
  • Use the velero-plugin-for-aws plugin to manage interactions with the S3 object store
  • Use the kubevirt-velero-plugin plugin to backup and restore KubeVirt resources

Confirm that Velero is installed and running:

kubectl -n velero get po

NAME                      READY   STATUS    RESTARTS         AGE
node-agent-875mr          1/1     Running   0                1d
velero-745645565f-5dqgr   1/1     Running   0                1d

Configure the velero CLI to output the backup and restore status of CSI objects:

velero client config set features=EnableCSI

Deploy the NFS CSI and Example Server​

Follow the instructions in the NFS CSI documentation to set up the NFS CSI driver, its storage class, and an example NFS server.

The NFS CSI volume snapshotting capability must also be enabled following the instructions here.

Confirm that the NFS CSI and example server are running:

kubectl get po -A -l 'app in (csi-nfs-node,csi-nfs-controller,nfs-server)'

NAMESPACE     NAME                                  READY   STATUS    RESTARTS    AGE
default       nfs-server-b767db8c8-9ltt4            1/1     Running   0           1d
kube-system   csi-nfs-controller-5bf646f7cc-6vfxn   5/5     Running   0           1d
kube-system   csi-nfs-node-9z6pt                    3/3     Running   0           1d

The default NFS CSI storage class is named nfs-csi:

kubectl get sc nfs-csi

NAME      PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-csi   nfs.csi.k8s.io   Delete          Immediate           true                   14d

Confirm that the default NFS CSI volume snapshot class csi-nfs-snapclass is also installed:

kubectl get volumesnapshotclass csi-nfs-snapclass

NAME                DRIVER           DELETIONPOLICY   AGE
csi-nfs-snapclass   nfs.csi.k8s.io   Delete           14d

Preparing the Virtual Machine and Image​

Create a custom namespace named demo-src:

kubectl create ns demo-src

Follow the instructions in the Image Management documentation to upload the Ubuntu 24.04 raw image from https://cloud-images.ubuntu.com/minimal/releases/noble/ to Harvester.

The storage class of the image must be set to nfs-csi, per the Third-Party Storage Support documentation.

Confirm the virtual machine image is successfully uploaded to Harvester:

Follow the instructions in the third-party storage documentation to create a virtual machine with NFS root and data volumes, using the image uploaded in the previous step.

For NFS CSI snapshot to work, the NFS data volume must have the volumeMode set to Filesystem:

Backup the Source Namespace​

Use the velero CLI to create a backup of the demo-src namespace using Velero's built-in data mover:

BACKUP_NAME=backup-demo-src-`date "+%s"`

velero backup create "${BACKUP_NAME}" \
  --include-namespaces demo-src \
  --snapshot-move-data

This creates a backup of the demo-src namespace containing resources like the virtual machine created earlier, its volumes, secrets and other associated configuration.

Depending on the size of the virtual machine and its volumes, the backup may take a while to complete.

The DataUpload custom resources provide insights into the backup progress:

kubectl -n velero get datauploads -l velero.io/backup-name="${BACKUP_NAME}"

Confirm that the backup completed successfully:

velero backup get "${BACKUP_NAME}"

NAME                         STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
backup-demo-src-1747954979   Completed   0        0          2025-05-22 16:04:46 -0700 PDT   29d       default            <none>

After the backup completes, Velero removes the CSI snapshots from the storage side to free up the snapshot data space.

Restore To A Different Namespace​

This section describes how to restore the backup from the demo-src namespace to a new namespace named demo-dst.

Save the following restore modifier to a local file named modifier-data-volumes.yaml:

cat <<EOF > modifier-data-volumes.yaml
version: v1
resourceModifierRules:
- conditions:
    groupResource: persistentvolumeclaims
    matches:
    - path: /metadata/annotations/harvesterhci.io~1volumeForVirtualMachine
      value: "\"true\""
  patches:
  - operation: remove
    path: /metadata/annotations/harvesterhci.io~1volumeForVirtualMachine
EOF

This restore modifier removes the harvesterhci.io/volumeForVirtualMachine annotation from the virtual machine data volumes to ensure that the restoration do not conflict with the CDI volume import populator.

Create the restore modifier:

kubectl -n velero create cm modifier-data-volumes --from-file=modifier-data-volumes.yaml

Assign the backup name to a shell variable:

BACKUP_NAME=backup-demo-src-1747954979

Start the restore operation:

velero restore create \
  --from-backup "${BACKUP_NAME}" \
  --namespace-mappings "demo-src:demo-dst" \
  --exclude-resources "virtualmachineimages.harvesterhci.io" \
  --resource-modifier-configmap "modifier-data-volumes" \
  --labels "velero.kubevirt.io/clear-mac-address=true,velero.kubevirt.io/generate-new-firmware-uuid=true"

• During the restore:

  • The virtual machine MAC address and firmware UUID are reset to avoid potential conflicts with existing virtual machines.
  • the virtual machine image manifest is excluded because Velero restores the entire state of the virtual machine from the backup.
  • the modifier-data-volumes restore modifier is invoked to modify the virtual machine data volumes metadata to prevent conflicts with the CDI volume import populator.

While the restore operation is still in-progress, the DataDownload custom resources can be used to examine the progress of the operation:

RESTORE_NAME=backup-demo-src-1747954979-20250522164015

kubectl -n velero get datadownload -l velero.io/restore-name="${RESTORE_NAME}"

Confirm that the restore completed successfully:

velero restore get

NAME                                        BACKUP                       STATUS      STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
backup-demo-src-1747954979-20250522164015   backup-demo-src-1747954979   Completed   2025-05-22 16:40:15 -0700 PDT   2025-05-22 16:40:49 -0700 PDT   0        6          2025-05-22 16:40:15 -0700 PDT   <none>

Verify that the virtual machine and its configuration are restored to the new demo-dst namespace:

Restore To A Different Cluster​

This section extends the above scenario to demonstrate the steps to restore the backup to a different Harvester cluster.

On the target cluster, install Velero, and set up the NFS CSI and NFS server following the instructions from the Deploy the NFS CSI and Example Server section.

Once Velero is configured to use the same backup location as the source cluster, it automatically discovers the available backups:

velero backup get

NAME                         STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
backup-demo-src-1747954979   Completed   0        0          2025-05-22 16:04:46 -0700 PDT   29d       default            <none>

Follow the steps in the Restore To A Different Namespace section to restore the backup on the target cluster.

Remove the --namespace-mappings option to set the restored namespace to demo-src on the target cluster.

Confirm that the virtual machine and its configuration are restored to the demo-src namespace:

Select Longhorn Volume Snapshot Class​

To perform Velero backup and restore of virtual machines with Longhorn volumes, label the Longhorn volume snapshot class longhorn as follows:

kubectl label volumesnapshotclass longhorn velero.io/csi-volumesnapshot-class

This helps Velero to find the correct Longhorn snapshot class to use during backup and restore.

Limitations​

Enhancements related to the limitations described in this section are tracked at https://github.com/harvester/harvester/issues/8367.

• By default, Velero only supports resource filtering by resource groups and labels. In order to backup/restore a single instance of virtual machine, custom labels must be applied to the virtual machine, and its virtual machine instance, pod, data volumes, persistent volumes claim, persistent volumes and cloudinit secret resources. It's recommended to backup the entire namespace and perform resource filtering during restore to ensure that backup contains all the dependency resources required by the virtual machine.

• The restoration of virtual machine image is not fully supported yet.

Pod Security Standards in Harvester can be used when provisioning VM workloads and also with the new experimental support for running baremetal container workloads.

The baseline policy is aimed at ease of adoption for common containerized workloads while preventing known privilege escalations. This policy is targeted at application operators and developers of non-critical applications.

Namespace level enablement​

To enable PSS a user simply needs to label their workload namespaces as follows:

kubectl label --overwrite ns <namespace>  pod-security.kubernetes.io/enforce=baseline

Cluster scoped enablement​

Cluster wide PSS can be enabled by passing an Admission Control configuration via kube-apiserver arguments. This can be done via Harvester's CloudInit using the following configuration which can be saved to cloudinit-pss.yaml file:

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: cluster-wide-pss-enforcement
spec:
  matchSelector:
    node-role.kubernetes.io/control-plane: "true"
  filename: 99-pss.yaml
  contents: |
    stages:
      initramfs:
        - name: "setup harvester pss"
          directories:
          - path: /etc/rancher/rke2/config
            owner: 0
            group: 0
            permissions: 384
          files:
          - content: |
              kube-apiserver-arg:
                - "admission-control-config-file=/etc/rancher/rke2/config/harvester-pss.yaml"
            path: /etc/rancher/rke2/config.yaml.d/99-harvester-pss.yaml
            permissions: 384
            owner: 0
            group: 0
          - content: |
              apiVersion: apiserver.config.k8s.io/v1
              kind: AdmissionConfiguration
              plugins:
                - name: PodSecurity
                  configuration:
                    apiVersion: pod-security.admission.config.k8s.io/v1
                    kind: PodSecurityConfiguration
                    defaults:
                      enforce: "baseline"
                      enforce-version: "latest"
                      audit: "baseline"
                      audit-version: "latest"
                      warn: "baseline"
                      warn-version: "latest"
                    exemptions:
                      usernames: []
                      runtimeClasses: []
                      namespaces: [calico-apiserver,
                                   calico-system,
                                   cattle-alerting,
                                   cattle-csp-adapter-system,
                                   cattle-elemental-system,
                                   cattle-epinio-system,
                                   cattle-externalip-system,
                                   cattle-fleet-local-system,
                                   cattle-fleet-system,
                                   cattle-gatekeeper-system,
                                   cattle-global-data,
                                   cattle-global-nt,
                                   cattle-impersonation-system,
                                   cattle-istio,
                                   cattle-istio-system,
                                   cattle-logging,
                                   cattle-logging-system,
                                   cattle-monitoring-system,
                                   cattle-neuvector-system,
                                   cattle-prometheus,
                                   cattle-provisioning-capi-system,
                                   cattle-resources-system,
                                   cattle-sriov-system,
                                   cattle-system,
                                   cattle-ui-plugin-system,
                                   cattle-windows-gmsa-system,
                                   cert-manager,
                                   cis-operator-system,
                                   fleet-default,
                                   ingress-nginx,
                                   istio-system,
                                   kube-node-lease,
                                   kube-public,
                                   kube-system,
                                   longhorn-system,
                                   rancher-alerting-drivers,
                                   security-scan,
                                   tigera-operator,
                                   harvester-system,
                                   harvester-public,
                                   rancher-vcluster]
            path: /etc/rancher/rke2/config/harvester-pss.yaml
            permissions: 384
            owner: 0
            group: 0
  paused: false

The cluster admin can apply this against the Harvester cluster using kubectl apply -f cloudinit-pss.yaml. The change requires a restart of the control plane nodes to ensure that the Elemental cloud-init directives are applied on boot. Once control plane nodes are rebooted, a default baseline pod security standard will be enforced against all current and subsequently created namespaces. The namespaces listed under exemptions will be skipped. Users are free to tweak the list, to better suit their use cases.

Security considerations​

Post application of a default PSS, end users, with permissions to create and edit namespaces, may still be able to override the respective policy by labeling their namespaces to support privileged workloads, for example, as follows:

kubectl label --overwrite ns <namespace> pod-security.kubernetes.io/enforce=privileged

To avoid this, we recommend users to create custom RBACs restricting who can create/update namespaces or to also deploy a Validating Admission Policy. The following policy will block namespace create/update requests containing a label pod-security.kubernetes.io/enforce, there by preventing namespace admins from changing the settings for their namespace.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: namespace-pss-label-rejection
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   [""]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["namespaces"]
  validations:
  - expression: |
      !("pod-security.kubernetes.io/enforce" in object.metadata.labels)
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: namespace-pss-label-rejection-binding
spec:
  policyName: namespace-pss-label-rejection
  validationActions: [Deny]

In case more tailored policies are needed, users can rely on security policy engines like Kubewarden's policy PSA Label Enforcer, or similar solution, to ensure that namespaces have the required PSS configuration for deployment in the cluster.

A security issue was discovered in Kubernetes where under certain conditions, an unauthenticated attacker with access to the pod network can achieve arbitrary code execution in the context of the ingress-nginx controller. This can lead to disclosure of secrets accessible to the controller. (Note that in the default installation, the controller can access all secrets cluster-wide.)

You can confirm the version of the RKE2 ingress-nginx pods by running this command on your Harvester cluster:

kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'

If the command returns one of the affected versions, disable the rke2-ingress-nginx-admission validating webhook configuration by performing the following steps:

1. On one of your control plane nodes, use kubectl to confirm the existence of the HelmChartConfig resource named rke2-ingress-nginx:

   $ kubectl -n kube-system get helmchartconfig rke2-ingress-nginx
   NAME                 AGE
   rke2-ingress-nginx   14d1h
   

   Copy

2. Use kubectl -n kube-system edit helmchartconfig rke2-ingress-nginx to add the following configurations to the resource:

   • .spec.valuesContent.controller.admissionWebhooks.enabled: false
   • .spec.valuesContent.controller.extraArgs.enable-annotation-validation: true

3. The following is an example of what the updated .spec.valuesContent configuration along with the default Harvester ingress-nginx configuration should look like:

   apiVersion: helm.cattle.io/v1
   kind: HelmChartConfig
   metadata:
     name: rke2-ingress-nginx
     namespace: kube-system
   spec:
     valuesContent: |-
       controller:
         admissionWebhooks:
           port: 8444
           enabled: false
         extraArgs:
           enable-annotation-validation: true
           default-ssl-certificate: cattle-system/tls-rancher-internal
         config:
           proxy-body-size: "0"
           proxy-request-buffering: "off"
         publishService:
           pathOverride: kube-system/ingress-expose
   

   Copy

   Exit the kubectl edit command execution to save the configuration.

   Harvester automatically applies the change once the content is saved.

   important

   The configuration disables the RKE2 ingress-nginx admission webhooks while preserving Harvester's default ingress-nginx configuration.

   If the HelmChartConfig resource contains other custom ingress-nginx configuration, you must retain them when editing the resource.

4. Verify that RKE2 deleted the rke2-ingress-nginx-admission validating webhook configuration.

   $ kubectl get validatingwebhookconfiguration rke2-ingress-nginx-admission
   Error from server (NotFound): validatingwebhookconfigurations.admissionregistration.k8s.io "rke2-ingress-nginx-admission" not found
   

   Copy

5. Verify that the ingress-nginx pods are restarted successfully.

   $ kubectl -n kube-system get po -lapp.kubernetes.io/instance=rke2-ingress-nginx
   NAME                                  READY   STATUS    RESTARTS   AGE
   rke2-ingress-nginx-controller-g8l49   1/1     Running   0          5s
   

   Copy

Once your Harvester cluster receives the RKE2 ingress-nginx patch, you can re-install the rke2-ingress-nginx-admission validating webhook configuration by removing the HelmChartConfig patch.

References​

• https://nvd.nist.gov/vuln/detail/CVE-2025-1974
• https://github.com/kubernetes/kubernetes/issues/131009
• https://github.com/rancher/rke2/issues/7953
• https://www.suse.com/support/kb/doc/?id=000021756

• An operating system was previously installed, particularly openSUSE Leap 15.5 or later and Harvester v1.3.1 or later. Other Linux distributions and recent versions of Windows may also be affected.
• UEFI secure boot is enabled.

This issue occurs when the Harvester ISO uses a shim bootloader that is older than the bootloader previously installed on the host. For example, the Harvester v1.3.1 ISO uses shim 15.4 but the system uses shim 15.8 after installation, which sets SBAT revocations for older shims. Subsequent attempts to boot the older shim on the ISO fail with the following error:

Verifying shim SBAT data failed: Security Policy Violation
Something has gone seriously wrong: SBAT self-check failed: Security Policy Violation

To mitigate the issue, perform the following workaround:

1. Disable Secure Boot.
2. Boot the ISO image and proceed with the installation.
3. Enable Secure Boot and boot into the installed system.

References​

• Harvester: Issue 7343
• openSUSE: Reset SBAT string for booting to old shim in old Leap image

KubeVirt Certificate Rotation Strategy​

KubeVirt provides a self-signed certificate mechanism that rotates both CA and certifcates on a defined recurring interval. You can check the setting certificateRotateStrategy by running the following command:

kubectl get kubevirt -n harvester-system -o yaml

By default, the value of certificateRotateStrategy is empty, which means that KubeVirt uses its default rotation settings and no manual configuration is required.

certificateRotateStrategy: {}

Configuration Fields​

You can use the following fields to configure certificateRotateStrategy.

• .ca.duration: Validity period of the CA certificate. The default value is "168h".
• .ca.renewBefore: Amount of time before a CA certificate expires during which a new certificate is issued. The default value is "33.6h".
• .server.duration: Validity period of server component certificates (for example, virt-api, virt-handler, and virt-operator). The default value is "24h".
• .server.renewBefore: Amount of time before a server certificate expires during which a new certificate is issued. The default value is "4.8h".

Example of a complete configuration:

certificateRotateStrategy:
  selfSigned:
    ca:
      duration: 168h
      renewBefore: 33.6h
    server:
      duration: 24h
      renewBefore: 4.8h

Certificate Rotation Triggers​

Certificate rotation can be triggered by several conditions. The following list only outlines key triggers and is not exhaustive.

• Missing certificate: A required certificate does not exist.
• Invalid CA signature: A certificate was not signed by the specified CA.
• Proactive renewal: The renewBefore value takes effect. A new certificate must be issued before the current one expires.
• CA expiration: The CA certificate has expired, so the certificate signed by the CA is also rotated.

When certificate rotation is triggered, you should see virt-operator log records similar to the following:

{"component":"virt-operator","level":"info","msg":"secret kubevirt-virt-api-certs updated","pos":"core.go:278","timestamp":"2024-12-06T08:02:01.045809Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-controller-certs updated","pos":"core.go:278","timestamp":"2024-12-06T08:02:01.056759Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-exportproxy-certs updated","pos":"core.go:278","timestamp":"2024-12-06T08:02:01.063530Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-virt-handler-server-certs updated","pos":"core.go:278","timestamp":"2024-12-06T08:02:01.068608Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-virt-handler-certs updated","pos":"core.go:278","timestamp":"2024-12-06T08:02:01.074555Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-operator-certs updated","pos":"core.go:278","timestamp":"2024-12-06T08:02:01.078719Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-export-ca updated","pos":"core.go:278","timestamp":"2024-12-06T08:03:36.063496Z"}
{"component":"virt-operator","level":"info","msg":"secret kubevirt-ca updated","pos":"core.go:278","timestamp":"2024-12-06T08:04:06.052750Z"}

References​

• Harvester: Issue 5798
• https://kubevirt.io/2020/KubeVirt-Security-Fundamentals.html
• https://github.com/kubevirt/kubevirt/blob/v1.1.1/pkg/virt-operator/resource/generate/components/secrets.go#L326
• https://github.com/kubevirt/kubevirt/blob/v1.1.1/pkg/virt-operator/resource/apply/certificates.go

1. The Harvester cluster is installed with 3+ nodes.

2. The Rancher Manager/Server is deployed independently. (Hereafter it is mentioned as Rancher Manager)

3. The Harvester cluster is imported to this Rancher Manager and works as a node driver.

4. The Rancher Manager deploys a couple of downstream K8s clusters, the machines/nodes of those clusters are backed by Harvester VMs.

5. There are also some traditional VMs deployed on the Harvester cluster, which have no direct connection with the Rancher Manager.

You plan to move those Harvester nodes geographically, or to power off the whole cluster for some time, it is essential to shutdown the Harvester cluster and restart later.

General Principle​

To safely shutdown a Harvester cluster, you need to follow the roughly reverse order of the cluster installation and the workload deployments.

Those facts need to be taken into account particularly:

1. The common methodology of Kubernetes operator/controller is to try things continuously until they meet expectations. When the cluster is shutting down node by node, if you don't stop those workloads in advance, they will try hard until the last node is off. It causes the last few nodes to have heavy CPU/memory/network/storage usage and increases the chance of data corruption.

2. Each Harvester node has limited capacity of CPU/memory/network/storage and the max-pod-number, when all workloads are crowded on the last few nodes, the unexpected pod eviction, scheduling failure and other phenomena may happen.

3. Harvester has embedded Longhorn as the default CSI driver, each PV can have 3 or more replicas, when replicas are rescheduled to other nodes, Longhorn will copy data from source node and rebuild the replica. Undoubtedly, stop the PVs as much as possible before the cluster shutdown to avoid the data moving.

4. Unlike normal Kubernetes deployments which have no PVs and are more flexible & agile to deploy anywhere on the cluster, the VMs are backed by massive sized PVs, slowly to move/migrate or even pinned on certain nodes to take the advantage of PCI-passthrough/vGPU/... and are much more sensitive to data consistency.

Needless to say, it is a bad practice to brutally power off the nodes on production environments.

1. Precondition​

1.1 Generate a Support-bundle File​

For trouble-shooting purpose, it is essential to follow this instruction to generate a support-bundle file before taking any actions. And make sure the workload namespaces are added.

1.2 Keep Network Stability​

A good practice is to have detailed documents about the infrastructure related settings.

• The bare metal server NIC slot/port connections with the remote (ToR) Switches.

• The VLAN for the management network.

• (Optional) The DHCP Server, ip-pools and ip-mac bindings for the Harvester cluster if DHCP server is used. If there is no fixed IP binding, when the server restarts after some days it may get a different IP from the DHCP server.

• The VLANs for the VM networks, the CIDRs, default gateways and optional DHCP servers.

• NTP servers.

• DNS servers.

• (Optional) The http proxy.

• (Optional) The private containerd-registry.

• (Optional) The firewall configurations.

See the Harvester ISO Installation to review the infrastructure related settings for the Harvester cluster.

Before the Harvester cluster is restarted later, check and test those settings again to make sure the infrastructure is ready.

2. Backup​

(Optional) Backup VMs if Possible​

It is always a good practice to backup things before a whole cluster shutdown.

(Optional) Backup Downstream K8s Clusters if Possible​

Harvester doesn't touch the (Rancher Manager managed) downstream K8s clusters' workload, when they are not able to be migrated to other node drivers, suggests to backup those clusters.

(Optional) Stop or Migrate Downstream K8s Clusters if Possible​

Harvester doesn't touch the downstream K8s clusters' workload, but suggests to stop or migrate the downstream clusters to avoid your service interruption.

3. Shutdown Workloads​

3.1 Shutdown Traditional VMs​

1. Shutdown VM from the VM shell (e.g. Linux shutdown command), the OS itself will save data to disks.

2. Check the VM status from Harvester UI - VM page, when it is not Off, then click the Stop command.

3.2 Shutdown Rancher Downstream Cluster Machines(VMs)​

Suppose your Harvester cluster was imported to Rancher as a node driver before.

When Rancher deploys a downstream cluster on node driver Harvester, it creates a couple of VMs on Harvester automatically. Directly stopping those VMs on Harvester is not a good practice when Rancher is still managing the downstream cluster. For example, Rancher may create new VMs if you stop them from Harvester.

If you have got a solution to shutdown those downstream clusters, and check those VMs are Off; or there is no downstream clusters, then jump to the step disable some addons.

Unless you have already deleted all the downstream clusters which are deploy on this Harvester, DO NOT remove this imported Harvester from the Rancher Manager. Harvester will get a different driver-id when it is imported later, but those aforementioned downstream clusters are connected to driver-id.

To safely shutdown those VMs but still keep the Rancher Manager managed downstream cluster alive, please follow the steps below:

Disconnect Harvester from the Rancher Manager​

The cattle-cluster-agent-*** pod is the direct connection between Rancher Manager and Harvester cluster, and this pod is monitored and managed by the embedded Rancher in Harvester, scaling down this pod does not work. The embedded Rancher will scale it up automatically.

Run steps below to suspend the connection.

All following CLI commands are executed upon Harvester cluster.

1. Set the management.cattle.io/scale-available of deployment rancher to be "" instead of "3" or other values.

This change will stop the auto-scaling.

harvester$ kubectl edit deployment -n cattle-system rancher
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
...
    management.cattle.io/scale-available: "3"  // record this value, and change it to ""
...
  generation: 16
  labels:
    app: rancher
    app.kubernetes.io/managed-by: Helm
...
  name: rancher
  namespace: cattle-system

2. Scale down the rancher deployment.

harvester$ kubectl scale deployment -n cattle-system rancher --replicas=0
deployment.apps/rancher scaled


harvester$ get deployment -n cattle-system rancher
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
rancher   0/0     0            0           33d

3. Make sure the rancher-* pods are gone.

Check the rancher-* pods on cattle-system are gone, if any of them is stucking at Terminating, use kubectl delete pod -n cattle-system rancher-pod-name --force to delete it.

harvester$ kubectl get pods -n cattle-system
NAME                                         READY   STATUS        RESTARTS       AGE
..
rancher-856f674f7d-5dqb6                     0/1     Terminating   0              3d22h
rancher-856f674f7d-h4vsw                     1/1     Running       23 (68m ago)   33d
rancher-856f674f7d-m6s4r                     0/1     Pending       0              3d19h
...

4. Scale down the cattle-cluster-agent deployment.

harvester$ kubectl scale deployment -n cattle-system cattle-cluster-agent --replicas=0
deployment.apps/cattle-cluster-agent scaled


harvester$ kubectl get deployment -n cattle-system
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
cattle-cluster-agent        0/0     0            0           23d

Please note:

1. From now on, this Harvester is Unavailable on the Rancher Manager.

2. The Harvester WebUI returns 503 Service Temporarily Unavailable, all operations below can be done via kubectl.

Shutdown Rancher Downstream Cluster Machines(VMs)​

1. Shutdown VM from the VM shell (e.g. Linux shutdown command).

2. Check the vmi instances, if any is still Running, stop it.

harvester$ kubectl get vmi
NAMESPACE   NAME   AGE    PHASE     IP            NODENAME   READY
default     vm1    5m6s   Running   10.52.0.214   harv41     True


harvester$ virtctl stop vm1 --namespace default
VM vm1 was scheduled to stop

harvester$ kubectl get vmi -A
NAMESPACE   NAME   AGE    PHASE     IP            NODENAME   READY
default     vm1    5m6s   Running   10.52.0.214   harv41     False


harvester$ kubectl get vmi -A
No resources found

harvester$ kubectl get vm -A
NAMESPACE   NAME   AGE   STATUS    READY
default     vm1    7d    Stopped   False

3.3 Disable Some Addons​

Harvester has some addons which are backed by PVCs, it is necessary to disable them.

The rancher-monitoring addon should be disabled.

The experimental Rancher Manager addon should be disabled.

For other addons, please follow the Harvester document to keep or disable them.

From Harvester UI addon page, write down those none-Disabled addons, click Disable menu to disable them, wait until the state becomes Disabled

From CLI:

$ kubectl get addons.harvesterhci.io -A

NAMESPACE                  NAME                    HELMREPO                                                 CHARTNAME                         ENABLED
cattle-logging-system      rancher-logging         http://harvester-cluster-repo.cattle-system.svc/charts   rancher-logging                   false
cattle-monitoring-system   rancher-monitoring      http://harvester-cluster-repo.cattle-system.svc/charts   rancher-monitoring                true
harvester-system           harvester-seeder        http://harvester-cluster-repo.cattle-system.svc/charts   harvester-seeder                  false
harvester-system           nvidia-driver-toolkit   http://harvester-cluster-repo.cattle-system.svc/charts   nvidia-driver-runtime             false
harvester-system           pcidevices-controller   http://harvester-cluster-repo.cattle-system.svc/charts   harvester-pcidevices-controller   false
harvester-system           vm-import-controller    http://harvester-cluster-repo.cattle-system.svc/charts   harvester-vm-import-controller    false

Example: disable rancher-monitoring

$ kubectl edit addons.harvesterhci.io -n cattle-monitoring-system rancher-monitoring

...
spec:
  chart: rancher-monitoring
  enabled: false               // set this field to be false
...

3.4 (Optional) Disable other Workloads​

If you have deployed some customized workloads on the Harvester cluster directly, it is better to disable/remove them.

3.5 Check Longhorn Volumes​

The volumes should be in state detached, check the related workload if some volumes are still in state attached.

harvester$ kubectl get volume -A
NAMESPACE         NAME                                       DATA ENGINE   STATE      ROBUSTNESS   SCHEDULED   SIZE          NODE     AGE
longhorn-system   pvc-3323944c-00d9-4b35-ae38-a00b1e8a8841   v1            detached   unknown                  5368709120             13d
longhorn-system   pvc-394713a4-d08c-4a45-bf7a-d44343f29dea   v1            attached   healthy                  6442450944    harv41   8d    // still attached and in use
longhorn-system   pvc-5cf00ae2-e85e-413e-a4f1-8bc4242d4584   v1            detached   unknown                  2147483648             13d
longhorn-system   pvc-620358ca-94b3-4bd4-b008-5c144fd815c9   v1            attached   healthy                  2147483648    harv41   8d    // still attached and in use
longhorn-system   pvc-8174f05c-919b-4a8b-b1ad-4fc110c5e2bf   v1            detached   unknown                  10737418240            13d

4. Shutdown Nodes​

Get all nodes from Harvester WebUI Host Management.

From CLI:

harvester$ kubectl get nodes -A
NAME     STATUS   ROLES                       AGE   VERSION
harv2    Ready    <none>                      24d   v1.27.10+rke2r1  // worker node
harv41   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1  // control-plane node
harv42   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1  // control-plane node
harv43   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1  // control-plane node

4.1 Shutdown the Worker Nodes​

1. SSH to the Harvester worker nodes.

2. Run command sudo -i shutdown.

$ sudo -i shutdown

Shutdown scheduled for Mon 2024-07-22 06:58:56 UTC, use 'shutdown -c' to cancel.

3. Wait until all those nodes are down.

4.2 Shutdown Control-plane Nodes and Witness Node​

So far, there are generally three control-plane nodes left, and three etcd-* pods are running in kube-system namespaces.

The first step is to find which one of the etcd-* pod is running as the leader.

Run below command upon any of the etcd-* pod, note the IS LEADER column.

$ kubectl exec -n kube-system etcd-harv41 -- env ETCDCTL_API=3 etcdctl endpoint status --cluster -w table --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key

+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.122.141:2379 | c70780b7862269c9 |   3.5.9 |   34 MB |      true |      false |        45 |    6538756 |            6538756 |        |
| https://192.168.122.142:2379 | db04095b49eb5352 |   3.5.9 |   34 MB |     false |       true |        45 |    6538756 |            6538756 |        |
| https://192.168.122.143:2379 | c27585769b2ce977 |   3.5.9 |   34 MB |     false |       true |        45 |    6538756 |            6538756 |        |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Witness Node​

If your cluster has one Witness Node and the etcd leader happens to be on this node.

harvester$ kubectl get nodes -A
NAME     STATUS     ROLES                       AGE    VERSION
harv2    Ready      <none>                      25d    v1.27.10+rke2r1  // worker node
harv41   Ready      control-plane,etcd,master   55d    v1.27.10+rke2r1  // control-plane node
harv42   Ready      control-plane,etcd,master   55d    v1.27.10+rke2r1  // control-plane node
harv43   Ready      etcd                         1d    v1.27.10+rke2r1  // witness node

+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.122.141:2379 | c70780b7862269c9 |   3.5.9 |   34 MB |     false |       true |        46 |    6538829 |            6538829 |        |
| https://192.168.122.142:2379 | db04095b49eb5352 |   3.5.9 |   34 MB |     false |       true |        46 |    6538829 |            6538829 |        |
| https://192.168.122.143:2379 | a21534d02463b347 |   3.5.9 |   34 MB |      true |      false |        46 |    6538829 |            6538829 |        |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Run kubectl delete pod -n kube-system etcd-name command to delete the etcd pod on the witness node to trigger the pod replacement and leader re-election so that the etcd leader will be located on one of the control-plane nodes. Check the etcd leader again to make sure.

+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.122.141:2379 | c70780b7862269c9 |   3.5.9 |   34 MB |      true |      false |        47 |    6538833 |            6538833 |        |
| https://192.168.122.142:2379 | db04095b49eb5352 |   3.5.9 |   34 MB |     false |       true |        47 |    6538833 |            6538833 |        |
| https://192.168.122.143:2379 | a21534d02463b347 |   3.5.9 |   34 MB |     false |       true |        47 |    6538833 |            6538833 |        |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

To now, the etcd has three running instances and the leader is located on the control-plane node.

Shutdown the two IS LEADER == false nodes one by one.

harvester-node-shell$ sudo -i shutdown

4.3 Shutdown the Last Control-plane Node​

Shutdown the last IS LEADER == true node. Remember its physical information for restarting it first in the steps below.

harvester-last-node-shell$ sudo -i shutdown

5. Restart​

If the Harvester cluster has been moved to a new location, or has been off for days, or your infrastructure has changes, check and test the network stability.

5.1 Restart the Control-plane Nodes and the Witness Node​

The first step is to start those etcd located nodes one after another.

Restart the Leader Control-plane Node​

Power on the last shutdown node first. After about three minutes, continue the next step.

When you check the etcd pod log on this node, the following message may be observed.

sent MsgPreVote request to db04095b49eb5352 at term 5

"msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"db04095b49eb5352","rtt":"0s","error":"dial tcp 192.168.122.142:2380: connect: no route to host"

The etcd is wating for the other two members to be online and then vote a leader.

Restart the Rest of Control-plane Nodes and the Witness Node​

Power on the rest nodes which also hosted the etcd pod before.

Wait until all the three control-plane nodes or possibly two control-plane and one witness nodes are Ready.

From CLI:

harvester$ kubectl get nodes -A
NAME     STATUS   ROLES                       AGE   VERSION
harv41   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1
harv42   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1
harv43   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1

The etcd forms a quorum and can tolerant the failure of one node.

Check the VIP​

The following EXTERNAL-IP should be the same as the VIP of the Harvester cluster.

harvester$ kubectl get service -n kube-system ingress-expose
NAME             TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)                      AGE
ingress-expose   LoadBalancer   10.53.50.107   192.168.122.144   443:32701/TCP,80:31480/TCP   34d

5.2 Restart the Worker Nodes​

Wait until all nodes are Ready.

From CLI:

harvester$ kubectl get nodes -A
NAME     STATUS   ROLES                       AGE   VERSION
harv2    Ready    <none>                      24d   v1.27.10+rke2r1  // worker node
harv41   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1
harv42   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1
harv43   Ready    control-plane,etcd,master   54d   v1.27.10+rke2r1

Healthy Check​

Basic Components​

Harvester deploys some basic components on the following namespaces. When a bare-metal server is powered on, it may take upto around 15 minutes for the Harvester OS to be running and all the deployments on this node to be ready.

If any of them continues to show the status like Failed/CrashLoopBackOff, a troubleshooting is needed to confirm the root cause.

NAMESPACE                         NAME                                                     READY   STATUS      RESTARTS       AGE
cattle-fleet-local-system         fleet-agent-645766877f-bt424                             1/1     Running     0              11m

cattle-fleet-system               fleet-controller-57f78dcd48-5tkkj                        1/1     Running     4 (14m ago)    42h
cattle-fleet-system               gitjob-d5bb7b548-jscgk                                   1/1     Running     2 (14m ago)    42h

cattle-system                     harvester-cluster-repo-6c6458bd46-7jcrl                  1/1     Running     2 (14m ago)    42h
cattle-system                     system-upgrade-controller-6f86d6d4df-f8jg7               1/1     Running     2 (14m ago)    42h
cattle-system                     rancher-7bc9d94b87-g4k4v                                 1/1     Running     3 (14m ago)    42h  // note: if embedded Rancher was stopped in the above steps, it is not Running now
cattle-system                     rancher-webhook-6c5c6fbb65-2cbbs                         1/1     Running     2 (14m ago)    42h

harvester-system                  harvester-787b467f4-qlfwt                                1/1     Running     2 (14m ago)    39h
harvester-system                  harvester-load-balancer-56d9c8758c-cvcmk                 1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-load-balancer-webhook-6b4d4d9d6b-4tsgl         1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-network-controller-9pzxh                       1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-network-controller-manager-69bcf67c7f-44zqj    1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-network-webhook-6c5d48bdf5-8kn9r               1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-node-disk-manager-c4c5k                        1/1     Running     3 (14m ago)    42h
harvester-system                  harvester-node-manager-qbvbr                             1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-node-manager-webhook-6d8b48f559-m5shk          1/1     Running     2 (14m ago)    42h
harvester-system                  harvester-webhook-87dc4cdd8-jg2q6                        1/1     Running     2 (14m ago)    39h
harvester-system                  kube-vip-n4s8l                                           1/1     Running     3 (14m ago)    42h
harvester-system                  virt-api-799b99fb65-g8wgq                                1/1     Running     2 (14m ago)    42h
harvester-system                  virt-controller-86b84c8f8f-4hhlg                         1/1     Running     2 (14m ago)    42h
harvester-system                  virt-controller-86b84c8f8f-krq4f                         1/1     Running     3 (14m ago)    42h
harvester-system                  virt-handler-j9gwn                                       1/1     Running     2 (14m ago)    42h
harvester-system                  virt-operator-7585847fbc-hvs26                           1/1     Running     2 (14m ago)    42h

kube-system                       cloud-controller-manager-harv41                          1/1     Running     5 (14m ago)    42h
kube-system                       etcd-harv41                                              1/1     Running     2              42h
kube-system                       harvester-snapshot-validation-webhook-8594c5f8f8-8mk57   1/1     Running     2 (14m ago)    42h
kube-system                       harvester-snapshot-validation-webhook-8594c5f8f8-dkjmf   1/1     Running     2 (14m ago)    42h
kube-system                       harvester-whereabouts-cpqvl                              1/1     Running     2 (14m ago)    42h

kube-system                       kube-apiserver-harv41                                    1/1     Running     2              42h
kube-system                       kube-controller-manager-harv41                           1/1     Running     4 (14m ago)    42h
kube-system                       kube-proxy-harv41                                        1/1     Running     2 (14m ago)    42h
kube-system                       kube-scheduler-harv41                                    1/1     Running     2 (14m ago)    42h
kube-system                       rke2-canal-d5kmc                                         2/2     Running     4 (14m ago)    42h
kube-system                       rke2-coredns-rke2-coredns-84b9cb946c-qbwnb               1/1     Running     2 (14m ago)    42h
kube-system                       rke2-coredns-rke2-coredns-autoscaler-b49765765-6bjsk     1/1     Running     2 (14m ago)    42h
kube-system                       rke2-ingress-nginx-controller-cphgw                      1/1     Running     2 (14m ago)    42h
kube-system                       rke2-metrics-server-655477f655-gsnsc                     1/1     Running     2 (14m ago)    42h
kube-system                       rke2-multus-8nqg4                                        1/1     Running     2 (14m ago)    42h
kube-system                       snapshot-controller-5fb6d65787-nmjdh                     1/1     Running     2 (14m ago)    42h
kube-system                       snapshot-controller-5fb6d65787-phvq7                     1/1     Running     3 (14m ago)    42h

longhorn-system                   backing-image-manager-5c32-ea70                          1/1     Running     0              13m
longhorn-system                   csi-attacher-749459cf65-2x792                            1/1     Running     6 (13m ago)    42h
longhorn-system                   csi-attacher-749459cf65-98tj4                            1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-attacher-749459cf65-nwglq                            1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-provisioner-775b4f76f4-h9mwd                         1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-provisioner-775b4f76f4-nvjzt                         1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-provisioner-775b4f76f4-zvd6w                         1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-resizer-68867d54f5-4hf5j                             1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-resizer-68867d54f5-fs9ht                             1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-resizer-68867d54f5-ht5hj                             1/1     Running     6 (13m ago)    42h
longhorn-system                   csi-snapshotter-8469656cc7-6c47f                         1/1     Running     6 (13m ago)    42h
longhorn-system                   csi-snapshotter-8469656cc7-9kk2v                         1/1     Running     5 (13m ago)    42h
longhorn-system                   csi-snapshotter-8469656cc7-vf9z4                         1/1     Running     5 (13m ago)    42h
longhorn-system                   engine-image-ei-94d5ee6c-pqx9h                           1/1     Running     2 (14m ago)    42h
longhorn-system                   instance-manager-beb75434e263a2aa9eedc0609862fed2        1/1     Running     0              13m
longhorn-system                   longhorn-csi-plugin-85qm7                                3/3     Running     14 (13m ago)   42h
longhorn-system                   longhorn-driver-deployer-6448498bc6-sv857                1/1     Running     2 (14m ago)    42h
longhorn-system                   longhorn-loop-device-cleaner-bqg9v                       1/1     Running     2 (14m ago)    42h
longhorn-system                   longhorn-manager-nhxbl                                   2/2     Running     6 (14m ago)    42h
longhorn-system                   longhorn-ui-7f56fcf5ff-clc8b                             1/1     Running     6 (13m ago)    42h
longhorn-system                   longhorn-ui-7f56fcf5ff-m95sh                             1/1     Running     7 (13m ago)    42h