How to Ensure Your Kubernetes 1.37 Upgrade Doesn't Corrupt Your Nodes and Static Pods

You’ve patched the critical CVEs and updated your operators. The pre-flight checks for the Kubernetes 1.37 API deprecations all passed. You start draining nodes, and the first few cycle perfectly—then one goes NotReady and never comes back.

TL;DR: Kubernetes 1.37 upgrades fail silently when nodes have incompatible cgroup configurations or when static pod manifests rely on now-immutable Secrets and ConfigMaps. This post provides the crictl and kubectl commands to audit your nodes for these specific incompatibilities before you drain them, preventing node corruption and upgrade-day disasters.

What you’ll walk away with:

• A one-liner to discover every static pod in your cluster.
• The crictl command to verify cgroup v2 compatibility on any node.
• A checklist for auditing static pod manifests for post-upgrade failures.
• The ability to distinguish between a DaemonSet and a static pod from the command line.

What’s the risk with static pods during an upgrade?

The primary risk is that static pods are invisible to the Kubernetes control plane’s admission and scheduling logic, yet they are critical for the cluster’s operation. A static pod is a pod managed directly by the kubelet daemon on a specific node, without the API server observing it. Because the kubelet creates them by reading local files, a syntax error in a manifest or an incompatibility with the upgraded runtime can prevent a node from rejoining the cluster, and kubectl won’t tell you why.

This node-local control flow is why they are often used for bootstrapping control plane components like etcd or the kube-apiserver itself. It’s also why they’re so dangerous during an upgrade. An issue with a static pod manifest is not a deployment failure; it’s a node failure.

graph TD
    subgraph Node Filesystem
        Manifest["/etc/kubernetes/manifests/pod.yaml"]
    end

    subgraph Kubernetes Control Plane
        APIServer["API Server"]
    end

    Kubelet["Kubelet"] -- "Watches" --> Manifest
    Kubelet -- "Creates Pod" --> ContainerRuntime["Container Runtime"]
    Kubelet -- "Reports Status (Read-Only)" --> APIServer

    APIServer -- "Cannot Create/Update" --x Kubelet

The control plane can’t fix what it doesn’t control. If you’ve ever had a cluster issue that felt like a phantom pod problem, a broken static pod is a likely culprit.

Always inspect static pod manifests during pre-upgrade checks; they are a control-plane blind spot that can take an entire node offline.

How do I find all static pods in my cluster?

You can list all static pods by querying for pods with the kubernetes.io/config.source: file annotation. The kubelet automatically adds this annotation when it creates a pod from a local manifest file. This lets you find every single one without SSHing into nodes to check for manifest files.

Run this command to get a list of all static pods and the nodes they are running on:

kubectl get pods --all-namespaces -o=jsonpath='{range .items[?(@.metadata.annotations.hasOwnProperty("kubernetes.ioio/config.source"))]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.nodeName}{"\n"}{end}'

The output will be a clean, tab-separated list you can easily parse or review.

kube-system	etcd-control-plane-1	        control-plane-1
kube-system	kube-apiserver-control-plane-1	control-plane-1
kube-system	kube-controller-manager-1	control-plane-1
kube-system	kube-scheduler-control-plane-1	control-plane-1

The --field-selector spec.nodeName=<node-name> flag is your best tool for isolating potential problems to a single node before you drain it.

How do I check for cgroup v2 compatibility?

To check for cgroup v2 compatibility, SSH into the node and use crictl info to inspect the container runtime configuration. The output contains a cgroupDriver field; for modern Kubernetes versions, this value must be systemd, which operates on the cgroup v2 hierarchy. An output of cgroupfs signals an impending failure.

The old way: Blindly drain a node, run the upgrade, and uncordon it. When the node fails to report Ready, you SSH in and run journalctl -u kubelet to find cryptic cgroup errors, losing 30 minutes trying to figure out why the runtime won’t start pods.

The new way is a two-minute pre-flight check on the node before you even drain it.

1. SSH into the control plane or worker node you plan to upgrade.
2. Run the crictl info command and grep for the cgroup driver.

crictl info | grep -i cgroup

A healthy, cgroup v2-compatible node will show systemd.

      "cgroupDriver": "systemd",

If you see cgroupfs, stop. That node cannot be upgraded until its runtime and OS are configured for cgroup v2. While Kubernetes 1.37 can technically work with cgroupfs, subsequent versions and their corresponding container runtimes are removing support entirely. For instance, CRI-O 1.28, a common runtime paired with Kubernetes 1.28, completely removed cgroup v1 support.

Feature	cgroup v1 (cgroupfs)	cgroup v2 (systemd)	Winner
Hierarchy	Multiple, per-controller	Single unified hierarchy	v2
Controller Mgt	Messy, process can be in different groups	Clean, process is in one group	v2
Future Support	Deprecated, being removed	Required for modern features	v2
Best For	Legacy systems on a short timeline to EOL	All Kubernetes 1.37+ production workloads	v2

This isn’t just a technical debt item; it’s a hard blocker for future upgrades and can lead to bizarre node-level resource contention issues similar to the ones seen with IPVS connection bugs.

Do not start a Kubernetes 1.37 upgrade unless crictl info reports the systemd cgroup driver on every single node.

How do I audit manifests for immutability issues?

Audit static pod manifests by ensuring that any mounted Secret or ConfigMap is not expected to be modified after the pod starts. Starting in Kubernetes 1.22, Secrets and ConfigMaps can be marked as immutable, and this behavior is now the reliable default. Any pod relying on an external process to update a mounted Secret’s data in-place will fail or use stale data after an upgrade.

Let’s say you found a static pod auth-proxy on your node.

First, get the manifest path from the pod’s annotations.

kubectl get pod auth-proxy -n kube-system -o=jsonpath='{.metadata.annotations.kubernetes\.io/config\.mirror}'

This gives you the pod hash. The source is in /etc/kubernetes/manifests/. SSH into the node and inspect /etc/kubernetes/manifests/auth-proxy.yaml.

Click to see a problematic `auth-proxy.yaml` manifest

apiVersion: v1
kind: Pod
metadata:
  name: auth-proxy
  namespace: kube-system
spec:
  containers:
  - name: proxy
    image: my-corp/auth-proxy:1.2
    volumeMounts:
    - name: api-token-vol
      mountPath: /etc/proxy/token
      readOnly: false # This is the red flag
  volumes:
  - name: api-token-vol
    secret:
      secretName: proxy-api-token
      # This secret is updated daily by a cronjob

The readOnly: false on a Secret volume mount is a huge code smell. A separate process updating the proxy-api-token secret will no longer propagate those changes into the running pod’s volume.

The fix is to re-architect the token refresh mechanism. Instead of updating the Secret in place, the proxy container should be responsible for fetching a new token itself, or the deployment process should create a new, versioned secret and roll the static pod.

Here’s the change to make the volume mount read-only, forcing the developers to adopt a safer pattern.

--- a/auth-proxy.yaml
+++ b/auth-proxy.yaml
@@ -8,7 +8,7 @@
     volumeMounts:
     - name: api-token-vol
       mountPath: /etc/proxy/token
-      readOnly: false
+      readOnly: true
   volumes:
   - name: api-token-vol
     secret:

Your pre-upgrade audit checklist for static pods should include:

• No readOnly: false mounts for Secrets or ConfigMaps.
• No reliance on a “file-watcher” sidecar that expects mounted config to change.
• The pod’s image and command are compatible with the new container runtime version (e.g., Docker Shim removal).
• All security contexts and permissions are still valid, as static pods often require high privileges. You can cross-reference best practices with a guide like the CKS exam reference.

Treat all mounted Secrets and ConfigMaps as immutable by default; if you need dynamic configuration, use a controller that can safely roll pods.

Bottom Line

An upgrade to Kubernetes 1.37 isn’t just about API versions. Node-level components and assumptions about pod behavior can break in subtle ways that only appear under pressure. Before you type kubectl drain, verify every node’s cgroup driver and audit every static pod manifest for dependencies on mutable configuration.

This post focused on identifying node-level risks. Next, we’ll tackle the network policies and CNI plugin changes in 1.37 that can silently segment your cluster.

FAQ

Can I run cgroup v1 on Kubernetes 1.37?

Yes, but you absolutely should not. While technically possible, it is deprecated and support is actively being removed from container runtimes like CRI-O and containerd. All production clusters should be migrated to cgroup v2 with the systemd driver before upgrading.

What is the difference between a static pod and a DaemonSet?

A static pod is managed by the kubelet on a single, specific node and is invisible to the scheduler. A DaemonSet is managed by the control plane and ensures that a copy of a pod runs on all (or a subset of) nodes in the cluster. Use a DaemonSet for cluster-wide agents and static pods only for bootstrapping node-local components.

How do I edit a static pod?

You don’t edit a running static pod with kubectl edit. You must edit the manifest file (e.g., in /etc/kubernetes/manifests/) on the node’s filesystem. The kubelet will detect the change and automatically stop the old pod and start a new one based on the updated file.

Does kubectl delete pod work on a static pod?

Temporarily. If you kubectl delete a static pod, the API server will remove it. However, the kubelet on the node will immediately notice the pod is gone and, reading its local manifest file, will recreate it. The correct way to remove it is to delete the manifest file from the node.

What happens if a static pod fails to start?

The kubelet will continuously try to start it, logging errors to the kubelet journal (journalctl -u kubelet). The node may go into a NotReady state if the failing static pod is critical (like the API server). The pod will appear in kubectl get pods with a status like CrashLoopBackOff or Pending, but you must check the node’s logs for the root cause.

Part of the series: k8s-1-37-upgrade

1. Why Your Kubernetes Cluster Might Silently Break After a 1.37 Upgrade
2. How to Ensure Your Kubernetes 1.37 Upgrade Doesn't Corrupt Your Nodes and Static Pods (you are here)

Further Reading

• https://kubernetes.io/docs/setup/production-environment/container-runtimes/
• https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/
• https://kubernetes.io/docs/concepts/configuration/secret/

---

🚀 Ready to get hands-on? Spin up an interactive AI or Kubernetes Sandbox at Aicademy Labs for free.