Is Your Kubernetes 1.37 Cluster Actually Stable? A 5-Minute Post-Upgrade Audit

Upgrading to Kubernetes 1.37 often reports success while masking fatal node-level regressions. Control planes restart silently, static pods drift from their configuration, and legacy networking rules drop packets in the background.

TL;DR: Running kubectl get nodes isn’t enough to verify a Kubernetes 1.37 upgrade. Silent failures in Kubelet, static pod corruption, and lingering IPVS configurations can break your cluster hours after the upgrade finishes. This post gives you a 5-minute checklist, specific command-line verifications, and a scoring rubric to prove your cluster is actually stable.

Run this audit in 5 minutes. Execute these commands sequentially on a control plane node to surface hidden post-upgrade regressions.

1. Check Kubelet journal for deprecated component errors.

   journalctl -u kubelet --since "1 hour ago" | grep -iE "error|warning|failed" | grep -v "not found"
   

May 20 10:15:22 aicademy-cp01 kubelet[1234]: E0520 10:15:22.123456    1234 server.go:300] "Failed to parse IPVS proxier rules" err="deprecated"

1. Verify event stream for unexpected scheduling failures.

   kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp | tail -n 5
   

default   4m20s   Warning   FailedScheduling   pod/api-server-xyz   0/3 nodes are available: 1 Insufficient DRA resources.

1. Audit Static Pod definitions for version drift.

   find /etc/kubernetes/manifests -name "*.yaml" -exec grep -H "image:" {} \;
   

/etc/kubernetes/manifests/kube-apiserver.yaml:    image: registry.k8s.io/kube-apiserver:v1.37.0
/etc/kubernetes/manifests/kube-controller-manager.yaml:    image: registry.k8s.io/kube-controller-manager:v1.37.0

Scoring Rubric:

• 0 findings = Healthy.
• 1-2 findings = Investigate before routing production traffic.
• 3+ findings = Rollback immediately.

How Do You Detect Silent IPVS Failures After a 1.37 Upgrade?

You detect silent IPVS failures by checking the kube-proxy logs for dropped synchronization rules. Kubernetes 1.37 deprecates legacy IPVS networking options, meaning older configurations drop traffic instead of crash-looping. A healthy node shows successful local endpoint synchronization without deprecated API warnings.

IPVS (IP Virtual Server) is an in-kernel load balancer that routes traffic to your cluster services. If you run a custom proxy layer, review Why Your Kubernetes Cluster Might Silently Break After a 1.37 Upgrade. Defaulting to iptables or eBPF avoids these lingering migration issues entirely.

View verbose kube-proxy deprecation log

I0520 10:15:22.123456    1234 proxier.go:500] "Syncing IPVS rules"
W0520 10:15:22.123500    1234 proxier.go:512] "Detected deprecated IPVS strict ARP configuration. This behavior will be removed in 1.38."
E0520 10:15:22.123600    1234 proxier.go:550] "Failed to bind VIP" ip="10.96.0.10"

Always drain a node and verify kube-proxy connectivity metrics before assuming IPVS migrated cleanly.

Why Are Static Pods Crashing After the API Server Updates?

Static pods crash after the API server updates because the Kubelet restarts them forcefully when their disk manifests drift from the runtime configuration. Kubernetes 1.37 tightens validation on host-mounted volumes, causing previously tolerated syntax to immediately fail pod startup. You must manually reconcile the YAML files.

If your API server refuses to come back online, you likely have malformed arguments in /etc/kubernetes/manifests/kube-apiserver.yaml. Read How to Ensure Your Kubernetes 1.37 Upgrade Doesn’t Corrupt Your Nodes and Static Pods to fix these specific volume mounts.

Use this checklist to verify your control plane configurations before restarting the Kubelet daemon:

• Verify image tags match v1.37.x exactly.
• Remove deprecated --enable-admission-plugins=PodSecurityPolicy flags.
• Confirm hostPath volumes use strictly valid directory formats.

-    - --enable-admission-plugins=NodeRestriction,PodSecurityPolicy
+    - --enable-admission-plugins=NodeRestriction,PodSecurity

Back up /etc/kubernetes/manifests/ before running kubeadm upgrade apply to guarantee a fast rollback.

How Can You Verify GPU Allocations in Kubernetes 1.37?

You verify GPU allocations by inspecting the node’s resource capacities against the new Dynamic Resource Allocation API. Kubernetes 1.37 changes how external resources map to pods, meaning legacy device plugins report available hardware that the scheduler cannot consume. You must validate the custom resource claims directly.

Dynamic Resource Allocation (DRA) is a flexible Kubernetes subsystem for requesting and sharing external hardware like GPUs across multiple containers. If your Aicademy rendering workloads fail to schedule, you must migrate to this new API structure. Refer to One kubectl Command That Unlocks GPU-Aware Scheduling with Kubernetes 1.37 DRA for the exact migration syntax.

sequenceDiagram
    participant P as "Pod"
    participant S as "Scheduler"
    participant D as "DRA Controller"
    P->>S: "Request ResourceClaim"
    S->>D: "Evaluate Capacity"
    D-->>S: "Bind Allocation"
    S-->>P: "Schedule to Node"

Migrate all legacy device plugin manifests to native DRA ResourceClaims for any GPU workloads running on 1.37.

What Are the Most Reliable Tools for Verifying Cluster Health?

You establish cluster health by combining native metrics, conformance tests, and direct node-level OS checks. No single tool catches every post-upgrade failure mode. You must use overlapping methodologies to guarantee stability.

Tool	Verification Focus	Kubernetes 1.37 Support	Best For
Kube-state-metrics	Cluster-level resources	Native	Long-term API monitoring
Sonobuoy	End-to-end conformance	Official release	Pre-production validation
Audit Scripts	Node-level OS logs	Custom implementations	5-minute manual checks

For engineers looking to build predictable, automated upgrade pipelines rather than reacting to errors, structured practice helps. Review Aicademy Workshops to work through controlled cluster upgrade scenarios.

Prefer native OS log parsing over dashboard summaries during the first 60 minutes post-upgrade.

Bottom Line

Do not trust a successful kubectl get nodes output immediately following an upgrade. You must verify Kubelet logs, static pod configurations, and DRA capacities to confirm actual cluster stability. Run the 5-minute audit checklist above on every control plane node before routing production traffic back to the cluster.

FAQ

How do you check which version of kubelet is running on a node?

Run kubectl get nodes -o wide to see the kubelet version for all connected machines. For a direct local check on a specific node, run kubelet --version in the terminal.

Why does kube-proxy log strict ARP deprecation warnings in 1.37?

Kubernetes 1.37 phases out certain legacy IPVS behaviors in favor of cleaner networking models. You must update your kube-proxy config map to reflect the new strict ARP requirements before version 1.38.

What is the default volume mount validation policy in Kubernetes 1.37?

The 1.37 API enforces strict hostPath validation by default, rejecting pods that attempt to mount non-existent paths if configured with DirectoryOrCreate. You must ensure underlying paths exist or update the manifest type.

How do you list all active DRA ResourceClaims in a namespace?

Run kubectl get resourceclaims -n <namespace>. This displays the current binding status and allocation details for all external hardware requests.

Can you roll back a Kubernetes 1.37 cluster upgrade to 1.36?

You cannot safely downgrade the control plane data store once etcd commits the 1.37 schema migrations. You must restore your pre-upgrade etcd snapshot to return the cluster to a functional 1.36 state.

This post is part of the ‘k8s-1-37-upgrade’ series; next, we examine upgrading your CNI plugins to match the new 1.37 networking requirements.

Part of the series: k8s-1-37-upgrade

1. Why Your Kubernetes Cluster Might Silently Break After a 1.37 Upgrade
2. How to Ensure Your Kubernetes 1.37 Upgrade Doesn't Corrupt Your Nodes and Static Pods
3. One kubectl Command That Unlocks GPU-Aware Scheduling with Kubernetes 1.37 DRA
4. Is Your Kubernetes 1.37 Cluster Actually Stable? A 5-Minute Post-Upgrade Audit (you are here)

Further Reading

• https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/
• https://kubernetes.io/docs/reference/kubectl/cheatsheet/

---

🚀 Ready to get hands-on? Spin up an interactive AI or Kubernetes Sandbox at Aicademy Labs for free.