Why Your Kubernetes Cluster Might Silently Break After a 1.37 Upgrade

You just upgraded to Kubernetes 1.37. The API server is healthy, deployments are rolling, and all your monitors are green. What you don’t see is that your kube-proxy configuration is now pointing to a deprecated feature, and your next node reboot could silently degrade or blackhole service traffic.

TL;DR: Kubernetes 1.37 deprecated the IPVS proxy mode. If your cluster uses it, upgrading without changing the configuration can lead to services failing to route traffic correctly after a kube-proxy restart. This post gives you the one-liners to check your current mode and the exact YAML diff to safely migrate to the supported iptables mode before you get paged.

What you’ll walk away with:

• A one-line command to definitively check which proxy mode your cluster uses.
• The exact ConfigMap patch to migrate from IPVS to iptables.
• Verification commands to prove the new mode is active and routing traffic.
• A pre-flight checklist for your next 1.37+ upgrade.

How Do I Check Which kube-proxy Mode My Cluster Is Using?

You can determine your kube-proxy mode by inspecting its ConfigMap in the kube-system namespace. The mode field directly specifies whether the proxy uses iptables, the now-deprecated ipvs, or userspace. Most clusters will use either iptables or ipvs.

Run this command to print the configuration and check the mode key:

kubectl get cm -n kube-system kube-proxy -o yaml | grep "mode:"

If your cluster is configured for IPVS, the output will be unambiguous.

    mode: "ipvs"

If you see this, you must take action before or during your next upgrade. IPVS (IP Virtual Server) is a transport-layer load balancing feature built into the Linux kernel, which kube-proxy used as an alternative to iptables for service routing. As of Kubernetes 1.26, it was already deprecated, and by 1.37 its removal is imminent.

You can also check the kube-proxy logs on any node to see which mode it started in.

# Pick a node in your cluster
NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')

# Find the kube-proxy pod on that node and check its logs
KUBE_PROXY_POD=$(kubectl get pods -n kube-system --field-selector spec.nodeName=$NODE_NAME -l k8s-app=kube-proxy -o jsonpath='{.items[0].metadata.name}')
kubectl logs $KUBE_PROXY_POD -n kube-system | grep "Using"

The output will explicitly state the mode.

I0521 10:00:00.123456       1 server.go:765] "Using ipvs Proxier"

This confirms the running process is using the deprecated mode.

Before any upgrade, always check the kube-proxy ConfigMap. It’s the source of truth for how services will be routed.

Why Was IPVS Mode Even Deprecated?

The kube-proxy IPVS mode was deprecated because it added significant maintenance overhead for a feature that saw limited adoption and whose benefits were increasingly being provided by CNI plugins and eBPF-based solutions. While offering potential performance gains for clusters with a massive number of services, it also introduced its own set of bugs and complexities.

The community consensus shifted towards simplifying kube-proxy and letting more advanced networking be handled lower in the stack. This aligns with a broader trend of pushing networking logic into the CNI layer, as seen with Cilium, or bypassing kube-proxy entirely. This kind of subtle dependency deprecation is exactly the sort of thing that can cause bizarre, hard-to-diagnose failures, much like the ‘phantom pod’ incident we faced with a faulty admission controller.

Here is a simplified diagram of the two data paths:

graph TD
    subgraph "IPVS Mode (Deprecated)"
        A["Request to <br/>Service IP:Port"] --> B{"IPVS<br/>in-kernel lookup"}
        B --> C1["Pod 1 IP"]
        B --> C2["Pod 2 IP"]
        B --> C3["..."]
    end

    subgraph "iptables Mode (Default)"
        X["Request to <br/>Service IP:Port"] --> Y{"iptables<br/>PREROUTING chain"}
        Y --> Z{"KUBE-SERVICES chain"}
        Z -- "DNAT to Pod IP" --> W["Pod Endpoint"]
    end

The key difference is that IPVS acts as a true layer-4 load balancer with more sophisticated scheduling algorithms, whereas iptables mode uses simple DNAT rules for packet redirection.

The deprecation signals a move away from kernel-level magic in kube-proxy towards more observable and CNI-integrated service routing.

What’s the Safe Way to Migrate from IPVS to iptables Mode?

The safest way to migrate is to apply a strategic merge patch to the kube-proxy ConfigMap. This change can be applied to a running cluster, and the kubelet on each node will automatically restart kube-proxy with the new configuration. Do this during a maintenance window, as there will be a brief moment of churn in network rules.

Here is the “before” and “after” kube-proxy-config.yaml snippet from the ConfigMap.

--- a/kube-proxy-config.yaml
+++ b/kube-proxy-config.yaml
@@ -1,5 +1,5 @@
 kind: KubeProxyConfiguration
 apiVersion: kubeproxy.config.k8s.io/v1alpha1
-mode: "ipvs"
+mode: "iptables"
 ipvs:
   strictARP: true

To apply this change live, use kubectl patch.

kubectl patch cm -n kube-system kube-proxy -p '{"data":{"config.conf":"apiVersion: kubeproxy.config.k8s.io/v1alpha1\nkind: KubeProxyConfiguration\nmode: iptables\n"}}'

This command replaces the entire config.conf key within the ConfigMap. Ensure you copy any other custom settings you might have from your existing configuration into the patch payload. A safer, more surgical approach for complex configs is to fetch, edit, and apply.

# 1. Get the current config
kubectl get cm -n kube-system kube-proxy -o yaml > kube-proxy-cm.yaml

# 2. Edit kube-proxy-cm.yaml, changing mode: "ipvs" to mode: "iptables"

# 3. Apply the updated file
kubectl apply -f kube-proxy-cm.yaml

Once applied, kube-proxy pods will restart one by one. Monitor them to ensure they come back up healthy.

kubectl get pods -n kube-system -l k8s-app=kube-proxy -w

Patch the ConfigMap during a maintenance window. While the change is fast, it is disruptive to the data plane on each node as rules are rewritten.

How Can I Verify the New Proxy Mode is Active?

After patching the ConfigMap, you must verify that kube-proxy has restarted in the correct mode and that the old IPVS rules are gone. First, check the logs again, as we did earlier. The output should now reflect the new mode.

# Re-run the log check from before
# ... find the pod name ...
kubectl logs $KUBE_PROXY_POD -n kube-system | grep "Using"

The expected output should now be:

I0521 10:05:00.654321       1 server.go:720] "Using iptables Proxier"

Next, and most importantly, check the IPVS ruleset on a node. If the migration was successful, it should be empty. You may need to apt-get install ipvsadm or yum install ipvsadm on the node if the tool isn’t present.

# exec into a privileged pod or ssh to a node
ipvsadm -Ln

A successful migration will produce empty output, as kube-proxy is no longer managing IPVS rules. Before the change, this command would have listed all your Kubernetes services. For a deeper dive into cluster networking security, check out the CKS Complete Exam Reference which covers kube-proxy and network policies in detail.

Your pre-upgrade checklist should look like this:

• Check current kube-proxy mode via ConfigMap.
• If ipvs, schedule a maintenance window to patch the ConfigMap to iptables.
• Apply the patch and monitor kube-proxy pod restarts.
• Verify the new mode by checking logs on multiple nodes.
• Verify IPVS rules are gone by running ipvsadm -Ln on a node.
• Proceed with the Kubernetes 1.37+ upgrade.

---

In the next part of this series, we’ll tackle the changes to storage volume topology-awareness and how it impacts stateful workloads.

Bottom Line

Do not upgrade to Kubernetes 1.37 or later without first auditing your kube-proxy mode. A configuration that worked for years will silently fail, leaving you with a networking puzzle that doesn’t throw obvious alerts. The fix is a simple one-line change, but only if you know to look for it.

FAQ

What is the performance difference between IPVS and iptables mode?

IPVS generally offers better performance and lower latency at very large scale (10,000+ services) due to its use of hash tables for lookups, which is more efficient than the sequential rule processing of iptables. For most clusters, the difference is negligible.

Can I switch to nftables mode instead?

The nftables mode is available but still considered beta in Kubernetes 1.37. It aims to replace the legacy iptables backend. Unless you have a specific reason to use it and have tested it thoroughly, migrating to the default and stable iptables mode is the recommended path.

Will this change affect my CNI plugin?

It depends. If your CNI (like Cilium) is configured to bypass kube-proxy entirely, this change has no effect. If your CNI relies on kube-proxy for service implementation, this migration is critical.

What happens if I upgrade without changing the mode from IPVS?

The kube-proxy pod will likely fail to start because the --proxy-mode=ipvs flag or config value is no longer valid. This means no new service rules will be programmed on that node, causing traffic to new or updated services to fail.

Is there an automated way to detect this during CI?

Yes, you can write a simple pre-flight check script that runs kubectl get cm -n kube-system kube-proxy -o jsonpath='{.data.config\.conf}' | grep 'mode: ipvs' as part of your upgrade pipeline. If the command finds a match, it should fail the build and alert the team.

Part of the series: k8s-1-37-upgrade

1. Why Your Kubernetes Cluster Might Silently Break After a 1.37 Upgrade (you are here)
2. How to Ensure Your Kubernetes 1.37 Upgrade Doesn't Corrupt Your Nodes and Static Pods

Further Reading

• https://github.com/kubernetes/sig-release/tree/master/releases/release-1.37
• https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/
• https://kubernetes.io/docs/concepts/services-networking/service/

---

🚀 Ready to get hands-on? Spin up an interactive AI or Kubernetes Sandbox at Aicademy Labs for free.