The 'Phantom Pod' Incident: How a Corrupted Admission Controller Grounded Our Cluster

It started on a Tuesday. A junior engineer on the payments team couldn’t deploy a simple Nginx container for a canary test. The deployment was stuck, with the ReplicaSet showing zero pods available and no helpful events to explain why.

TL;DR: A misconfigured Validating Admission Webhook with failurePolicy: Fail can silently block resource creation, leading to “phantom pod” incidents where the API server rejects requests before any record is created. This makes debugging impossible with standard kubectl get events. This post walks through the exact API server logs and webhook configuration that caused our outage and provides a playbook for safe webhook rollouts.

The setup was textbook: a GKE cluster running Kubernetes 1.36, a standard deployment pipeline, and multiple teams working independently. The payments team’s deployment YAML was trivial, something we’d all run a thousand times. Yet, kubectl apply would return success, and then… nothing. No new pod, no “Pending” status, no event in the namespace. The request simply vanished into the ether.

This is the story of how we found the ghost in the machine.

What Causes a Pod Creation Request to Disappear?

When kubectl apply succeeds but no pod object appears and no events are generated, the request is likely being rejected synchronously by the Kubernetes API server itself. This almost always points to a malfunctioning validating admission webhook, which is a mechanism to intercept and validate API requests before they are persisted to etcd. If a webhook is configured with a strict failure policy, its failure will block the request entirely.

The incident began with this deceptive state. A kubectl get pods showed nothing new, and kubectl describe replicaset <name> was maddeningly unhelpful.

Name:         nginx-deployment-5754944d6c
Namespace:    payments
Selector:     app=nginx,pod-template-hash=5754944d6c
Labels:       app=nginx
              pod-template-hash=5754944d6c
Annotations:  deployment.kubernetes.io/desired-replicas: 1
              deployment.kubernetes.io/max-replicas: 2
              deployment.kubernetes.io/revision: 1
Replicas:     0 current / 1 desired
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  # ... template details ...
Events:       <none>

Zero events. The ReplicaSet controller wanted to create a pod, but its request to the API server never resulted in a pod object it could track. This told us the problem wasn’t the scheduler, a CNI issue, or a resource quota problem. The failure was happening earlier in the lifecycle.

sequenceDiagram
    participant User as "kubectl apply"
    participant KubeAPI as "Kube API Server"
    participant Webhook as "Admission Webhook"
    participant ETCD

    User ->> KubeAPI: CREATE Pod request
    KubeAPI ->> Webhook: "Is this Pod spec valid?"
    Webhook --x KubeAPI: Error / Timeout
    KubeAPI -->> User: Error (500 Internal Server Error)
    Note over KubeAPI,ETCD: No Pod object is<br/>written to etcd.

Our “phantom pod” was a request that died inside the API server’s admission control phase, before it could be persisted. The user’s kubectl client got an error, but the controllers trying to reconcile state just saw their creation requests fail silently.

When you see a controller’s desired state at N > 0 but current state is 0 with zero events, suspect an admission controller is blocking the requests.

How Do You Find a Failing Admission Webhook?

You can identify a potentially problematic admission webhook by listing all ValidatingWebhookConfiguration resources and looking for any with failurePolicy: Fail. Then, you must inspect the Kubernetes API server logs for errors related to the webhook’s service, as this is the only place the failure will be recorded.

First, we listed the cluster’s webhooks.

kubectl get validatingwebhookconfigurations -o yaml

The output was long, but one entry, recently added by our security team, caught my eye.

Click to see the problematic webhook configuration

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: policy-enforcer.example.com
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: LS0tLS1CRU...
    service:
      name: policy-enforcer-svc
      namespace: security-tools
      path: /validate
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: vpolicy.example.com
  namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values:
      - kube-system
      - security-tools
  objectSelector: {}
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - pods
  sideEffects: None
  timeoutSeconds: 5

The key line was failurePolicy: Fail. The default value for this field is Ignore, which means that if the API server can’t reach the webhook, it simply allows the request to proceed. Setting it to Fail turns the webhook into a hard dependency for any resource creation matching its rules—in this case, all pods outside the kube-system namespace. This is a common requirement for security policies and a key part of the CKS exam curriculum.

Now we knew the “what”, but not the “why”. The policy-enforcer pods were running and healthy. So why was the API server failing to reach them? The only place to find the answer was the API server’s own logs.

On a managed service like GKE, this means going to the cloud provider’s logging console and filtering for logs from the control plane. We quickly found a flood of errors like this:

E0314 15:02:10.123456       1 webhook.go:142] Failed calling webhook "vpolicy.example.com": failed to call webhook: Post "https://policy-enforcer-svc.security-tools.svc:443/validate?timeout=5s": dial tcp 10.42.7.99:443: connect: connection refused

“Connection refused.” The API server could resolve the service IP (10.42.7.99) but something was refusing the connection on port 443. We checked the policy-enforcer-svc Service object:

kubectl get svc policy-enforcer-svc -n security-tools -o yaml

And there it was. The needle in the haystack.

apiVersion: v1
kind: Service
metadata:
  name: policy-enforcer-svc
  namespace: security-tools
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 8443 # <-- THE MISMATCH
  selector:
    app: policy-enforcer
  type: ClusterIP

The Service was listening on port 443, but forwarding traffic to targetPort 8443 on the backing pods. However, the ValidatingWebhookConfiguration explicitly told the API server to connect to port: 443. The API server connects directly to the pod IP and port specified by the service endpoint, but it uses the port number from the clientConfig, not the targetPort. A classic mismatch.

Always verify the port in a webhook’s clientConfig matches the targetPort on the backing Service if they are different. The API server does not automatically translate them.

How Do You Safely Fix a Broken Webhook?

The safest way to fix and redeploy a broken webhook is to first change its failurePolicy to Ignore, apply the change to unblock the cluster, and then apply the configuration fix. Once you’ve verified the webhook is receiving requests and responding correctly, you can switch the failurePolicy back to Fail.

We immediately patched the webhook configuration to unblock deployments.

--- a/webhook.yaml
+++ b/webhook.yaml
@@ -12,7 +12,7 @@
       namespace: security-tools
       path: /validate
       port: 8443
-  failurePolicy: Fail
+  failurePolicy: Ignore
   matchPolicy: Equivalent
   name: vpolicy.example.com
   namespaceSelector:

We made two changes. The immediate fix was changing failurePolicy to Ignore. This instantly unblocked the payments team. The second, correct fix was changing the port in the clientConfig to match the pod’s targetPort of 8443.

After applying this, we watched the policy-enforcer logs and saw validation requests flow in successfully. We then promoted the failurePolicy back to Fail and confirmed pod creation was still working. This two-step process—de-risk, then fix—is critical for any component that can single-handedly bring down your control plane. This is the kind of resilient thinking required when building out a true Internal Developer Platform.

Your first priority during a webhook-related outage is to change failurePolicy: Fail to Ignore to restore cluster functionality. Diagnose the root cause after the immediate impact is mitigated.

Bottom Line

A validating admission webhook with failurePolicy: Fail is a single point of failure for your cluster’s API. A simple typo in a service port can prevent all pod creations and updates. Never roll out a new webhook directly with a Fail policy; always start with Ignore, monitor the webhook’s metrics and logs, and only enable the Fail policy once you have high confidence in its stability and correctness.

FAQ

What is the difference between a validating and a mutating admission webhook?

A mutating webhook can change the object before it’s saved (e.g., to inject a sidecar container), while a validating webhook can only accept or reject the object as-is. Both run in sequence, with mutations happening before validations.

How can I monitor admission webhook performance?

The Kubernetes API server exposes Prometheus metrics like apiserver_admission_webhook_admission_duration_seconds. You should create alerts for high latency or error rates on these metrics for any webhook with failurePolicy: Fail.

Can a NetworkPolicy block an admission webhook?

Yes. If you have a default-deny NetworkPolicy, you must create a specific policy that allows ingress to your webhook’s pods from the control plane’s IP address block. A failure to do so will cause timeouts, which will block requests if failurePolicy is Fail.

What does the timeoutSeconds field in a webhook configuration do?

It tells the API server how long to wait for a response from the webhook. According to the Kubernetes 1.36 documentation, the value must be between 1 and 30 seconds, with a default of 10. A timeout is considered a failure, triggering the failurePolicy.

Is it safe to set failurePolicy: Fail on a mutating webhook?

It carries the same risks. A failing mutating webhook with a Fail policy can also block resource creation. The only difference is it runs earlier in the admission chain. The same rollout strategy (start with Ignore, monitor, then switch to Fail) is highly recommended.

Further Reading

• https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/
• https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/
• https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
• https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

---

🚀 Ready to get hands-on? Spin up an interactive AI or Kubernetes Sandbox at Aicademy Labs for free.