One kubectl Command That Unlocks GPU-Aware Scheduling with Kubernetes 1.37 DRA

Your expensive H100s are likely sitting idle because standard node labels cannot express granular device states. Relying on basic integer counting forces high-priority ML jobs into pending states while web servers hog mixed-use nodes. Kubernetes 1.37 fixes this by natively integrating device-specific taints and topology-aware scheduling.

TL;DR: Kubernetes 1.37 graduates Dynamic Resource Allocation (DRA) to support native device taints and tolerations for GPUs. This allows you to declaratively reserve specific GPU models (like A100s running nvidia-driver 560.10) without writing custom schedulers. This post gives you the exact server-side apply command and manifest structure to adopt DRA today.

What is Dynamic Resource Allocation in Kubernetes 1.37?

Dynamic Resource Allocation (DRA) is a Kubernetes API pattern that treats hardware devices as claimable objects rather than simple integer capacities. It replaces the legacy device plugin framework by allowing pods to reference specific ResourceClaim objects. The scheduler evaluates GPU topology and availability before binding, preventing workloads from landing on nodes with broken drivers.

If you have ever experienced The ‘Phantom Pod’ Incident: How a Corrupted Admission Controller Grounded Our Cluster, you know the pain of failed device bindings. DRA shifts allocation logic from the local kubelet to a centralized controller. This requires specific configurations in the kubelet DRA manager to function properly.

Stop using nvidia.com/gpu: 1 resource limits for specialized workloads; default to DRA ResourceClaims to prevent silent scheduling failures.

How Do You Apply DRA GPU Configurations via kubectl?

Creating a deterministic hardware claim requires Server-Side Apply to avoid field conflicts with external DRA drivers.

kubectl apply --server-side --field-manager=dra-admin --force-conflicts -f aicademy-gpu-claim.yaml

resourceclaim.resource.k8s.io/aicademy-h100-claim serverside-applied

Flag/Argument	Function
--server-side	Moves merge conflict resolution to the API server, required for DRA claim status fields.
--field-manager=dra-admin	Tags the operation so DRA controllers know who owns the claim specification.
--force-conflicts	Overwrites any existing claim definitions injected by legacy mutating webhooks.
-f aicademy-gpu-claim.yaml	The manifest containing the ResourceClaimTemplate and Pod definition.

Variations:

1. Check claim allocation status:

   kubectl get resourceclaims -l workload=ai-training -o wide
   

2. Debug a failed scheduling event:

   kubectl describe resourceclaim aicademy-h100-claim
   

3. Delete a bound claim and its pod:

   kubectl delete -f aicademy-gpu-claim.yaml --cascade=foreground
   

When NOT to use it: Avoid using --force-conflicts if your cluster runs an external autoscaler that actively mutates claim capacities on the fly.

Always run kubectl diff -f against your claims before forcing conflicts on production endpoints.

How Do You Configure DRA Device Taints in K8s 1.37?

Kubernetes 1.37 allows nodes to taint themselves based on device health, and pods must tolerate these if they strictly require the hardware. This prevents situations where a node upgrade destroys local routing, similar to Why Your Kubernetes Cluster Might Silently Break After a 1.37 Upgrade. You must update your manifests to leverage this.

  apiVersion: v1
  kind: Pod
  metadata:
    name: aicademy-inference
  spec:
    containers:
    - name: model-runner
      image: tensorrt:latest
-     resources:
-       limits:
-         nvidia.com/gpu: 1
+     resources:
+       claims:
+       - name: gpu-req
+         request: aicademy-h100-claim
+   tolerations:
+   - key: "dra.kubernetes.io/device-unready"
+     operator: "Exists"
+     effect: "NoSchedule"

This shift requires careful coordination with node lifecycle tools. Read How to Ensure Your Kubernetes 1.37 Upgrade Doesn’t Corrupt Your Nodes and Static Pods to ensure your kubelets register the DRA socket correctly. You can read more about standard node protections in the official taint and toleration documentation.

Click to view the full ResourceClaim manifest

apiVersion: resource.k8s.io/v1alpha2
kind: ResourceClaim
metadata:
  name: aicademy-h100-claim
spec:
  resourceClassName: nvidia-gpu
  parametersRef:
    apiGroup: dra.nvidia.com/v1alpha1
    kind: GpuClaimParameters
    name: h100-driver-560

Apply the dra.kubernetes.io/device-unready toleration exclusively to workloads that have custom application logic to handle degraded GPU states.

Which GPU Allocation Method Should You Use?

The legacy device plugin framework relies on the kubelet to manage integer counters. DRA shifts this responsibility to external resource drivers, offering granular control over exact driver versions like nvidia-driver: 560.10.

Feature	Legacy Device Plugin	K8s 1.37 DRA	Winner
Allocation	Node-local integer counts	Cluster-wide API objects	DRA
Topology	Best-effort NUMA alignment	Native scheduler integration	DRA
Device Sharing	Requires MPS or Time-Slicing	Native Multi-instance GPU (MIG)	DRA
Setup Complexity	DaemonSet deployment	Custom controllers & CRDs	Legacy

Migrate to DRA immediately if your environment requires pinning specific ML training jobs to exact PCIe bus topologies.

How Does the DRA Controller Bind Claims?

The scheduler evaluates the ResourceClaim before the pod ever reaches a node. This prevents the traditional race condition where a pod lands on a node but fails to start due to exhausted GPU memory.

sequenceDiagram
    participant Pod as "AI Workload"
    participant API as "API Server"
    participant Sched as "K8s Scheduler"
    participant DRA as "DRA Driver"

    Pod ->> API: "Create Pod + ResourceClaim"
    API ->> Sched: "Watch Pending Pod"
    Sched ->> DRA: "Request GPU Allocation"
    DRA -->> Sched: "Node A (nvidia-driver 560.10)"
    Sched ->> API: "Bind Pod to Node A"

Verify your external DRA driver logs if the ResourceClaim gets stuck in the Pending state for more than 30 seconds.

Bottom Line

Relying on integer-based device plugins in Kubernetes 1.37 artificially limits your infrastructure capabilities. DRA provides the declarative, topology-aware scheduling required for high-performance AI workloads running on specific hardware setups like nvidia-driver: 560.10. Audit your clusters today and transition your critical ML jobs to ResourceClaims.

FAQ

What happens if a node’s GPU driver crashes while using DRA?

The DRA controller dynamically applies a device-specific taint to the node. Unless your pod has a matching toleration, the scheduler will evict it and reschedule the ResourceClaim to a healthy node.

Does DRA support partial GPU allocation?

Yes. Unlike the legacy device plugin which deals in whole integers, DRA supports custom parameters. You can define exact memory partitions using technologies like Nvidia MIG.

Can I run DRA and Legacy Device Plugins simultaneously?

Yes, but not for the same physical device class. You can run legacy plugins for generic accelerators while reserving high-end GPUs strictly for DRA-managed ResourceClass objects.

What is the minimum NVIDIA driver version required for Kubernetes 1.37 DRA?

While exact requirements depend on your hardware, implementing full topology-aware DRA scheduling in 1.37 typically requires nvidia-driver version 560.10 or higher.

Next up in the k8s-1-37-upgrade series: Troubleshooting CNI plugin failures after deprecation removals.

Part of the series: k8s-1-37-upgrade

1. Why Your Kubernetes Cluster Might Silently Break After a 1.37 Upgrade
2. How to Ensure Your Kubernetes 1.37 Upgrade Doesn't Corrupt Your Nodes and Static Pods
3. One kubectl Command That Unlocks GPU-Aware Scheduling with Kubernetes 1.37 DRA (you are here)

Further Reading

• https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
• https://github.com/kubernetes/kubernetes/tree/master/pkg/kubelet/cm/dra

---

🚀 Ready to get hands-on? Spin up an interactive AI or Kubernetes Sandbox at Aicademy Labs for free.