Finding a Regression in NVIDIA Container Toolkit 1.20.0

11 अक्टूबर 2026 · 5 मिनट में पढ़ें

Cloud Physicist

GPU workloads on our EKS clusters started failing after a routine node group upgrade. The error was cryptic: nvidia.com/gpu resource not found on nodes that clearly had GPUs. Every GPU pod stuck in Pending. This is the story of finding and reproducing a regression in NVIDIA Container Toolkit 1.20.0 that shipped in the EKS NVIDIA AMI v20260818.

The Symptom

After Karpenter launched new nodes using the latest EKS NVIDIA AMI (v20260818, which included NVIDIA Container Toolkit 1.20.0), pods requesting nvidia.com/gpu stuck in Pending:

$ kubectl describe pod llm-inference-xyz
Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 3 Insufficient nvidia.com/gpu.

But the nodes had GPUs — nvidia-smi showed them, drivers loaded, everything looked fine. The NVIDIA device plugin just wasn't registering the nvidia.com/gpu resource. Something in the toolkit upgrade broke the device plugin's ability to start.

The Root Cause

Toolkit 1.20.0 added a new update-application-profile CDI hook. On every container create, this hook runs:

mkdir -p /etc/nvidia/nvidia-application-profiles-rc.d

The NVIDIA device plugin mounts its config at /etc/nvidia as a read-only ConfigMap (--config-file=/etc/nvidia/config). When the hook runs mkdir inside that read-only mount, it fails with EROFS (read-only file system). The hook exits non-zero, createContainer fails, and the device plugin never starts — so the node advertises 0 nvidia.com/gpu.

Error in kubelet logs:

createContainer hook #6: exit status 1 ... mkdirat /etc/nvidia/nvidia-application-profiles-rc.d: read-only file system

Minimal Repro

I stripped it down to the smallest possible test:

Launch a g5.xlarge (A10G), g6.xlarge (L4), or g6e.xlarge (L40S) with the affected EKS AMI (v20260818)
Deploy a pod that triggers the CDI hook — even a pod with no GPU request but with environment-based injection triggers it
Mount any ConfigMap at /etc/nvidia (read-only)
Container creation fails with EROFS

The same manifests worked on toolkit 1.19.x.

# Minimal repro - deploy this and watch it fail on 1.20.0
apiVersion: v1
kind: Pod
metadata:
  name: toolkit-repro
spec:
  containers:
  - name: test
    image: alpine:3.20
    command: ["sleep", "3600"]
    env:
    - name: NVIDIA_VISIBLE_DEVICES
      value: "all"  # triggers CDI hook
    volumeMounts:
    - name: config
      mountPath: /etc/nvidia
      readOnly: true
  volumes:
  - name: config
    configMap:
      name: dummy-config  # any ConfigMap

Reporting & Fix

I filed the issue at NVIDIA/nvidia-container-toolkit#2026 with:

Minimal repro steps
Exact error from kubelet logs
EKS AMI version (v20260818)
Root cause analysis (read-only ConfigMap + mkdir in CDI hook)

NVIDIA engineers confirmed the regression. The fix landed in PR #2028 (by @henry118, merged 27 Aug 2026) — the hook now ignores read-only filesystem errors when creating the application profiles directory.

Released in NVIDIA Container Toolkit 1.20.1.

I also filed a tracking issue at awslabs/amazon-eks-ami#2790 for the EKS AMI update.

EKS Workaround (Until AMI Update)

If you're on EKS and can't wait for the updated AMI, you have two options:

Option 1: Pin to the previous EKS NVIDIA AMI

# Terraform - pin to known-good AMI ID
resource "aws_eks_node_group" "gpu" {
  ami_type = "AL2_x86_64_GPU"  # or BOTTLEROCKET_x86_64_NVIDIA
  # Use the AMI ID from before v20260818, not "latest"
  # Find it in EC2 > AMIs > Owned by Amazon > amazon-eks-nvidia-*
}

Option 2: Mount the device-plugin config elsewhere (keep /etc/nvidia writable)

# Device plugin daemonset patch
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: nvidia-device-plugin-ctr
        # Change config mount path away from /etc/nvidia
        volumeMounts:
        - name: config
          mountPath: /etc/nvidia-device-plugin  # not /etc/nvidia
      volumes:
      - name: config
        configMap:
          name: nvidia-device-plugin-config

What This Taught Me

Pin critical AMIs. The EKS NVIDIA AMI "latest" moved under us. For GPU workloads, pin the AMI ID in Terraform/Karpenter.
Test GPU registration in CI. Add a step that launches a test node, installs the device plugin, and asserts nvidia.com/gpu > 0.
Upstream first. The fix was a one-line change once reproduced. Reporting it saved everyone else running GPU workloads on EKS.
CDI hooks run on every container create. Not just GPU pods — any pod with NVIDIA_VISIBLE_DEVICES triggers the hook. That's why even non-GPU workloads were affected.

The Symptom​

The Root Cause​

Minimal Repro​

Reporting & Fix​

EKS Workaround (Until AMI Update)​

What This Taught Me​

Links​