GPU workloads and AI/ML infrastructure
View all tags
GPU workloads on our EKS clusters started failing after a routine node group upgrade. The error was cryptic: nvidia.com/gpu resource not found on nodes that clearly had GPUs. Every GPU pod stuck in Pending. This is the story of finding and reproducing a regression in NVIDIA Container Toolkit 1.20.0 that shipped in the EKS NVIDIA AMI v20260818.
g6 was not there when the session started. Not on Spot, and not as a normal On-Demand launch either. Karpenter would sit on InsufficientInstanceCapacity while a user was already waiting. I used an On-Demand Capacity Reservation to hold the instance before business hours, then release it when the day was over.
The point was the GPU existing at the right moment. The smaller bill was a side effect.
A roughly 31B-parameter multimodal model, quantized to NVFP4, runs only on Blackwell. The 4-bit kernels are not a format you can fall back onto an L40S or an L4. The service is spot-only, with no fallback on purpose. If there is no Blackwell capacity, the pod stays Pending and nothing is billed.
Each GPU workload in my current cluster has its own Karpenter NodePool. The isolation trick is small: keep the taint key as nvidia.com/gpu, and change only the value.
Scale-to-zero is cheap until the first request waits on an image pull. I benchmarked NVMe instance storage, Bottlerocket, SOCI, AL2023 FastImagePull, and EBS throughput on a fresh node every time.
Setup: g4dn.xlarge, a 3.6 GB image, one run per configuration. Treat the numbers as indicative, not a paper.
Two embedding models, one physical GPU. That was the May design. By August each pod had its own L40S. Time-slicing had added sharing and no throughput.
A GPU node that sits idle overnight is not spare capacity. It is a line item. In my current role I run LLM, embedding, OCR, and vision inference on AWS EKS. GPU capacity across EKS and standalone EC2 peaked at 116 NVIDIA GPUs, including about 55 RTX PRO 6000s (96 GB, Blackwell). The scale-to-zero pools idle at $0.
A full cold start still takes minutes. That is the trade.
We had four GPU workloads running on GCP virtual machines — speech-to-text, a general Triton inference server, Stable Diffusion 3, and an open-source LLM server. They were unmanaged, undocumented, and expensive. I moved all four to AWS EKS in about a month. Here is how it went and what I learned.