GPU workloads and AI/ML infrastructure
सारे टैग देखें
GPU workloads on our EKS clusters started failing after a routine node group upgrade. The error was cryptic: nvidia.com/gpu resource not found on nodes that clearly had GPUs. Every GPU pod stuck in Pending. This is the story of finding and reproducing a regression in NVIDIA Container Toolkit 1.20.0 that shipped in the EKS NVIDIA AMI v20260818.
g6 was not there when the session started. Not on Spot, and not as a normal On-Demand launch either. Karpenter would sit on InsufficientInstanceCapacity while a user was already waiting. I used an On-Demand Capacity Reservation to hold the instance before business hours, then release it when the day was over.
The point was the GPU existing at the right moment. The smaller bill was a side effect.
A roughly 31B-parameter multimodal model, quantized to NVFP4, runs only on Blackwell. The 4-bit kernels are not a format you can fall back onto an L40S or an L4. The service is spot-only, with no fallback on purpose. If there is no Blackwell capacity, the pod stays Pending and nothing is billed.
Each GPU workload in my current cluster has its own Karpenter NodePool. The isolation trick is small: keep the taint key as nvidia.com/gpu, and change only the value.
Scale-to-zero is cheap until the first request waits on an image pull. I benchmarked NVMe instance storage, Bottlerocket, SOCI, AL2023 FastImagePull, and EBS throughput on a fresh node every time.
Setup: g4dn.xlarge, a 3.6 GB image, one run per configuration. Treat the numbers as indicative, not a paper.
Two embedding models, one physical GPU. That was the May design. By August each pod had its own L40S. Time-slicing had added sharing and no throughput.
रात भर idle पड़ा GPU node "available capacity" नहीं है। वह एक line item है। Intermittent inference के लिए g6e या g7e 24/7 चलाओ तो पहला token जाने से पहले ही लगभग $10k/माह जल सकता है। मैंने तीन महीने में AWS EKS पर एक event-driven MLOps platform बनाया जो GPU replicas को zero तक scale करता है, mission-critical sessions को Spot से दूर रखता है, और 15 GB+ LLM images पर sub-60-second cold starts का लक्ष्य रखता है।
यह GPU path है: architecture, hardware constraints, और scars।
हमारे पास GCP virtual machines पर चार GPU workloads चल रहे थे — speech-to-text (Whisper), एक general Triton inference server, Stable Diffusion 3, और एक open-source LLM server (vLLM)। ये unmanaged थे, undocumented थे, और महँगे थे। मैंने चारों को लगभग एक महीने में AWS EKS पर शिफ़्ट किया। यह रहा पूरा अनुभव और क्या सीखा।