Kubernetes and container orchestration
सारे टैग देखें
GPU workloads on our EKS clusters started failing after a routine node group upgrade. The error was cryptic: nvidia.com/gpu resource not found on nodes that clearly had GPUs. Every GPU pod stuck in Pending. This is the story of finding and reproducing a regression in NVIDIA Container Toolkit 1.20.0 that shipped in the EKS NVIDIA AMI v20260818.
g6 was not there when the session started. Not on Spot, and not as a normal On-Demand launch either. Karpenter would sit on InsufficientInstanceCapacity while a user was already waiting. I used an On-Demand Capacity Reservation to hold the instance before business hours, then release it when the day was over.
The point was the GPU existing at the right moment. The smaller bill was a side effect.
A roughly 31B-parameter multimodal model, quantized to NVFP4, runs only on Blackwell. The 4-bit kernels are not a format you can fall back onto an L40S or an L4. The service is spot-only, with no fallback on purpose. If there is no Blackwell capacity, the pod stays Pending and nothing is billed.
Each GPU workload in my current cluster has its own Karpenter NodePool. The isolation trick is small: keep the taint key as nvidia.com/gpu, and change only the value.
Scale-to-zero is cheap until the first request waits on an image pull. I benchmarked NVMe instance storage, Bottlerocket, SOCI, AL2023 FastImagePull, and EBS throughput on a fresh node every time.
Setup: g4dn.xlarge, a 3.6 GB image, one run per configuration. Treat the numbers as indicative, not a paper.
The first Langfuse write-up on this site was an ECS sketch from earlier in the year. What I actually built, in my current role, is Langfuse v3 on EKS. It is validated on a lean tier. Apps are still cutting over from the old Docker Compose box.
These are the seven things that cause real outages, plus how the keys survive the move.
Two embedding models, one physical GPU. That was the May design. By August each pod had its own L40S. Time-slicing had added sharing and no throughput.
रात भर idle पड़ा GPU node "available capacity" नहीं है। वह एक line item है। Intermittent inference के लिए g6e या g7e 24/7 चलाओ तो पहला token जाने से पहले ही लगभग $10k/माह जल सकता है। मैंने तीन महीने में AWS EKS पर एक event-driven MLOps platform बनाया जो GPU replicas को zero तक scale करता है, mission-critical sessions को Spot से दूर रखता है, और 15 GB+ LLM images पर sub-60-second cold starts का लक्ष्य रखता है।
यह GPU path है: architecture, hardware constraints, और scars।
जब मैंने शुरू किया, EKS setup में basics थे — workloads चल रहे थे, traffic flow हो रहा था, deployments काम कर रहे थे। लेकिन security की नज़र से, काफ़ी surface area था: apps में baked static AWS access keys, default Amazon Linux nodes, कोई container image scanning नहीं, और publicly accessible Kubernetes API। पूरे stack को मैंने एक महीने में harden किया, एक हफ़्ता एक बार। यह रही पूरी journey।
मैंने एक पूरी तिमाही क्लाउड कॉस्ट को समझने और कम करने में लगाई — बिना reliability या developer velocity को नुकसान पहुँचाए। अंत तक नीचे दिए line items मिलाकर लगभग $1,800 USD प्रति माह — यानी सालाना $20k से ज़्यादा — AWS और GCP बिल से कट गए। Spot और GPU Reserved Instance discounts इसके ऊपर हैं, और उन्हें एक ही संख्या में बाँधना मुश्किल है।
यहाँ वो 8 चीज़ें हैं जिनसे असल में फ़र्क पड़ा।