What GPU Time-Slicing Does Not Give You
· 3 min read
Two embedding models, one physical GPU. That was the May design. By August each pod had its own L40S. Time-slicing had added sharing and no throughput.
3 posts tagged with "MLOps"
MLOps platforms, inference serving, and GPU orchestration
View all tagsScale-to-Zero for CPU Jobs: the KEDA In-Flight Trap
· 4 min read
Scale-to-zero is not only a GPU trick. A CPU indexing lane — MySQL extract, chunk, Elasticsearch index, Parquet to S3 — now sits at zero pods and zero nodes when nothing is submitted. The GPU path is in Scale-to-zero GPU inference on EKS. This is the part that does not need a GPU, and the bug that killed a job that was still working.
Scale-to-Zero GPU Inference on EKS: $0 Idle with Karpenter and KEDA
· 7 min read
A GPU node that sits idle overnight is not spare capacity. It is a line item. In my current role I run LLM, embedding, OCR, and vision inference on AWS EKS. GPU capacity across EKS and standalone EC2 peaked at 116 NVIDIA GPUs, including about 55 RTX PRO 6000s (96 GB, Blackwell). The scale-to-zero pools idle at $0.
A full cold start still takes minutes. That is the trade.