Amazon Web Services
View all tags
Scale-to-zero is not only a GPU trick. A CPU indexing lane — MySQL extract, chunk, Elasticsearch index, Parquet to S3 — now sits at zero pods and zero nodes when nothing is submitted. The GPU path is in Scale-to-zero GPU inference on EKS. This is the part that does not need a GPU, and the bug that killed a job that was still working.
A GPU node that sits idle overnight is not spare capacity. It is a line item. In my current role I run LLM, embedding, OCR, and vision inference on AWS EKS. GPU capacity across EKS and standalone EC2 peaked at 116 NVIDIA GPUs, including about 55 RTX PRO 6000s (96 GB, Blackwell). The scale-to-zero pools idle at $0.
A full cold start still takes minutes. That is the trade.
When I started, the EKS setup had the basics — workloads running, traffic flowing, deployments working. But from a security perspective, there was a lot of surface area: static AWS access keys baked into apps, default Amazon Linux nodes, no container image scanning, and a publicly accessible Kubernetes API. I hardened the stack in one month, one week at a time. Here is the journey.
Our team was building LLM-powered features — agents, chat interfaces, document analysis — and we had zero visibility into what the models were actually doing. No trace of which prompts were sent, what the latency was, how much each call cost, or whether the outputs were any good.
I evaluated hosted observability platforms, but data residency requirements meant we needed to keep everything in our own cloud account. So I self-hosted Langfuse — an open-source LLM observability platform — on AWS. Here is the architecture and what I learned.
We had four GPU workloads running on GCP virtual machines — speech-to-text, a general Triton inference server, Stable Diffusion 3, and an open-source LLM server. They were unmanaged, undocumented, and expensive. I moved all four to AWS EKS in about a month. Here is how it went and what I learned.
I spent a focused quarter understanding and reducing cloud costs without hurting reliability or developer velocity. By the end of it, the line items below added up to roughly $1,800 USD per month — over $20k annualized — from our AWS and GCP bills. Spot and GPU Reserved Instance discounts sit on top of that and are harder to pin to a single number.
Here are the 8 things that actually moved the needle.