Posts about reducing cloud costs and optimization strategies
View all tags
Credits hide the bill. The day they end, the same usage shows up as cash. The useful move is to split that bill into a baseline you can commit to, and a variable line you refuse to commit to, then buy the commitment only after the baseline is real.
g6 was not there when the session started. Not on Spot, and not as a normal On-Demand launch either. Karpenter would sit on InsufficientInstanceCapacity while a user was already waiting. I used an On-Demand Capacity Reservation to hold the instance before business hours, then release it when the day was over.
The point was the GPU existing at the right moment. The smaller bill was a side effect.
Scale-to-zero is cheap until the first request waits on an image pull. I benchmarked NVMe instance storage, Bottlerocket, SOCI, AL2023 FastImagePull, and EBS throughput on a fresh node every time.
Setup: g4dn.xlarge, a 3.6 GB image, one run per configuration. Treat the numbers as indicative, not a paper.
The first Langfuse design on EKS was full HA and about $3,400 a month at the load I expected. Scenarios ran from about $1.8K to $7.7K. I re-checked it before buying the hardware. The estimate was about 3.5× too high. The tier that shipped is about $1,000 a month, roughly 71% below the original number.
Two embedding models, one physical GPU. That was the May design. By August each pod had its own L40S. Time-slicing had added sharing and no throughput.
StarRocks and Airflow stay up on their own schedule. They do not scale to zero when an inference session ends, and they do not share a NodePool with a GPU. Both are live in my current role. Airflow runs on the KubernetesExecutor with Karpenter. It does not use KEDA.
Scale-to-zero is not only a GPU trick. A CPU indexing lane — MySQL extract, chunk, Elasticsearch index, Parquet to S3 — now sits at zero pods and zero nodes when nothing is submitted. The GPU path is in Scale-to-zero GPU inference on EKS. This is the part that does not need a GPU, and the bug that killed a job that was still working.
A GPU node that sits idle overnight is not spare capacity. It is a line item. In my current role I run LLM, embedding, OCR, and vision inference on AWS EKS. GPU capacity across EKS and standalone EC2 peaked at 116 NVIDIA GPUs, including about 55 RTX PRO 6000s (96 GB, Blackwell). The scale-to-zero pools idle at $0.
A full cold start still takes minutes. That is the trade.
I spent a focused quarter understanding and reducing cloud costs without hurting reliability or developer velocity. By the end of it, the line items below added up to roughly $1,800 USD per month — over $20k annualized — from our AWS and GCP bills. Spot and GPU Reserved Instance discounts sit on top of that and are harder to pin to a single number.
Here are the 8 things that actually moved the needle.