AI/ML and LLM related
View all tags
A roughly 31B-parameter multimodal model, quantized to NVFP4, runs only on Blackwell. The 4-bit kernels are not a format you can fall back onto an L40S or an L4. The service is spot-only, with no fallback on purpose. If there is no Blackwell capacity, the pod stays Pending and nothing is billed.
The first Langfuse design on EKS was full HA and about $3,400 a month at the load I expected. Scenarios ran from about $1.8K to $7.7K. I re-checked it before buying the hardware. The estimate was about 3.5× too high. The tier that shipped is about $1,000 a month, roughly 71% below the original number.
The first Langfuse write-up on this site was an ECS sketch from earlier in the year. What I actually built, in my current role, is Langfuse v3 on EKS. It is validated on a lean tier. Apps are still cutting over from the old Docker Compose box.
These are the seven things that cause real outages, plus how the keys survive the move.
A GPU node that sits idle overnight is not spare capacity. It is a line item. In my current role I run LLM, embedding, OCR, and vision inference on AWS EKS. GPU capacity across EKS and standalone EC2 peaked at 116 NVIDIA GPUs, including about 55 RTX PRO 6000s (96 GB, Blackwell). The scale-to-zero pools idle at $0.
A full cold start still takes minutes. That is the trade.
Our team was building LLM-powered features — agents, chat interfaces, document analysis — and we had zero visibility into what the models were actually doing. No trace of which prompts were sent, what the latency was, how much each call cost, or whether the outputs were any good.
I evaluated hosted observability platforms, but data residency requirements meant we needed to keep everything in our own cloud account. So I self-hosted Langfuse — an open-source LLM observability platform — on AWS. Here is the architecture and what I learned.