Serving a 31B 4-bit Model on Blackwell Spot GPUs

A roughly 31B-parameter multimodal model, quantized to NVFP4, runs only on Blackwell. The 4-bit kernels are not a format you can fall back onto an L40S or an L4. The service is spot-only, with no fallback on purpose. If there is no Blackwell capacity, the pod stays Pending and nothing is billed.

Weights are baked into a 29.5 GB image. The ALB idle timeout moved from 900 to 1,800 seconds. progressDeadlineSeconds on the Deployment is 3,600. The startup probe allows 15.5 minutes for kernel JIT on first start.

Probes

The liveness probe (180 seconds) is deliberately slower than the readiness probe (60 seconds). Heavy compute must not look like a dead process. Without the long startup budget, the kubelet kills the pod mid-compile and it loops forever.

Forcing the FlashInfer attention backend failed. Let vLLM pick the backend.

Why no fallback

A spot waterfall is the right default for the rest of the platform, written up in Scale-to-zero GPU inference on EKS. This model is the exception. An L40S cannot run NVFP4. Scheduling it there would Pending in a more confusing way, or serve an error after you paid for the node. Pending on the right family is cheaper.

Canary the AMI on one node and confirm it advertises a GPU before you raise the pool limit. A node that never advertises the device still bills.