You're Not Paying for Compute. You're Paying for Memory Bandwidth

TL;DR— Inference cost conversations obsess over FLOPs and token prices, but the real constraint on LLM serving is memory bandwidth— specifically the cost of moving the KV cache in and out of GPU memory on every decode step. Teams that optimize for compute utilization instead of memory traffic end up overpaying for capacity they never use. The fix is architectural: disaggregating prefill from decode, right-sizing batch and context, and treating bandwidth as the scarce resource it actually is.